2024-03-28T18:19:17.486 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T18:19:17.486 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} --verbose --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387 --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/orig.config.yaml 2024-03-28T18:19:17.537 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T18:19:17.631 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} email: mchangir@redhat.com first_in_suite: false job_id: '7625387' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3D2qClkYq8nAEu0rZhpmCEfxBkuWlfDN9eMuzEB+giRqXCcxNfU/Q/95KjT8tI9mZyHFB8XoWlbjVGhezySRY= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKsiWx3n6ppHzrTRPwWzQgokE0w78ZQ/LD2WGKTuWbkTncUKSqWiAYECxeD/DE/dGPOiPY+Z0ltCPxHhvgNMUJM= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd8P2VxQIiY24rYqHX0TVDB22s7ROE0EWQlxXTK5wXxAgVlTIgOEuRcmwPH3udymPbUxcO38D8N2MXGLM/4GAY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T18:19:17.631 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T18:19:17.631 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T18:19:17.632 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T18:19:17.633 INFO:teuthology.task.internal:Checking packages... 2024-03-28T18:19:17.654 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T18:19:17.654 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T18:19:17.654 INFO:teuthology.packaging:ref: None 2024-03-28T18:19:17.654 INFO:teuthology.packaging:tag: None 2024-03-28T18:19:17.654 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T18:19:17.654 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:19:17.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T18:19:17.871 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T18:19:17.872 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T18:19:17.878 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T18:19:17.878 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T18:19:17.897 INFO:teuthology.task.internal:Saving configuration 2024-03-28T18:19:17.916 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T18:19:17.923 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:19:17.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.843969', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3D2qClkYq8nAEu0rZhpmCEfxBkuWlfDN9eMuzEB+giRqXCcxNfU/Q/95KjT8tI9mZyHFB8XoWlbjVGhezySRY='} 2024-03-28T18:19:17.967 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.847614', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKsiWx3n6ppHzrTRPwWzQgokE0w78ZQ/LD2WGKTuWbkTncUKSqWiAYECxeD/DE/dGPOiPY+Z0ltCPxHhvgNMUJM='} 2024-03-28T18:19:17.986 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.845921', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd8P2VxQIiY24rYqHX0TVDB22s7ROE0EWQlxXTK5wXxAgVlTIgOEuRcmwPH3udymPbUxcO38D8N2MXGLM/4GAY='} 2024-03-28T18:19:17.986 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T18:19:17.994 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T18:19:17.994 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T18:19:17.994 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T18:19:17.994 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T18:19:18.119 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8a7d5d00d0>, signals=[15]) 2024-03-28T18:19:18.120 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T18:19:18.127 INFO:teuthology.task.internal:Opening connections... 2024-03-28T18:19:18.127 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:19:18.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:18.218 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-03-28T18:19:18.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:18.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-28T18:19:18.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:19:18.388 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T18:19:18.395 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-03-28T18:19:18.413 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-03-28T18:19:18.413 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:19:18.471 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:19:18.472 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:19:18.472 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:19:18.472 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:19:18.472 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:19:18.472 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-03-28T18:19:18.493 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-03-28T18:19:18.511 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-03-28T18:19:18.511 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:19:18.568 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:19:18.569 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:19:18.569 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:19:18.569 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-03-28T18:19:18.593 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-28T18:19:18.610 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-28T18:19:18.610 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-28T18:19:18.664 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-28T18:19:18.664 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-28T18:19:18.664 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-28T18:19:18.664 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-28T18:19:18.664 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T18:19:18.665 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T18:19:18.665 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-28T18:19:18.684 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T18:19:18.696 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T18:19:18.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T18:19:18.710 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:19:18.713 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:19:18.715 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-28T18:19:18.731 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T18:19:18.741 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T18:19:18.741 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:19:18.769 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:19:18.771 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-28T18:19:18.872 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T18:19:18.945 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T18:19:18.962 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T18:19:18.963 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-28T18:19:18.963 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-28T18:19:18.963 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:19:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:19:18.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-28T18:19:19.271 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:19:19.271 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-28T18:19:19.271 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-28T18:19:19.288 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-361.el9.x86_64 2024-03-28T18:19:19.288 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:19:19.289 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-28T18:19:19.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum upgrade -y linux-firmware 2024-03-28T18:19:21.415 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T18:19:23.090 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-28T18:19:24.339 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-03-28T18:19:46.446 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 918 kB/s | 19 MB 00:21 2024-03-28T18:19:50.563 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-28T18:19:52.063 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-28T18:19:53.167 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-28T18:19:58.023 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-28T18:20:00.321 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-28T18:20:00.330 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 1 Package 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 375 M 2024-03-28T18:20:00.331 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-28T18:20:07.116 INFO:teuthology.orchestra.run.smithi119.stdout:linux-firmware-20240219-143.el9.noarch.rpm 55 MB/s | 375 MB 00:06 2024-03-28T18:20:07.117 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:20:07.118 INFO:teuthology.orchestra.run.smithi119.stdout:Total 55 MB/s | 375 MB 00:06 2024-03-28T18:20:07.118 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-28T18:20:07.214 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-28T18:20:07.214 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-28T18:20:09.613 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-28T18:20:09.613 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-28T18:20:15.830 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-28T18:20:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:20:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:20:17.566 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:20:17.566 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-28T18:20:17.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-28T18:20:17.951 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:20:17.951 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-28T18:20:17.951 INFO:teuthology.orchestra.run.smithi119.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-28T18:20:17.951 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:20:17.951 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-28T18:20:18.218 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T18:20:18.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-28T18:20:18.433 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ 2024-03-28T18:20:18.434 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-28T18:20:18.436 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi119.front.sepia.ceph.com to complete... 2024-03-28T18:20:31.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28 18:20:31 URL:https://3.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/9/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [336317861/336317861] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-28T18:20:31.146 INFO:teuthology.orchestra.run.smithi119.stderr:FINISHED --2024-03-28 18:20:31-- 2024-03-28T18:20:31.146 INFO:teuthology.orchestra.run.smithi119.stderr:Total wall clock time: 13s 2024-03-28T18:20:31.146 INFO:teuthology.orchestra.run.smithi119.stderr:Downloaded: 1 files, 321M in 12s (26.0 MB/s) 2024-03-28T18:20:31.149 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-28T18:20:31.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-28T18:20:33.069 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying... ######################################## 2024-03-28T18:20:33.130 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing... ######################################## 2024-03-28T18:20:33.209 INFO:teuthology.orchestra.run.smithi119.stdout:Updating / installing... 2024-03-28T18:20:40.690 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-28T18:20:56.585 INFO:teuthology.orchestra.run.smithi119.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T18:21:19.086 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-28T18:21:19.086 DEBUG:teuthology.orchestra.run.smithi119:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-28T18:21:19.157 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-28T18:21:19.158 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-28T18:21:19.159 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-28T18:21:19.160 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-28T18:21:19.161 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-28T18:21:19.162 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-28T18:21:19.163 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-28T18:21:19.164 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-28T18:21:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-28T18:21:19.166 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-28T18:21:19.167 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-28T18:21:19.168 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-28T18:21:19.169 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-28T18:21:19.170 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-28T18:21:19.171 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-28T18:21:19.172 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-28T18:21:19.173 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-28T18:21:19.174 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-28T18:21:19.175 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-28T18:21:19.176 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-28T18:21:19.177 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-28T18:21:19.178 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-28T18:21:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-28T18:21:19.180 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-28T18:21:19.181 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-28T18:21:19.182 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-28T18:21:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-28T18:21:19.184 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-28T18:21:19.185 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-28T18:21:19.186 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-28T18:21:19.187 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-28T18:21:19.188 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-28T18:21:19.189 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-28T18:21:19.190 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-28T18:21:19.191 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-28T18:21:19.192 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-28T18:21:19.193 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-28T18:21:19.194 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-28T18:21:19.195 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-28T18:21:19.196 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-28T18:21:19.197 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-28T18:21:19.198 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-28T18:21:19.199 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-28T18:21:19.200 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-28T18:21:19.201 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-28T18:21:19.202 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-28T18:21:19.203 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-28T18:21:19.204 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-28T18:21:19.205 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-28T18:21:19.206 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-28T18:21:19.207 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-28T18:21:19.208 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-28T18:21:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-28T18:21:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-28T18:21:19.211 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-28T18:21:19.212 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-28T18:21:19.213 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-28T18:21:19.214 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-28T18:21:19.215 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-28T18:21:19.216 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-28T18:21:19.217 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-28T18:21:19.218 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-28T18:21:19.219 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-28T18:21:19.220 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-28T18:21:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-28T18:21:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-28T18:21:19.223 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-28T18:21:19.224 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-28T18:21:19.225 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-28T18:21:19.226 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-28T18:21:19.227 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-28T18:21:19.228 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-28T18:21:19.229 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-28T18:21:19.230 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-28T18:21:19.231 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-28T18:21:19.232 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-28T18:21:19.233 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-28T18:21:19.234 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-28T18:21:19.235 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-28T18:21:19.236 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-28T18:21:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-28T18:21:19.238 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-28T18:21:19.239 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-28T18:21:19.240 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-28T18:21:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-28T18:21:19.242 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-28T18:21:19.243 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-28T18:21:19.244 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-28T18:21:19.245 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-28T18:21:19.246 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-28T18:21:19.247 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-28T18:21:19.248 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-28T18:21:19.249 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-28T18:21:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-28T18:21:19.251 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-28T18:21:19.252 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-28T18:21:19.253 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-28T18:21:19.254 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-28T18:21:19.255 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-28T18:21:19.256 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-28T18:21:19.257 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-28T18:21:19.258 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-28T18:21:19.259 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-28T18:21:19.260 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-28T18:21:19.261 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-28T18:21:19.262 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-28T18:21:19.263 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-28T18:21:19.264 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-28T18:21:19.265 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-28T18:21:19.266 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-28T18:21:19.267 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-28T18:21:19.268 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-28T18:21:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-28T18:21:19.270 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-28T18:21:19.271 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-28T18:21:19.272 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-28T18:21:19.273 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-28T18:21:19.274 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-28T18:21:19.275 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-28T18:21:19.276 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-28T18:21:19.277 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-28T18:21:19.278 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-28T18:21:19.279 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-28T18:21:19.280 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-28T18:21:19.281 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-28T18:21:19.282 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-28T18:21:19.282 INFO:teuthology.orchestra.run.smithi119.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:21:19.285 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-28T18:21:19.285 DEBUG:teuthology.orchestra.run.smithi119:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-28T18:21:19.310 INFO:teuthology.orchestra.run.smithi119.stdout:postinstall scriptlet (using /bin/sh): 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout:fi 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout:for file in vmlinuz System.map config; do 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-28T18:21:19.311 INFO:teuthology.orchestra.run.smithi119.stdout: fi 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:done 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:fi 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-28T18:21:19.312 INFO:teuthology.orchestra.run.smithi119.stdout:fi 2024-03-28T18:21:19.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-03-28T18:21:19.386 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-03-28T18:21:19.386 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 68.el9 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8264254 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-03-28T18:21:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:21:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T18:21:19.389 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T18:21:19.389 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 6.8.0-g1df0c7e8a235 2024-03-28T18:21:19.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T18:21:19.986 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-03-28T18:21:21.391 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T18:21:21.429 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-03-28T18:21:21.438 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-03-28T18:21:21.453 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.96g0iaVj57 2024-03-28T18:21:21.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.96g0iaVj57 2024-03-28T18:21:21.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.96g0iaVj57 2024-03-28T18:21:21.681 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.96g0iaVj57 is 6KB 2024-03-28T18:21:21.730 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.96g0iaVj57 2024-03-28T18:21:21.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T18:21:21.816 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T18:21:21.816 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T18:21:21.816 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T18:21:21.816 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235.conf 2024-03-28T18:21:21.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g1df0c7e8a235 2024-03-28T18:21:21.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-03-28T18:21:51.970 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T18:21:51.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:21:51.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:22:10.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-28T18:22:19.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:22:19.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:22:22.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-28T18:22:34.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:22:34.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:22:52.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-28T18:23:07.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:23:07.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:23:08.327 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-28T18:23:09.031 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:23:09.031 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-28T18:23:09.031 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-28T18:23:09.068 INFO:teuthology.orchestra.run.smithi119.stdout:6.8.0-g1df0c7e8a235 2024-03-28T18:23:09.068 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-28T18:23:09.068 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T18:23:09.068 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T18:23:10.070 DEBUG:teuthology.parallel:result is None 2024-03-28T18:23:10.070 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T18:23:10.078 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T18:23:10.078 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:23:10.086 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:23:10.088 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T18:23:10.132 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T18:23:10.143 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T18:23:10.156 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T18:23:10.157 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:23:10.161 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:23:10.163 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T18:23:10.268 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T18:23:10.279 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T18:23:10.279 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:23:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:23:10.296 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T18:23:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:23:10.312 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:23:10.317 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:23:10.339 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T18:23:10.386 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.404 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.413 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.436 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.447 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.534 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T18:23:10.539 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T18:23:10.547 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T18:23:10.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:23:10.553 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:23:10.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T18:23:10.633 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T18:23:10.646 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T18:23:10.646 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:23:10.650 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:23:10.652 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T18:23:10.687 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:10.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:10.883 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:10.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:11.028 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:23:11.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:23:11.103 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:11.141 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:11.205 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:11.285 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:11.333 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:23:11.333 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:23:11.402 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:11.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T18:23:11.509 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:11.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T18:23:11.595 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:23:11.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T18:23:11.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-03-28T18:23:11.670 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-03-28T18:23:11.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-28T18:23:11.713 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:23:11.744 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:23:11.786 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:23:12.221 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T18:23:12.229 INFO:teuthology.task.internal:Starting timer... 2024-03-28T18:23:12.229 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T18:23:12.262 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T18:23:12.273 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T18:23:12.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-03-28T18:23:12.394 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-03-28T18:23:12.397 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-03-28T18:23:12.464 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-03-28T18:23:12.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-28T18:23:12.534 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-28T18:23:12.536 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T18:23:12.536 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-03-28T18:23:12.605 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-03-28T18:23:12.606 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-03-28T18:23:12.631 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-03-28T18:23:12.631 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-28T18:23:12.651 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-28T18:23:12.652 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-28T18:23:12.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:23:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:23:12.729 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:23:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:23:12.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:23:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:23:12.787 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T18:23:12.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:23:12.876 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:23:12.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-28T18:23:12.945 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T18:23:12.955 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T18:23:12.961 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T18:23:13.108 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T18:23:13.122 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T18:23:13.123 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com,smithi131.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T18:29:14.152 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-28T18:29:14.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:29:14.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:29:14.293 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-28T18:29:14.440 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-28T18:29:14.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-28T18:29:14.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:29:14.527 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-03-28T18:29:14.597 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-28T18:29:14.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-28T18:29:14.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T18:29:14.685 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-28T18:29:14.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-28T18:29:14.754 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T18:29:14.766 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T18:29:14.766 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:29:14.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:29:14.774 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:29:14.774 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:29:14.776 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T18:29:14.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:29:14.805 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:29:14.822 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:29:14.826 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:29:14.843 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:29:14.851 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-03-28T18:29:14.865 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-03-28T18:29:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T18:29:14.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-28T18:29:14.881 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:29:14.887 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-28T18:29:14.897 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:29:14.904 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:29:14.920 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:29:14.920 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T18:29:14.941 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.963 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:14.967 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-03-28T18:29:15.060 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-03-28T18:29:15.110 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T18:29:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T18:29:15.290 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:15.298 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:29:15.300 INFO:teuthology.run_tasks:Running task install... 2024-03-28T18:29:15.308 DEBUG:teuthology.task.install:project ceph 2024-03-28T18:29:15.308 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T18:29:15.308 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T18:29:15.308 INFO:teuthology.task.install:Using flavor: default 2024-03-28T18:29:15.314 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T18:29:15.314 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T18:29:15.314 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:29:15.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:29:15.316 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:29:15.316 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:29:15.317 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T18:29:15.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:29:15.455 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:29:15.455 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:29:15.463 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:29:15.463 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:29:15.530 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T18:29:15.530 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T18:29:15.629 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:29:15.629 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:29:15.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:29:15.632 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:29:15.632 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:29:15.633 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:29:15.636 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T18:29:15.637 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:29:15.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T18:29:15.661 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:29:15.661 DEBUG:teuthology.orchestra.run.smithi131:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:29:15.666 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:29:15.666 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:29:15.729 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T18:29:15.730 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T18:29:15.731 DEBUG:teuthology.orchestra.run.smithi131:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:29:15.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:29:15.810 DEBUG:teuthology.orchestra.run.smithi131:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:29:15.815 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:29:15.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T18:29:15.877 INFO:teuthology.orchestra.run.smithi131.stdout:check_obsoletes = 1 2024-03-28T18:29:15.878 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum clean all 2024-03-28T18:29:15.880 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-28T18:29:15.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-28T18:29:15.999 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T18:29:16.091 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-03-28T18:29:16.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-03-28T18:29:16.113 INFO:teuthology.orchestra.run.smithi131.stdout:74 files removed 2024-03-28T18:29:16.117 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-28T18:29:16.139 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:29:16.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:29:16.457 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-03-28T18:29:16.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T18:29:16.883 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 184 kB/s | 83 kB 00:00 2024-03-28T18:29:16.891 INFO:teuthology.orchestra.run.smithi131.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-28T18:29:17.267 INFO:teuthology.orchestra.run.smithi131.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-28T18:29:17.284 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-28T18:29:17.546 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 155 kB/s | 83 kB 00:00 2024-03-28T18:29:17.609 INFO:teuthology.orchestra.run.smithi131.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-28T18:29:17.676 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-28T18:29:18.045 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T18:29:18.087 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-28T18:29:18.096 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-28T18:29:18.647 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-28T18:29:19.241 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-28T18:29:19.404 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-28T18:29:19.554 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-28T18:29:20.236 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-28T18:29:20.876 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-28T18:29:21.111 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-03-28T18:29:21.847 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-28T18:29:21.921 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-28T18:29:23.212 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-28T18:29:26.206 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-28T18:29:27.005 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-28T18:29:27.639 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-28T18:29:28.464 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-28T18:29:29.101 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T18:29:30.482 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-28T18:29:34.207 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-28T18:29:35.523 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-28T18:29:36.574 INFO:teuthology.orchestra.run.smithi131.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:36.575 INFO:teuthology.orchestra.run.smithi131.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:36.587 INFO:teuthology.orchestra.run.smithi131.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:29:36.588 INFO:teuthology.orchestra.run.smithi131.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:29:36.589 INFO:teuthology.orchestra.run.smithi131.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:29:36.590 INFO:teuthology.orchestra.run.smithi131.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:36.592 INFO:teuthology.orchestra.run.smithi131.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:36.594 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:29:36.595 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:29:36.676 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================================ 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================================ 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:29:36.688 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:29:36.689 INFO:teuthology.orchestra.run.smithi131.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:29:36.690 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrading: 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:29:36.691 INFO:teuthology.orchestra.run.smithi131.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:29:36.692 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:29:36.693 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:29:36.694 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:29:36.695 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:29:36.696 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:29:36.697 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:29:36.698 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:29:36.699 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout:Installing weak dependencies: 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:29:36.700 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================================ 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout:Install 150 Packages 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrade 24 Packages 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 225 M 2024-03-28T18:29:36.701 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-03-28T18:29:37.121 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-28T18:29:37.466 INFO:teuthology.orchestra.run.smithi131.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 35 kB/s | 6.6 kB 00:00 2024-03-28T18:29:37.850 INFO:teuthology.orchestra.run.smithi131.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.3 MB/s | 890 kB 00:00 2024-03-28T18:29:37.925 INFO:teuthology.orchestra.run.smithi131.stdout:(3/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 8.3 MB/s | 5.4 MB 00:00 2024-03-28T18:29:37.947 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:37.948 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:37.950 INFO:teuthology.orchestra.run.smithi131.stdout:(4/174): ceph-immutable-object-cache-19.0.0-214 1.4 MB/s | 145 kB 00:00 2024-03-28T18:29:37.963 INFO:teuthology.orchestra.run.smithi196.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:29:37.964 INFO:teuthology.orchestra.run.smithi196.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:29:37.965 INFO:teuthology.orchestra.run.smithi196.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:29:37.966 INFO:teuthology.orchestra.run.smithi196.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:37.968 INFO:teuthology.orchestra.run.smithi196.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:37.971 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:29:37.972 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:29:38.054 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:29:38.071 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:29:38.072 INFO:teuthology.orchestra.run.smithi196.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:29:38.073 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:29:38.074 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:38.075 INFO:teuthology.orchestra.run.smithi196.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:29:38.076 INFO:teuthology.orchestra.run.smithi196.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:29:38.077 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:29:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:29:38.079 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:29:38.080 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:29:38.081 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:29:38.082 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:29:38.083 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:29:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout:Install 150 Packages 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 24 Packages 2024-03-28T18:29:38.085 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:29:38.086 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 225 M 2024-03-28T18:29:38.086 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-28T18:29:38.109 INFO:teuthology.orchestra.run.smithi131.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 2.3 MB 00:00 2024-03-28T18:29:38.484 INFO:teuthology.orchestra.run.smithi131.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-28T18:29:38.517 INFO:teuthology.orchestra.run.smithi131.stdout:(7/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-28T18:29:39.032 INFO:teuthology.orchestra.run.smithi196.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-28T18:29:39.776 INFO:teuthology.orchestra.run.smithi131.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 8.2 MB/s | 20 MB 00:02 2024-03-28T18:29:39.865 INFO:teuthology.orchestra.run.smithi196.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 890 kB 00:00 2024-03-28T18:29:39.935 INFO:teuthology.orchestra.run.smithi131.stdout:(9/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 17 MB 00:01 2024-03-28T18:29:39.951 INFO:teuthology.orchestra.run.smithi131.stdout:(10/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el 144 kB/s | 25 kB 00:00 2024-03-28T18:29:39.957 INFO:teuthology.orchestra.run.smithi196.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.6 MB/s | 145 kB 00:00 2024-03-28T18:29:40.002 INFO:teuthology.orchestra.run.smithi131.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 4.3 MB/s | 220 kB 00:00 2024-03-28T18:29:40.068 INFO:teuthology.orchestra.run.smithi131.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 406 kB/s | 27 kB 00:00 2024-03-28T18:29:40.135 INFO:teuthology.orchestra.run.smithi131.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 469 kB/s | 31 kB 00:00 2024-03-28T18:29:40.294 INFO:teuthology.orchestra.run.smithi131.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.6 MB/s | 746 kB 00:00 2024-03-28T18:29:40.332 INFO:teuthology.orchestra.run.smithi196.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 3.1 MB/s | 5.4 MB 00:01 2024-03-28T18:29:40.377 INFO:teuthology.orchestra.run.smithi131.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.9 MB/s | 163 kB 00:00 2024-03-28T18:29:40.470 INFO:teuthology.orchestra.run.smithi131.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-28T18:29:40.620 INFO:teuthology.orchestra.run.smithi131.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.3 MB/s | 501 kB 00:00 2024-03-28T18:29:40.750 INFO:teuthology.orchestra.run.smithi196.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 3.6 MB/s | 1.5 MB 00:00 2024-03-28T18:29:41.453 INFO:teuthology.orchestra.run.smithi131.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 6.2 MB/s | 5.1 MB 00:00 2024-03-28T18:29:41.529 INFO:teuthology.orchestra.run.smithi131.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 604 kB/s | 45 kB 00:00 2024-03-28T18:29:41.789 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.1 MB 00:00 2024-03-28T18:29:41.799 INFO:teuthology.orchestra.run.smithi196.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.3 MB/s | 2.3 MB 00:01 2024-03-28T18:29:42.037 INFO:teuthology.orchestra.run.smithi131.stdout:(20/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 2.8 MB/s | 9.9 MB 00:03 2024-03-28T18:29:42.154 INFO:teuthology.orchestra.run.smithi131.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 162 kB 00:00 2024-03-28T18:29:42.262 INFO:teuthology.orchestra.run.smithi131.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 321 kB 00:00 2024-03-28T18:29:42.371 INFO:teuthology.orchestra.run.smithi131.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 298 kB 00:00 2024-03-28T18:29:42.413 INFO:teuthology.orchestra.run.smithi131.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 2.3 MB/s | 99 kB 00:00 2024-03-28T18:29:42.455 INFO:teuthology.orchestra.run.smithi131.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.0 MB/s | 85 kB 00:00 2024-03-28T18:29:42.872 INFO:teuthology.orchestra.run.smithi131.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.4 MB/s | 3.1 MB 00:00 2024-03-28T18:29:42.922 INFO:teuthology.orchestra.run.smithi131.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T18:29:43.208 INFO:teuthology.orchestra.run.smithi196.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-03-28T18:29:43.281 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-28T18:29:43.381 INFO:teuthology.orchestra.run.smithi131.stdout:(28/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-28T18:29:43.406 INFO:teuthology.orchestra.run.smithi131.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 70 kB/s | 34 kB 00:00 2024-03-28T18:29:43.431 INFO:teuthology.orchestra.run.smithi131.stdout:(30/174): python3-ceph-common-19.0.0-2145.g3be9 69 kB/s | 131 kB 00:01 2024-03-28T18:29:43.457 INFO:teuthology.orchestra.run.smithi131.stdout:(31/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.8 MB/s | 139 kB 00:00 2024-03-28T18:29:43.515 INFO:teuthology.orchestra.run.smithi131.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 4.2 MB/s | 248 kB 00:00 2024-03-28T18:29:43.582 INFO:teuthology.orchestra.run.smithi131.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 739 kB/s | 49 kB 00:00 2024-03-28T18:29:43.649 INFO:teuthology.orchestra.run.smithi131.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 247 kB/s | 16 kB 00:00 2024-03-28T18:29:43.724 INFO:teuthology.orchestra.run.smithi131.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.4 MB/s | 264 kB 00:00 2024-03-28T18:29:43.850 INFO:teuthology.orchestra.run.smithi131.stdout:(36/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 4.1 MB/s | 1.8 MB 00:00 2024-03-28T18:29:43.883 INFO:teuthology.orchestra.run.smithi131.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-28T18:29:43.903 INFO:teuthology.orchestra.run.smithi131.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 347 kB/s | 18 kB 00:00 2024-03-28T18:29:44.053 INFO:teuthology.orchestra.run.smithi131.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 662 kB 00:00 2024-03-28T18:29:44.095 INFO:teuthology.orchestra.run.smithi131.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.5 MB/s | 920 kB 00:00 2024-03-28T18:29:44.120 INFO:teuthology.orchestra.run.smithi131.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T18:29:44.145 INFO:teuthology.orchestra.run.smithi131.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T18:29:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 760 kB/s | 32 kB 00:00 2024-03-28T18:29:44.255 INFO:teuthology.orchestra.run.smithi131.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-03-28T18:29:44.283 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-28T18:29:44.313 INFO:teuthology.orchestra.run.smithi131.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-28T18:29:44.330 INFO:teuthology.orchestra.run.smithi131.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 103 kB/s | 17 kB 00:00 2024-03-28T18:29:44.347 INFO:teuthology.orchestra.run.smithi131.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 10 kB 00:00 2024-03-28T18:29:44.372 INFO:teuthology.orchestra.run.smithi131.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-28T18:29:44.409 INFO:teuthology.orchestra.run.smithi131.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 564 kB/s | 44 kB 00:00 2024-03-28T18:29:44.448 INFO:teuthology.orchestra.run.smithi131.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-28T18:29:44.473 INFO:teuthology.orchestra.run.smithi131.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 9.0 MB/s | 578 kB 00:00 2024-03-28T18:29:44.498 INFO:teuthology.orchestra.run.smithi131.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 528 kB/s | 46 kB 00:00 2024-03-28T18:29:44.524 INFO:teuthology.orchestra.run.smithi131.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-28T18:29:44.549 INFO:teuthology.orchestra.run.smithi131.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T18:29:44.574 INFO:teuthology.orchestra.run.smithi131.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-28T18:29:44.599 INFO:teuthology.orchestra.run.smithi131.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 459 kB/s | 34 kB 00:00 2024-03-28T18:29:44.650 INFO:teuthology.orchestra.run.smithi196.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 6.8 MB/s | 9.9 MB 00:01 2024-03-28T18:29:44.699 INFO:teuthology.orchestra.run.smithi131.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-03-28T18:29:44.700 INFO:teuthology.orchestra.run.smithi196.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 506 kB/s | 25 kB 00:00 2024-03-28T18:29:44.758 INFO:teuthology.orchestra.run.smithi131.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-28T18:29:44.783 INFO:teuthology.orchestra.run.smithi131.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 429 kB/s | 78 kB 00:00 2024-03-28T18:29:44.892 INFO:teuthology.orchestra.run.smithi131.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-28T18:29:44.909 INFO:teuthology.orchestra.run.smithi131.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-28T18:29:44.925 INFO:teuthology.orchestra.run.smithi131.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 129 kB/s | 18 kB 00:00 2024-03-28T18:29:44.951 INFO:teuthology.orchestra.run.smithi131.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-28T18:29:44.976 INFO:teuthology.orchestra.run.smithi131.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-28T18:29:45.060 INFO:teuthology.orchestra.run.smithi131.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-28T18:29:45.085 INFO:teuthology.orchestra.run.smithi131.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-28T18:29:45.110 INFO:teuthology.orchestra.run.smithi131.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-28T18:29:45.135 INFO:teuthology.orchestra.run.smithi131.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-28T18:29:45.161 INFO:teuthology.orchestra.run.smithi131.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-28T18:29:45.186 INFO:teuthology.orchestra.run.smithi131.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T18:29:45.211 INFO:teuthology.orchestra.run.smithi131.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T18:29:45.228 INFO:teuthology.orchestra.run.smithi131.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-28T18:29:45.253 INFO:teuthology.orchestra.run.smithi131.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 709 kB/s | 47 kB 00:00 2024-03-28T18:29:45.278 INFO:teuthology.orchestra.run.smithi131.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 643 kB/s | 43 kB 00:00 2024-03-28T18:29:45.328 INFO:teuthology.orchestra.run.smithi131.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 515 kB/s | 39 kB 00:00 2024-03-28T18:29:45.454 INFO:teuthology.orchestra.run.smithi131.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 236 kB/s | 41 kB 00:00 2024-03-28T18:29:45.629 INFO:teuthology.orchestra.run.smithi131.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-28T18:29:45.654 INFO:teuthology.orchestra.run.smithi131.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 928 kB/s | 300 kB 00:00 2024-03-28T18:29:45.679 INFO:teuthology.orchestra.run.smithi131.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 252 kB/s | 56 kB 00:00 2024-03-28T18:29:45.705 INFO:teuthology.orchestra.run.smithi131.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-28T18:29:45.730 INFO:teuthology.orchestra.run.smithi131.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-28T18:29:45.755 INFO:teuthology.orchestra.run.smithi131.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T18:29:45.780 INFO:teuthology.orchestra.run.smithi131.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T18:29:45.822 INFO:teuthology.orchestra.run.smithi131.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-28T18:29:45.848 INFO:teuthology.orchestra.run.smithi131.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T18:29:45.873 INFO:teuthology.orchestra.run.smithi131.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-28T18:29:45.898 INFO:teuthology.orchestra.run.smithi131.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-28T18:29:46.173 INFO:teuthology.orchestra.run.smithi131.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 710 kB/s | 195 kB 00:00 2024-03-28T18:29:46.198 INFO:teuthology.orchestra.run.smithi131.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 576 kB/s | 187 kB 00:00 2024-03-28T18:29:46.233 INFO:teuthology.orchestra.run.smithi196.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.7 MB/s | 20 MB 00:07 2024-03-28T18:29:46.265 INFO:teuthology.orchestra.run.smithi131.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-28T18:29:46.357 INFO:teuthology.orchestra.run.smithi131.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-28T18:29:46.384 INFO:teuthology.orchestra.run.smithi196.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.4 MB/s | 220 kB 00:00 2024-03-28T18:29:46.408 INFO:teuthology.orchestra.run.smithi131.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 103 kB/s | 21 kB 00:00 2024-03-28T18:29:46.424 INFO:teuthology.orchestra.run.smithi131.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 160 kB/s | 25 kB 00:00 2024-03-28T18:29:46.450 INFO:teuthology.orchestra.run.smithi196.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 408 kB/s | 27 kB 00:00 2024-03-28T18:29:46.517 INFO:teuthology.orchestra.run.smithi196.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 469 kB/s | 31 kB 00:00 2024-03-28T18:29:46.849 INFO:teuthology.orchestra.run.smithi196.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.2 MB/s | 746 kB 00:00 2024-03-28T18:29:46.867 INFO:teuthology.orchestra.run.smithi131.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 2.0 MB/s | 911 kB 00:00 2024-03-28T18:29:46.933 INFO:teuthology.orchestra.run.smithi131.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-03-28T18:29:46.976 INFO:teuthology.orchestra.run.smithi196.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.3 MB/s | 163 kB 00:00 2024-03-28T18:29:47.035 INFO:teuthology.orchestra.run.smithi196.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 127 kB 00:00 2024-03-28T18:29:47.227 INFO:teuthology.orchestra.run.smithi196.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.6 MB/s | 501 kB 00:00 2024-03-28T18:29:47.284 INFO:teuthology.orchestra.run.smithi131.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 1.8 MB/s | 1.6 MB 00:00 2024-03-28T18:29:47.413 INFO:teuthology.orchestra.run.smithi131.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-28T18:29:47.554 INFO:teuthology.orchestra.run.smithi131.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-03-28T18:29:47.646 INFO:teuthology.orchestra.run.smithi131.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.2 MB/s | 4.1 MB 00:01 2024-03-28T18:29:47.705 INFO:teuthology.orchestra.run.smithi131.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 617 kB/s | 36 kB 00:00 2024-03-28T18:29:47.730 INFO:teuthology.orchestra.run.smithi131.stdout:(101/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-03-28T18:29:47.764 INFO:teuthology.orchestra.run.smithi131.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-03-28T18:29:47.789 INFO:teuthology.orchestra.run.smithi131.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-28T18:29:47.906 INFO:teuthology.orchestra.run.smithi131.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 5.9 MB/s | 5.8 MB 00:00 2024-03-28T18:29:48.199 INFO:teuthology.orchestra.run.smithi131.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-28T18:29:48.290 INFO:teuthology.orchestra.run.smithi131.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-03-28T18:29:48.332 INFO:teuthology.orchestra.run.smithi131.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-03-28T18:29:48.440 INFO:teuthology.orchestra.run.smithi131.stdout:(108/174): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-03-28T18:29:48.652 INFO:teuthology.orchestra.run.smithi196.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.6 MB/s | 5.1 MB 00:01 2024-03-28T18:29:48.724 INFO:teuthology.orchestra.run.smithi131.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-28T18:29:48.727 INFO:teuthology.orchestra.run.smithi196.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 604 kB/s | 45 kB 00:00 2024-03-28T18:29:48.749 INFO:teuthology.orchestra.run.smithi131.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 84 kB/s | 26 kB 00:00 2024-03-28T18:29:48.775 INFO:teuthology.orchestra.run.smithi131.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-03-28T18:29:48.808 INFO:teuthology.orchestra.run.smithi131.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-28T18:29:48.834 INFO:teuthology.orchestra.run.smithi131.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-28T18:29:48.844 INFO:teuthology.orchestra.run.smithi196.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.1 MB/s | 131 kB 00:00 2024-03-28T18:29:48.917 INFO:teuthology.orchestra.run.smithi131.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-28T18:29:48.919 INFO:teuthology.orchestra.run.smithi196.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-28T18:29:48.942 INFO:teuthology.orchestra.run.smithi131.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-28T18:29:48.968 INFO:teuthology.orchestra.run.smithi131.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 886 kB/s | 43 kB 00:00 2024-03-28T18:29:48.993 INFO:teuthology.orchestra.run.smithi131.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-03-28T18:29:49.018 INFO:teuthology.orchestra.run.smithi131.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-28T18:29:49.028 INFO:teuthology.orchestra.run.smithi196.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 321 kB 00:00 2024-03-28T18:29:49.043 INFO:teuthology.orchestra.run.smithi131.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-28T18:29:49.069 INFO:teuthology.orchestra.run.smithi131.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-28T18:29:49.102 INFO:teuthology.orchestra.run.smithi131.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-28T18:29:49.119 INFO:teuthology.orchestra.run.smithi131.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 704 kB/s | 35 kB 00:00 2024-03-28T18:29:49.153 INFO:teuthology.orchestra.run.smithi196.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.3 MB/s | 298 kB 00:00 2024-03-28T18:29:49.161 INFO:teuthology.orchestra.run.smithi131.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-28T18:29:49.287 INFO:teuthology.orchestra.run.smithi131.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-28T18:29:49.312 INFO:teuthology.orchestra.run.smithi131.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 964 kB/s | 144 kB 00:00 2024-03-28T18:29:49.320 INFO:teuthology.orchestra.run.smithi196.stdout:(24/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.2 MB/s | 17 MB 00:07 2024-03-28T18:29:49.337 INFO:teuthology.orchestra.run.smithi131.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-28T18:29:49.345 INFO:teuthology.orchestra.run.smithi196.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 518 kB/s | 99 kB 00:00 2024-03-28T18:29:49.363 INFO:teuthology.orchestra.run.smithi131.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-28T18:29:49.387 INFO:teuthology.orchestra.run.smithi196.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T18:29:49.405 INFO:teuthology.orchestra.run.smithi131.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-28T18:29:49.430 INFO:teuthology.orchestra.run.smithi131.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-03-28T18:29:49.455 INFO:teuthology.orchestra.run.smithi131.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-28T18:29:49.513 INFO:teuthology.orchestra.run.smithi196.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.3 MB/s | 170 kB 00:00 2024-03-28T18:29:49.588 INFO:teuthology.orchestra.run.smithi196.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 453 kB/s | 34 kB 00:00 2024-03-28T18:29:49.655 INFO:teuthology.orchestra.run.smithi196.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-28T18:29:49.706 INFO:teuthology.orchestra.run.smithi131.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 3.7 MB/s | 1.0 MB 00:00 2024-03-28T18:29:49.756 INFO:teuthology.orchestra.run.smithi131.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 154 kB/s | 46 kB 00:00 2024-03-28T18:29:49.798 INFO:teuthology.orchestra.run.smithi131.stdout:(133/174): protobuf-compiler-3.14.0-13.el9.x86_ 430 kB/s | 863 kB 00:02 2024-03-28T18:29:49.823 INFO:teuthology.orchestra.run.smithi131.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-28T18:29:49.848 INFO:teuthology.orchestra.run.smithi131.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-03-28T18:29:49.874 INFO:teuthology.orchestra.run.smithi131.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-03-28T18:29:49.899 INFO:teuthology.orchestra.run.smithi131.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 331 kB/s | 16 kB 00:00 2024-03-28T18:29:49.924 INFO:teuthology.orchestra.run.smithi131.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-28T18:29:49.950 INFO:teuthology.orchestra.run.smithi131.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 372 kB/s | 28 kB 00:00 2024-03-28T18:29:49.975 INFO:teuthology.orchestra.run.smithi131.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-28T18:29:50.000 INFO:teuthology.orchestra.run.smithi131.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-28T18:29:50.025 INFO:teuthology.orchestra.run.smithi131.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-03-28T18:29:50.050 INFO:teuthology.orchestra.run.smithi131.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-03-28T18:29:50.076 INFO:teuthology.orchestra.run.smithi131.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-28T18:29:50.101 INFO:teuthology.orchestra.run.smithi131.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-28T18:29:50.122 INFO:teuthology.orchestra.run.smithi196.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.0 MB/s | 3.1 MB 00:00 2024-03-28T18:29:50.126 INFO:teuthology.orchestra.run.smithi131.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-28T18:29:50.151 INFO:teuthology.orchestra.run.smithi131.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-03-28T18:29:50.177 INFO:teuthology.orchestra.run.smithi131.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-28T18:29:50.191 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T18:29:50.202 INFO:teuthology.orchestra.run.smithi131.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-28T18:29:50.272 INFO:teuthology.orchestra.run.smithi196.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.9 MB/s | 1.8 MB 00:00 2024-03-28T18:29:50.327 INFO:teuthology.orchestra.run.smithi131.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-28T18:29:50.430 INFO:teuthology.orchestra.run.smithi196.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.5 MB/s | 248 kB 00:00 2024-03-28T18:29:50.461 INFO:teuthology.orchestra.run.smithi131.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2024-03-28T18:29:50.506 INFO:teuthology.orchestra.run.smithi196.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 657 kB/s | 49 kB 00:00 2024-03-28T18:29:50.581 INFO:teuthology.orchestra.run.smithi196.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 220 kB/s | 16 kB 00:00 2024-03-28T18:29:50.703 INFO:teuthology.orchestra.run.smithi131.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.3 MB/s | 3.1 MB 00:00 2024-03-28T18:29:50.715 INFO:teuthology.orchestra.run.smithi196.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 264 kB 00:00 2024-03-28T18:29:50.936 INFO:teuthology.orchestra.run.smithi131.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.9 MB/s | 2.3 MB 00:00 2024-03-28T18:29:51.012 INFO:teuthology.orchestra.run.smithi131.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 559 kB/s | 42 kB 00:00 2024-03-28T18:29:51.104 INFO:teuthology.orchestra.run.smithi131.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.6 MB/s | 248 kB 00:00 2024-03-28T18:29:51.163 INFO:teuthology.orchestra.run.smithi131.stdout:(156/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.4 MB/s | 3.4 MB 00:00 2024-03-28T18:29:51.204 INFO:teuthology.orchestra.run.smithi131.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.7 MB/s | 1.8 MB 00:00 2024-03-28T18:29:51.230 INFO:teuthology.orchestra.run.smithi131.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 567 kB/s | 71 kB 00:00 2024-03-28T18:29:51.247 INFO:teuthology.orchestra.run.smithi131.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 435 kB/s | 36 kB 00:00 2024-03-28T18:29:51.272 INFO:teuthology.orchestra.run.smithi131.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-28T18:29:51.307 INFO:teuthology.orchestra.run.smithi196.stdout:(36/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 7.2 MB/s | 48 MB 00:06 2024-03-28T18:29:51.314 INFO:teuthology.orchestra.run.smithi131.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-28T18:29:51.332 INFO:teuthology.orchestra.run.smithi196.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 1.2 MB/s | 748 kB 00:00 2024-03-28T18:29:51.339 INFO:teuthology.orchestra.run.smithi131.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 250 kB/s | 23 kB 00:00 2024-03-28T18:29:51.356 INFO:teuthology.orchestra.run.smithi131.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T18:29:51.382 INFO:teuthology.orchestra.run.smithi131.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-28T18:29:51.391 INFO:teuthology.orchestra.run.smithi196.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 219 kB/s | 18 kB 00:00 2024-03-28T18:29:51.407 INFO:teuthology.orchestra.run.smithi131.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-28T18:29:51.432 INFO:teuthology.orchestra.run.smithi131.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 845 kB/s | 64 kB 00:00 2024-03-28T18:29:51.457 INFO:teuthology.orchestra.run.smithi131.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-28T18:29:51.541 INFO:teuthology.orchestra.run.smithi196.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-28T18:29:51.575 INFO:teuthology.orchestra.run.smithi196.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 920 kB 00:00 2024-03-28T18:29:51.583 INFO:teuthology.orchestra.run.smithi131.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-28T18:29:51.600 INFO:teuthology.orchestra.run.smithi196.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-28T18:29:51.625 INFO:teuthology.orchestra.run.smithi196.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T18:29:51.642 INFO:teuthology.orchestra.run.smithi196.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 765 kB/s | 32 kB 00:00 2024-03-28T18:29:51.666 INFO:teuthology.orchestra.run.smithi131.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.5 MB/s | 465 kB 00:00 2024-03-28T18:29:51.717 INFO:teuthology.orchestra.run.smithi196.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-28T18:29:51.725 INFO:teuthology.orchestra.run.smithi131.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.4 MB/s | 2.2 MB 00:00 2024-03-28T18:29:51.743 INFO:teuthology.orchestra.run.smithi196.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2024-03-28T18:29:51.750 INFO:teuthology.orchestra.run.smithi131.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 372 kB/s | 31 kB 00:00 2024-03-28T18:29:51.771 INFO:teuthology.orchestra.run.smithi196.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 188 kB/s | 10 kB 00:00 2024-03-28T18:29:51.796 INFO:teuthology.orchestra.run.smithi196.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.9 MB/s | 159 kB 00:00 2024-03-28T18:29:51.822 INFO:teuthology.orchestra.run.smithi196.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-28T18:29:51.855 INFO:teuthology.orchestra.run.smithi196.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T18:29:51.859 INFO:teuthology.orchestra.run.smithi131.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-28T18:29:51.881 INFO:teuthology.orchestra.run.smithi196.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-28T18:29:51.884 INFO:teuthology.orchestra.run.smithi131.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-28T18:29:51.906 INFO:teuthology.orchestra.run.smithi196.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 935 kB/s | 46 kB 00:00 2024-03-28T18:29:51.932 INFO:teuthology.orchestra.run.smithi196.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T18:29:51.942 INFO:teuthology.orchestra.run.smithi131.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.7 MB/s | 2.3 MB 00:00 2024-03-28T18:29:51.950 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:29:51.951 INFO:teuthology.orchestra.run.smithi131.stdout:Total 15 MB/s | 225 MB 00:15 2024-03-28T18:29:51.957 INFO:teuthology.orchestra.run.smithi196.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T18:29:51.982 INFO:teuthology.orchestra.run.smithi196.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T18:29:52.007 INFO:teuthology.orchestra.run.smithi196.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-28T18:29:52.108 INFO:teuthology.orchestra.run.smithi196.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T18:29:52.150 INFO:teuthology.orchestra.run.smithi196.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.3 MB/s | 1.0 MB 00:00 2024-03-28T18:29:52.175 INFO:teuthology.orchestra.run.smithi196.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-28T18:29:52.267 INFO:teuthology.orchestra.run.smithi196.stdout:(59/174): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-03-28T18:29:52.351 INFO:teuthology.orchestra.run.smithi196.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-28T18:29:52.376 INFO:teuthology.orchestra.run.smithi196.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 123 kB/s | 24 kB 00:00 2024-03-28T18:29:52.401 INFO:teuthology.orchestra.run.smithi196.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-28T18:29:52.426 INFO:teuthology.orchestra.run.smithi196.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T18:29:52.451 INFO:teuthology.orchestra.run.smithi196.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-28T18:29:52.525 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-03-28T18:29:52.535 INFO:teuthology.orchestra.run.smithi196.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-28T18:29:52.560 INFO:teuthology.orchestra.run.smithi196.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-28T18:29:52.586 INFO:teuthology.orchestra.run.smithi196.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-28T18:29:52.611 INFO:teuthology.orchestra.run.smithi196.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 917 kB/s | 69 kB 00:00 2024-03-28T18:29:52.636 INFO:teuthology.orchestra.run.smithi196.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-28T18:29:52.661 INFO:teuthology.orchestra.run.smithi196.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T18:29:52.678 INFO:teuthology.orchestra.run.smithi196.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-28T18:29:52.704 INFO:teuthology.orchestra.run.smithi196.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T18:29:52.729 INFO:teuthology.orchestra.run.smithi196.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-28T18:29:52.754 INFO:teuthology.orchestra.run.smithi196.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-28T18:29:52.792 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:52.794 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T18:29:52.802 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-03-28T18:29:52.802 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-03-28T18:29:52.816 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T18:29:52.817 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T18:29:52.819 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T18:29:52.820 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:52.823 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T18:29:52.826 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T18:29:52.828 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T18:29:52.915 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-28T18:29:52.937 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-28T18:29:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T18:29:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T18:29:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T18:29:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T18:29:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T18:29:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T18:29:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T18:29:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T18:29:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T18:29:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T18:29:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T18:29:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T18:29:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:Install 150 Packages 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 24 Packages 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 225 M 2024-03-28T18:29:52.950 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-28T18:29:53.038 INFO:teuthology.orchestra.run.smithi196.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-28T18:29:53.063 INFO:teuthology.orchestra.run.smithi196.stdout:(76/174): python3-toml-0.10.2-6.el9.noarch.rpm 126 kB/s | 39 kB 00:00 2024-03-28T18:29:53.088 INFO:teuthology.orchestra.run.smithi196.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 124 kB/s | 41 kB 00:00 2024-03-28T18:29:53.113 INFO:teuthology.orchestra.run.smithi196.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-28T18:29:53.138 INFO:teuthology.orchestra.run.smithi196.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T18:29:53.164 INFO:teuthology.orchestra.run.smithi196.stdout:(80/174): uuid-1.6.2-55.el9.x86_64.rpm 752 kB/s | 56 kB 00:00 2024-03-28T18:29:53.189 INFO:teuthology.orchestra.run.smithi196.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-28T18:29:53.214 INFO:teuthology.orchestra.run.smithi196.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T18:29:53.239 INFO:teuthology.orchestra.run.smithi196.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T18:29:53.273 INFO:teuthology.orchestra.run.smithi196.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T18:29:53.298 INFO:teuthology.orchestra.run.smithi196.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-28T18:29:53.323 INFO:teuthology.orchestra.run.smithi196.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-28T18:29:53.340 INFO:teuthology.orchestra.run.smithi196.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-28T18:29:53.968 INFO:teuthology.orchestra.run.smithi119.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-28T18:29:54.176 INFO:teuthology.orchestra.run.smithi119.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.2 MB/s | 890 kB 00:00 2024-03-28T18:29:54.199 INFO:teuthology.orchestra.run.smithi196.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 228 kB/s | 195 kB 00:00 2024-03-28T18:29:54.224 INFO:teuthology.orchestra.run.smithi196.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 202 kB/s | 187 kB 00:00 2024-03-28T18:29:54.234 INFO:teuthology.orchestra.run.smithi119.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.5 MB/s | 145 kB 00:00 2024-03-28T18:29:54.534 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-03-28T18:29:54.534 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-03-28T18:29:54.568 INFO:teuthology.orchestra.run.smithi119.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 6.2 MB/s | 5.4 MB 00:00 2024-03-28T18:29:54.643 INFO:teuthology.orchestra.run.smithi119.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 5.7 MB/s | 2.3 MB 00:00 2024-03-28T18:29:54.744 INFO:teuthology.orchestra.run.smithi119.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 8.6 MB/s | 1.5 MB 00:00 2024-03-28T18:29:54.816 INFO:teuthology.orchestra.run.smithi196.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-03-28T18:29:54.949 INFO:teuthology.orchestra.run.smithi196.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 499 kB/s | 803 kB 00:01 2024-03-28T18:29:54.975 INFO:teuthology.orchestra.run.smithi196.stdout:(92/174): python3-requests-2.25.1-8.el9.noarch. 161 kB/s | 125 kB 00:00 2024-03-28T18:29:55.228 INFO:teuthology.orchestra.run.smithi119.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 8.1 MB/s | 4.7 MB 00:00 2024-03-28T18:29:55.417 INFO:teuthology.orchestra.run.smithi196.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 42 kB/s | 25 kB 00:00 2024-03-28T18:29:56.411 INFO:teuthology.orchestra.run.smithi119.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 17 MB 00:01 2024-03-28T18:29:56.461 INFO:teuthology.orchestra.run.smithi119.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 512 kB/s | 25 kB 00:00 2024-03-28T18:29:56.620 INFO:teuthology.orchestra.run.smithi119.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 7.1 MB/s | 9.9 MB 00:01 2024-03-28T18:29:56.695 INFO:teuthology.orchestra.run.smithi119.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-28T18:29:56.737 INFO:teuthology.orchestra.run.smithi119.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 660 kB/s | 27 kB 00:00 2024-03-28T18:29:56.979 INFO:teuthology.orchestra.run.smithi119.stdout:(13/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 6.3 MB/s | 20 MB 00:03 2024-03-28T18:29:57.004 INFO:teuthology.orchestra.run.smithi119.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 117 kB/s | 31 kB 00:00 2024-03-28T18:29:57.071 INFO:teuthology.orchestra.run.smithi119.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 163 kB 00:00 2024-03-28T18:29:57.130 INFO:teuthology.orchestra.run.smithi119.stdout:(16/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 746 kB 00:00 2024-03-28T18:29:57.133 INFO:teuthology.orchestra.run.smithi196.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 531 kB/s | 911 kB 00:01 2024-03-28T18:29:57.163 INFO:teuthology.orchestra.run.smithi119.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-28T18:29:57.222 INFO:teuthology.orchestra.run.smithi119.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.4 MB/s | 501 kB 00:00 2024-03-28T18:29:57.314 INFO:teuthology.orchestra.run.smithi119.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 496 kB/s | 45 kB 00:00 2024-03-28T18:29:57.398 INFO:teuthology.orchestra.run.smithi119.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.6 MB/s | 131 kB 00:00 2024-03-28T18:29:57.481 INFO:teuthology.orchestra.run.smithi119.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T18:29:57.573 INFO:teuthology.orchestra.run.smithi119.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.5 MB/s | 321 kB 00:00 2024-03-28T18:29:57.665 INFO:teuthology.orchestra.run.smithi119.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.2 MB/s | 298 kB 00:00 2024-03-28T18:29:57.717 INFO:teuthology.orchestra.run.smithi196.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 593 kB/s | 1.6 MB 00:02 2024-03-28T18:29:57.774 INFO:teuthology.orchestra.run.smithi119.stdout:(24/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 8.4 MB/s | 5.1 MB 00:00 2024-03-28T18:29:57.807 INFO:teuthology.orchestra.run.smithi119.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 703 kB/s | 99 kB 00:00 2024-03-28T18:29:57.824 INFO:teuthology.orchestra.run.smithi119.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T18:29:57.891 INFO:teuthology.orchestra.run.smithi119.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-28T18:29:57.925 INFO:teuthology.orchestra.run.smithi196.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 168 kB/s | 133 kB 00:00 2024-03-28T18:29:57.950 INFO:teuthology.orchestra.run.smithi119.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 592 kB/s | 34 kB 00:00 2024-03-28T18:29:58.017 INFO:teuthology.orchestra.run.smithi119.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.1 MB/s | 139 kB 00:00 2024-03-28T18:29:58.217 INFO:teuthology.orchestra.run.smithi119.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 7.6 MB/s | 3.1 MB 00:00 2024-03-28T18:29:58.559 INFO:teuthology.orchestra.run.smithi119.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.3 MB/s | 1.8 MB 00:00 2024-03-28T18:29:58.634 INFO:teuthology.orchestra.run.smithi119.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.3 MB/s | 248 kB 00:00 2024-03-28T18:29:58.701 INFO:teuthology.orchestra.run.smithi119.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 750 kB/s | 49 kB 00:00 2024-03-28T18:29:58.768 INFO:teuthology.orchestra.run.smithi119.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 250 kB/s | 16 kB 00:00 2024-03-28T18:29:58.834 INFO:teuthology.orchestra.run.smithi196.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 270 kB/s | 245 kB 00:00 2024-03-28T18:29:58.860 INFO:teuthology.orchestra.run.smithi119.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.8 MB/s | 264 kB 00:00 2024-03-28T18:29:59.127 INFO:teuthology.orchestra.run.smithi119.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.1 MB/s | 7.4 MB 00:00 2024-03-28T18:29:59.169 INFO:teuthology.orchestra.run.smithi119.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 2.4 MB/s | 748 kB 00:00 2024-03-28T18:29:59.203 INFO:teuthology.orchestra.run.smithi119.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 244 kB/s | 18 kB 00:00 2024-03-28T18:29:59.420 INFO:teuthology.orchestra.run.smithi119.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-28T18:29:59.462 INFO:teuthology.orchestra.run.smithi119.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-28T18:29:59.487 INFO:teuthology.orchestra.run.smithi119.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T18:29:59.521 INFO:teuthology.orchestra.run.smithi119.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-28T18:29:59.546 INFO:teuthology.orchestra.run.smithi119.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 564 kB/s | 32 kB 00:00 2024-03-28T18:29:59.646 INFO:teuthology.orchestra.run.smithi119.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-28T18:29:59.671 INFO:teuthology.orchestra.run.smithi119.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 148 kB/s | 17 kB 00:00 2024-03-28T18:29:59.705 INFO:teuthology.orchestra.run.smithi119.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 312 kB/s | 10 kB 00:00 2024-03-28T18:29:59.730 INFO:teuthology.orchestra.run.smithi119.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-28T18:29:59.742 INFO:teuthology.orchestra.run.smithi196.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 273 kB/s | 248 kB 00:00 2024-03-28T18:29:59.756 INFO:teuthology.orchestra.run.smithi119.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 917 kB/s | 44 kB 00:00 2024-03-28T18:29:59.798 INFO:teuthology.orchestra.run.smithi119.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-28T18:29:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-03-28T18:29:59.865 INFO:teuthology.orchestra.run.smithi119.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 756 kB/s | 46 kB 00:00 2024-03-28T18:29:59.899 INFO:teuthology.orchestra.run.smithi119.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-28T18:29:59.932 INFO:teuthology.orchestra.run.smithi119.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-28T18:29:59.982 INFO:teuthology.orchestra.run.smithi119.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-28T18:30:00.008 INFO:teuthology.orchestra.run.smithi119.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 470 kB/s | 34 kB 00:00 2024-03-28T18:30:00.142 INFO:teuthology.orchestra.run.smithi119.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-28T18:30:00.192 INFO:teuthology.orchestra.run.smithi119.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-28T18:30:00.225 INFO:teuthology.orchestra.run.smithi119.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.4 MB/s | 78 kB 00:00 2024-03-28T18:30:00.359 INFO:teuthology.orchestra.run.smithi119.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-28T18:30:00.384 INFO:teuthology.orchestra.run.smithi119.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 169 kB/s | 24 kB 00:00 2024-03-28T18:30:00.426 INFO:teuthology.orchestra.run.smithi119.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 446 kB/s | 18 kB 00:00 2024-03-28T18:30:00.451 INFO:teuthology.orchestra.run.smithi119.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-28T18:30:00.485 INFO:teuthology.orchestra.run.smithi119.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 979 kB/s | 32 kB 00:00 2024-03-28T18:30:00.602 INFO:teuthology.orchestra.run.smithi119.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-28T18:30:00.627 INFO:teuthology.orchestra.run.smithi119.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-28T18:30:00.669 INFO:teuthology.orchestra.run.smithi119.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-28T18:30:00.694 INFO:teuthology.orchestra.run.smithi119.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-28T18:30:00.695 INFO:teuthology.orchestra.run.smithi196.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 232 kB/s | 221 kB 00:00 2024-03-28T18:30:00.744 INFO:teuthology.orchestra.run.smithi119.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 836 kB/s | 41 kB 00:00 2024-03-28T18:30:00.770 INFO:teuthology.orchestra.run.smithi119.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T18:30:01.001 INFO:teuthology.orchestra.run.smithi196.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 699 kB/s | 4.1 MB 00:06 2024-03-28T18:30:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:(70/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.9 MB/s | 48 MB 00:04 2024-03-28T18:30:01.293 INFO:teuthology.orchestra.run.smithi196.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 60 kB/s | 36 kB 00:00 2024-03-28T18:30:01.295 INFO:teuthology.orchestra.run.smithi119.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 252 kB/s | 132 kB 00:00 2024-03-28T18:30:01.320 INFO:teuthology.orchestra.run.smithi119.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 383 kB/s | 211 kB 00:00 2024-03-28T18:30:01.345 INFO:teuthology.orchestra.run.smithi119.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 593 kB/s | 47 kB 00:00 2024-03-28T18:30:01.387 INFO:teuthology.orchestra.run.smithi119.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 653 kB/s | 43 kB 00:00 2024-03-28T18:30:01.529 INFO:teuthology.orchestra.run.smithi119.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 228 kB/s | 39 kB 00:00 2024-03-28T18:30:01.622 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-03-28T18:30:01.644 INFO:teuthology.orchestra.run.smithi196.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-03-28T18:30:01.738 INFO:teuthology.orchestra.run.smithi196.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 241 kB/s | 22 kB 00:00 2024-03-28T18:30:01.738 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:30:01.739 INFO:teuthology.orchestra.run.smithi119.stdout:(76/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-28T18:30:01.772 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:30:01.780 INFO:teuthology.orchestra.run.smithi119.stdout:(77/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 108 kB/s | 41 kB 00:00 2024-03-28T18:30:01.812 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:01.813 INFO:teuthology.orchestra.run.smithi119.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-28T18:30:01.821 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:01.839 INFO:teuthology.orchestra.run.smithi119.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-28T18:30:01.864 INFO:teuthology.orchestra.run.smithi119.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 784 kB/s | 63 kB 00:00 2024-03-28T18:30:01.869 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:30:01.889 INFO:teuthology.orchestra.run.smithi119.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 598 kB/s | 44 kB 00:00 2024-03-28T18:30:01.906 INFO:teuthology.orchestra.run.smithi119.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-28T18:30:01.923 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:30:01.930 INFO:teuthology.orchestra.run.smithi196.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-03-28T18:30:01.931 INFO:teuthology.orchestra.run.smithi119.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-28T18:30:01.956 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:30:01.982 INFO:teuthology.orchestra.run.smithi119.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-28T18:30:01.990 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:30:01.998 INFO:teuthology.orchestra.run.smithi119.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 124 kB 00:00 2024-03-28T18:30:02.024 INFO:teuthology.orchestra.run.smithi119.stdout:(86/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-28T18:30:02.032 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:30:02.041 INFO:teuthology.orchestra.run.smithi119.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-28T18:30:02.063 INFO:teuthology.orchestra.run.smithi196.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 122 kB/s | 94 kB 00:00 2024-03-28T18:30:02.166 INFO:teuthology.orchestra.run.smithi119.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-28T18:30:02.191 INFO:teuthology.orchestra.run.smithi119.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-28T18:30:02.225 INFO:teuthology.orchestra.run.smithi119.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-28T18:30:02.300 INFO:teuthology.orchestra.run.smithi119.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-28T18:30:02.325 INFO:teuthology.orchestra.run.smithi119.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 162 kB/s | 21 kB 00:00 2024-03-28T18:30:02.351 INFO:teuthology.orchestra.run.smithi119.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 203 kB/s | 25 kB 00:00 2024-03-28T18:30:02.352 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:30:02.372 INFO:teuthology.orchestra.run.smithi196.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-28T18:30:02.408 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:02.439 INFO:teuthology.orchestra.run.smithi196.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 825 kB/s | 309 kB 00:00 2024-03-28T18:30:02.464 INFO:teuthology.orchestra.run.smithi196.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-03-28T18:30:02.530 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:02.539 INFO:teuthology.orchestra.run.smithi196.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-28T18:30:02.551 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:02.568 INFO:teuthology.orchestra.run.smithi119.stdout:(94/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-28T18:30:02.615 INFO:teuthology.orchestra.run.smithi196.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-03-28T18:30:02.626 INFO:teuthology.orchestra.run.smithi119.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-03-28T18:30:02.641 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:02.672 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:30:02.685 INFO:teuthology.orchestra.run.smithi119.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-03-28T18:30:02.690 INFO:teuthology.orchestra.run.smithi196.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-03-28T18:30:02.713 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:30:02.735 INFO:teuthology.orchestra.run.smithi119.stdout:(97/174): postgresql-13.14-1.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-03-28T18:30:02.757 INFO:teuthology.orchestra.run.smithi196.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-28T18:30:02.769 INFO:teuthology.orchestra.run.smithi119.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-28T18:30:02.782 INFO:teuthology.orchestra.run.smithi196.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-28T18:30:02.803 INFO:teuthology.orchestra.run.smithi119.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-03-28T18:30:02.836 INFO:teuthology.orchestra.run.smithi119.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-03-28T18:30:02.857 INFO:teuthology.orchestra.run.smithi196.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-28T18:30:02.861 INFO:teuthology.orchestra.run.smithi119.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 623 kB/s | 36 kB 00:00 2024-03-28T18:30:02.887 INFO:teuthology.orchestra.run.smithi119.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 958 kB/s | 47 kB 00:00 2024-03-28T18:30:02.891 INFO:teuthology.orchestra.run.smithi196.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-03-28T18:30:02.894 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:30:02.904 INFO:teuthology.orchestra.run.smithi119.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-28T18:30:02.925 INFO:teuthology.orchestra.run.smithi196.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-03-28T18:30:02.946 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:30:02.958 INFO:teuthology.orchestra.run.smithi196.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-03-28T18:30:02.973 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:02.992 INFO:teuthology.orchestra.run.smithi196.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-28T18:30:03.042 INFO:teuthology.orchestra.run.smithi196.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-28T18:30:03.068 INFO:teuthology.orchestra.run.smithi196.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-28T18:30:03.090 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:03.121 INFO:teuthology.orchestra.run.smithi119.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 12 MB/s | 5.8 MB 00:00 2024-03-28T18:30:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:03.126 INFO:teuthology.orchestra.run.smithi196.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-28T18:30:03.143 INFO:teuthology.orchestra.run.smithi196.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 470 kB/s | 35 kB 00:00 2024-03-28T18:30:03.188 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:03.210 INFO:teuthology.orchestra.run.smithi196.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-28T18:30:03.228 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:30:03.251 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:03.260 INFO:teuthology.orchestra.run.smithi196.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-28T18:30:03.267 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:03.294 INFO:teuthology.orchestra.run.smithi196.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-28T18:30:03.327 INFO:teuthology.orchestra.run.smithi196.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-28T18:30:03.353 INFO:teuthology.orchestra.run.smithi196.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-28T18:30:03.395 INFO:teuthology.orchestra.run.smithi196.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 345 kB/s | 23 kB 00:00 2024-03-28T18:30:03.412 INFO:teuthology.orchestra.run.smithi196.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-03-28T18:30:03.453 INFO:teuthology.orchestra.run.smithi196.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-28T18:30:03.459 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:03.491 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:03.520 INFO:teuthology.orchestra.run.smithi196.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 695 kB/s | 46 kB 00:00 2024-03-28T18:30:03.531 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:30:03.554 INFO:teuthology.orchestra.run.smithi196.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-28T18:30:03.571 INFO:teuthology.orchestra.run.smithi119.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-28T18:30:03.588 INFO:teuthology.orchestra.run.smithi119.stdout:(106/174): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-03-28T18:30:03.596 INFO:teuthology.orchestra.run.smithi196.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-28T18:30:03.621 INFO:teuthology.orchestra.run.smithi196.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 860 kB/s | 58 kB 00:00 2024-03-28T18:30:03.629 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:30:03.680 INFO:teuthology.orchestra.run.smithi196.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-28T18:30:03.688 INFO:teuthology.orchestra.run.smithi119.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 196 kB/s | 19 kB 00:00 2024-03-28T18:30:03.705 INFO:teuthology.orchestra.run.smithi196.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-28T18:30:03.721 INFO:teuthology.orchestra.run.smithi119.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-03-28T18:30:03.746 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:03.747 INFO:teuthology.orchestra.run.smithi196.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-28T18:30:03.772 INFO:teuthology.orchestra.run.smithi119.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 529 kB/s | 26 kB 00:00 2024-03-28T18:30:03.773 INFO:teuthology.orchestra.run.smithi196.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 421 kB/s | 28 kB 00:00 2024-03-28T18:30:03.793 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:03.823 INFO:teuthology.orchestra.run.smithi196.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-03-28T18:30:03.830 INFO:teuthology.orchestra.run.smithi119.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 843 kB/s | 49 kB 00:00 2024-03-28T18:30:03.840 INFO:teuthology.orchestra.run.smithi196.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-28T18:30:03.866 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:03.889 INFO:teuthology.orchestra.run.smithi119.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-28T18:30:03.898 INFO:teuthology.orchestra.run.smithi196.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-28T18:30:03.911 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:03.924 INFO:teuthology.orchestra.run.smithi196.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-28T18:30:03.982 INFO:teuthology.orchestra.run.smithi196.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 611 kB/s | 51 kB 00:00 2024-03-28T18:30:03.989 INFO:teuthology.orchestra.run.smithi119.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-28T18:30:04.008 INFO:teuthology.orchestra.run.smithi196.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-28T18:30:04.023 INFO:teuthology.orchestra.run.smithi119.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-28T18:30:04.050 INFO:teuthology.orchestra.run.smithi196.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-28T18:30:04.073 INFO:teuthology.orchestra.run.smithi119.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 838 kB 00:00 2024-03-28T18:30:04.091 INFO:teuthology.orchestra.run.smithi196.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-28T18:30:04.115 INFO:teuthology.orchestra.run.smithi119.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-28T18:30:04.117 INFO:teuthology.orchestra.run.smithi196.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-03-28T18:30:04.118 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:04.182 INFO:teuthology.orchestra.run.smithi119.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-28T18:30:04.192 INFO:teuthology.orchestra.run.smithi196.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-28T18:30:04.207 INFO:teuthology.orchestra.run.smithi119.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 353 kB/s | 32 kB 00:00 2024-03-28T18:30:04.224 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:30:04.234 INFO:teuthology.orchestra.run.smithi196.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-28T18:30:04.241 INFO:teuthology.orchestra.run.smithi119.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 259 kB/s | 15 kB 00:00 2024-03-28T18:30:04.275 INFO:teuthology.orchestra.run.smithi119.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-28T18:30:04.310 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:30:04.341 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:30:04.368 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:30:04.375 INFO:teuthology.orchestra.run.smithi119.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-28T18:30:04.403 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:30:04.408 INFO:teuthology.orchestra.run.smithi119.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-28T18:30:04.475 INFO:teuthology.orchestra.run.smithi119.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 355 kB/s | 35 kB 00:00 2024-03-28T18:30:04.477 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:30:04.509 INFO:teuthology.orchestra.run.smithi119.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-28T18:30:04.531 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:30:04.570 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:30:04.609 INFO:teuthology.orchestra.run.smithi119.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-28T18:30:04.633 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:30:04.659 INFO:teuthology.orchestra.run.smithi119.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 216 kB/s | 11 kB 00:00 2024-03-28T18:30:04.669 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:30:04.702 INFO:teuthology.orchestra.run.smithi119.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 431 kB/s | 18 kB 00:00 2024-03-28T18:30:04.713 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:30:04.743 INFO:teuthology.orchestra.run.smithi119.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 566 kB/s | 23 kB 00:00 2024-03-28T18:30:04.745 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:30:04.773 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:30:04.776 INFO:teuthology.orchestra.run.smithi196.stdout:(150/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-03-28T18:30:04.811 INFO:teuthology.orchestra.run.smithi119.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.1 MB/s | 2.0 MB 00:00 2024-03-28T18:30:04.818 INFO:teuthology.orchestra.run.smithi196.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-28T18:30:04.842 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:30:04.852 INFO:teuthology.orchestra.run.smithi119.stdout:(129/174): protobuf-compiler-3.14.0-13.el9.x86_ 443 kB/s | 863 kB 00:01 2024-03-28T18:30:04.878 INFO:teuthology.orchestra.run.smithi119.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-03-28T18:30:04.887 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:30:04.903 INFO:teuthology.orchestra.run.smithi119.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-03-28T18:30:04.930 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:30:04.953 INFO:teuthology.orchestra.run.smithi119.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-03-28T18:30:04.959 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:30:04.995 INFO:teuthology.orchestra.run.smithi119.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 8.8 MB/s | 1.0 MB 00:00 2024-03-28T18:30:05.020 INFO:teuthology.orchestra.run.smithi119.stdout:(134/174): python3-logutils-0.3.5-21.el9.noarch 330 kB/s | 46 kB 00:00 2024-03-28T18:30:05.026 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:30:05.046 INFO:teuthology.orchestra.run.smithi119.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 631 kB/s | 58 kB 00:00 2024-03-28T18:30:05.071 INFO:teuthology.orchestra.run.smithi119.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 331 kB/s | 16 kB 00:00 2024-03-28T18:30:05.073 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:30:05.096 INFO:teuthology.orchestra.run.smithi119.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-28T18:30:05.118 INFO:teuthology.orchestra.run.smithi196.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.7 MB/s | 3.4 MB 00:00 2024-03-28T18:30:05.130 INFO:teuthology.orchestra.run.smithi119.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-28T18:30:05.145 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:30:05.156 INFO:teuthology.orchestra.run.smithi119.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 340 kB/s | 28 kB 00:00 2024-03-28T18:30:05.180 INFO:teuthology.orchestra.run.smithi119.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-03-28T18:30:05.214 INFO:teuthology.orchestra.run.smithi119.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-28T18:30:05.239 INFO:teuthology.orchestra.run.smithi119.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 623 kB/s | 36 kB 00:00 2024-03-28T18:30:05.265 INFO:teuthology.orchestra.run.smithi119.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 713 kB/s | 59 kB 00:00 2024-03-28T18:30:05.279 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:30:05.290 INFO:teuthology.orchestra.run.smithi119.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 682 kB/s | 51 kB 00:00 2024-03-28T18:30:05.334 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:30:05.341 INFO:teuthology.orchestra.run.smithi119.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-28T18:30:05.374 INFO:teuthology.orchestra.run.smithi119.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 830 kB/s | 90 kB 00:00 2024-03-28T18:30:05.391 INFO:teuthology.orchestra.run.smithi119.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 405 kB/s | 20 kB 00:00 2024-03-28T18:30:05.425 INFO:teuthology.orchestra.run.smithi119.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-28T18:30:05.458 INFO:teuthology.orchestra.run.smithi119.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-03-28T18:30:05.608 INFO:teuthology.orchestra.run.smithi119.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-28T18:30:05.659 INFO:teuthology.orchestra.run.smithi119.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.7 MB/s | 321 kB 00:00 2024-03-28T18:30:05.794 INFO:teuthology.orchestra.run.smithi196.stdout:(153/174): postgresql-server-13.14-1.el9.x86_64 733 kB/s | 5.8 MB 00:08 2024-03-28T18:30:05.809 INFO:teuthology.orchestra.run.smithi119.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 8.8 MB/s | 3.4 MB 00:00 2024-03-28T18:30:05.851 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:30:05.937 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:30:05.976 INFO:teuthology.orchestra.run.smithi119.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 7.3 MB/s | 2.3 MB 00:00 2024-03-28T18:30:06.023 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:30:06.034 INFO:teuthology.orchestra.run.smithi119.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 734 kB/s | 42 kB 00:00 2024-03-28T18:30:06.083 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:06.085 INFO:teuthology.orchestra.run.smithi119.stdout:(155/174): NetworkManager-libnm-1.47.2-1.el9.x8 6.7 MB/s | 1.8 MB 00:00 2024-03-28T18:30:06.093 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:06.118 INFO:teuthology.orchestra.run.smithi119.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T18:30:06.123 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:30:06.143 INFO:teuthology.orchestra.run.smithi119.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-28T18:30:06.164 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:30:06.169 INFO:teuthology.orchestra.run.smithi119.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 732 kB/s | 36 kB 00:00 2024-03-28T18:30:06.202 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:30:06.210 INFO:teuthology.orchestra.run.smithi119.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-28T18:30:06.231 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:30:06.294 INFO:teuthology.orchestra.run.smithi119.stdout:(160/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 3.8 MB/s | 3.1 MB 00:00 2024-03-28T18:30:06.319 INFO:teuthology.orchestra.run.smithi119.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 213 kB/s | 23 kB 00:00 2024-03-28T18:30:06.353 INFO:teuthology.orchestra.run.smithi119.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-03-28T18:30:06.404 INFO:teuthology.orchestra.run.smithi119.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-03-28T18:30:06.429 INFO:teuthology.orchestra.run.smithi119.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-28T18:30:06.454 INFO:teuthology.orchestra.run.smithi119.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.3 MB/s | 64 kB 00:00 2024-03-28T18:30:06.456 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:30:06.511 INFO:teuthology.orchestra.run.smithi196.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 59 kB/s | 42 kB 00:00 2024-03-28T18:30:06.565 INFO:teuthology.orchestra.run.smithi119.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 212 kB/s | 29 kB 00:00 2024-03-28T18:30:06.596 INFO:teuthology.orchestra.run.smithi119.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 560 kB/s | 136 kB 00:00 2024-03-28T18:30:06.619 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:30:06.663 INFO:teuthology.orchestra.run.smithi119.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.8 MB/s | 1.2 MB 00:00 2024-03-28T18:30:06.693 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:30:06.736 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:30:06.747 INFO:teuthology.orchestra.run.smithi119.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.5 MB/s | 465 kB 00:00 2024-03-28T18:30:06.797 INFO:teuthology.orchestra.run.smithi119.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 633 kB/s | 31 kB 00:00 2024-03-28T18:30:06.876 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:30:06.881 INFO:teuthology.orchestra.run.smithi119.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-28T18:30:06.956 INFO:teuthology.orchestra.run.smithi119.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.4 MB/s | 2.3 MB 00:00 2024-03-28T18:30:06.965 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:30:06.998 INFO:teuthology.orchestra.run.smithi119.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-28T18:30:07.023 INFO:teuthology.orchestra.run.smithi119.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 527 kB/s | 74 kB 00:00 2024-03-28T18:30:07.032 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:30:07.032 INFO:teuthology.orchestra.run.smithi119.stdout:Total 16 MB/s | 225 MB 00:14 2024-03-28T18:30:07.160 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:30:07.371 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:30:07.403 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:30:07.469 INFO:teuthology.orchestra.run.smithi196.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 260 kB/s | 248 kB 00:00 2024-03-28T18:30:07.513 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:30:07.688 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:30:07.723 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:30:07.766 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:30:07.809 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:30:07.940 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:30:07.997 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:30:08.027 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:30:08.068 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:30:08.152 INFO:teuthology.orchestra.run.smithi196.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 618 kB/s | 1.8 MB 00:03 2024-03-28T18:30:08.160 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:30:08.178 INFO:teuthology.orchestra.run.smithi196.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 101 kB/s | 71 kB 00:00 2024-03-28T18:30:08.202 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:30:08.241 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:30:08.268 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:30:08.292 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:30:08.364 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:30:08.411 INFO:teuthology.orchestra.run.smithi196.stdout:(158/174): NetworkManager-1.47.2-1.el9.x86_64.r 657 kB/s | 2.3 MB 00:03 2024-03-28T18:30:08.484 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:30:08.582 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:30:08.626 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:30:08.656 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-28T18:30:08.674 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:30:08.729 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:30:08.819 INFO:teuthology.orchestra.run.smithi196.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 55 kB/s | 36 kB 00:00 2024-03-28T18:30:08.911 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:30:08.964 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:30:09.036 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-28T18:30:09.036 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-28T18:30:09.103 INFO:teuthology.orchestra.run.smithi196.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 208 kB/s | 192 kB 00:00 2024-03-28T18:30:09.108 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:30:09.145 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:30:09.177 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:30:09.208 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:30:09.236 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:30:09.271 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:30:09.318 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:30:09.349 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:30:09.380 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:30:09.454 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:30:09.470 INFO:teuthology.orchestra.run.smithi196.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 254 kB/s | 256 kB 00:01 2024-03-28T18:30:09.487 INFO:teuthology.orchestra.run.smithi196.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 35 kB/s | 23 kB 00:00 2024-03-28T18:30:09.490 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:30:09.523 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:30:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:30:09.599 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:30:09.688 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:30:09.779 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:30:09.822 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:30:09.862 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:30:09.909 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:30:09.912 INFO:teuthology.orchestra.run.smithi196.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 139 kB/s | 108 kB 00:00 2024-03-28T18:30:10.006 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:30:10.055 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:30:10.279 INFO:teuthology.orchestra.run.smithi196.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 171 kB/s | 136 kB 00:00 2024-03-28T18:30:10.305 INFO:teuthology.orchestra.run.smithi196.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 186 kB/s | 155 kB 00:00 2024-03-28T18:30:10.580 INFO:teuthology.orchestra.run.smithi196.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 96 kB/s | 64 kB 00:00 2024-03-28T18:30:10.771 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:30:10.819 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:30:10.853 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:30:10.855 INFO:teuthology.orchestra.run.smithi196.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 50 kB/s | 29 kB 00:00 2024-03-28T18:30:10.901 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:30:10.980 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:30:11.021 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:30:11.196 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:30:11.226 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:30:11.304 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:30:11.354 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-28T18:30:11.354 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-28T18:30:11.936 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:30:12.056 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:30:12.447 INFO:teuthology.orchestra.run.smithi196.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 579 kB/s | 1.2 MB 00:02 2024-03-28T18:30:13.260 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:30:13.345 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:30:13.375 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:30:13.407 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:30:13.434 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:30:13.748 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:30:13.773 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:13.839 INFO:teuthology.orchestra.run.smithi196.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 334 kB/s | 465 kB 00:01 2024-03-28T18:30:13.871 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:13.905 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:30:13.948 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:30:14.014 INFO:teuthology.orchestra.run.smithi196.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 643 kB/s | 2.2 MB 00:03 2024-03-28T18:30:14.389 INFO:teuthology.orchestra.run.smithi196.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 662 kB/s | 2.3 MB 00:03 2024-03-28T18:30:14.416 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:30:14.451 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:14.506 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:14.523 INFO:teuthology.orchestra.run.smithi196.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 46 kB/s | 31 kB 00:00 2024-03-28T18:30:14.536 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:30:15.098 INFO:teuthology.orchestra.run.smithi196.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 105 kB/s | 74 kB 00:00 2024-03-28T18:30:16.014 INFO:teuthology.orchestra.run.smithi196.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 550 kB/s | 1.1 MB 00:01 2024-03-28T18:30:16.020 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:30:16.020 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.9 MB/s | 225 MB 00:37 2024-03-28T18:30:16.608 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-28T18:30:16.674 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:16.708 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:16.878 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-28T18:30:16.878 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-28T18:30:17.244 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:17.278 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:17.351 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:17.351 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:30:17.351 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:30:17.351 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:17.379 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:18.683 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-28T18:30:18.683 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-28T18:30:18.835 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-28T18:30:18.999 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:30:19.071 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:30:19.134 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:19.179 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:19.307 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:30:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:30:19.484 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:30:19.516 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:30:19.793 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:30:20.223 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:30:20.264 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:20.405 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:20.432 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:20.546 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:20.613 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:30:20.680 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:30:20.987 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:30:21.041 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:30:21.066 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:21.214 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:21.242 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:21.332 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:21.385 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:30:21.411 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:21.471 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:21.759 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:21.860 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:30:22.185 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:30:22.414 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:22.507 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:22.696 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:22.770 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:23.139 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:23.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:30:23.446 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:30:23.485 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:30:23.521 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:30:23.578 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:30:23.690 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:30:23.758 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:30:23.800 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:30:23.889 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:30:23.931 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:30:23.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:30:24.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:30:24.068 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:30:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:30:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:30:24.281 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:30:24.312 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:30:24.429 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:30:24.488 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:30:24.586 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:30:24.758 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:30:24.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:30:25.471 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:30:25.566 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-28T18:30:25.572 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:30:25.675 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:30:25.676 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T18:30:25.710 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T18:30:25.732 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:25.741 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:25.750 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T18:30:25.759 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:25.790 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T18:30:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:30:25.844 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T18:30:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:30:25.878 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T18:30:25.912 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T18:30:25.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:30:25.954 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T18:30:25.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:30:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T18:30:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:26.397 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:30:26.451 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:26.480 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:26.572 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:26.601 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:30:26.622 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:30:26.638 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T18:30:26.744 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:30:26.802 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T18:30:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:30:26.866 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T18:30:26.894 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:27.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:30:27.082 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:30:27.084 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T18:30:27.132 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T18:30:27.154 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:30:27.155 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:27.172 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T18:30:27.241 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:30:27.299 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:30:27.313 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:27.338 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:30:27.344 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T18:30:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T18:30:27.407 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:30:27.455 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:30:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T18:30:27.518 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:30:27.570 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:27.586 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:30:27.630 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T18:30:27.701 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:27.756 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:27.776 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:30:27.828 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:30:27.864 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:30:27.912 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:30:27.963 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T18:30:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:30:28.070 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T18:30:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:30:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:30:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:30:28.166 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T18:30:28.178 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:30:28.211 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T18:30:28.238 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T18:30:28.274 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T18:30:28.327 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:30:28.365 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T18:30:28.418 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T18:30:28.457 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T18:30:28.501 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:30:28.520 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T18:30:28.564 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T18:30:28.608 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T18:30:28.635 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:30:28.640 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T18:30:28.668 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T18:30:28.689 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:30:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T18:30:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:30:28.773 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T18:30:28.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T18:30:28.834 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:30:28.845 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T18:30:28.914 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T18:30:28.962 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T18:30:29.034 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T18:30:29.077 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:30:29.136 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:30:29.175 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T18:30:29.228 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T18:30:29.311 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:30:29.341 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /sys 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /proc 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /mnt 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /var/tmp 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /home 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /root 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /tmp 2024-03-28T18:30:29.342 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:29.358 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:30:29.397 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:30:29.444 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:30:29.485 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:30:29.513 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:29.529 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:29.529 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:29.529 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:30:29.529 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:29.531 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:30:29.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:30:29.657 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:30:29.750 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T18:30:29.822 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:30:29.824 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T18:30:29.876 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:30:29.911 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:30:29.912 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T18:30:29.963 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:30:29.969 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:29.979 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T18:30:30.009 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T18:30:30.015 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:30:30.059 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T18:30:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T18:30:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T18:30:30.159 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:30:30.271 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:30:30.327 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:30:30.363 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:30:30.365 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T18:30:30.421 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:30:30.520 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T18:30:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:30:30.579 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T18:30:30.614 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T18:30:30.645 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:30:30.741 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T18:30:30.800 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T18:30:30.855 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T18:30:30.930 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T18:30:30.962 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T18:30:30.998 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T18:30:31.017 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:31.042 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:31.045 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T18:30:31.081 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T18:30:31.124 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T18:30:31.166 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T18:30:31.267 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:31.289 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:31.299 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T18:30:31.347 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T18:30:31.377 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T18:30:31.419 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T18:30:31.513 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T18:30:31.560 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T18:30:31.600 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T18:30:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:30:31.626 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T18:30:31.650 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T18:30:31.661 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:30:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:30:31.723 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T18:30:31.734 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:31.753 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:31.757 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:31.845 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:30:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T18:30:31.863 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:30:31.912 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:30:31.960 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T18:30:32.001 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T18:30:32.050 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T18:30:32.105 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T18:30:32.143 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:30:32.175 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:30:32.221 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:32.252 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:32.271 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:30:32.284 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:30:32.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T18:30:32.346 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T18:30:32.352 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:30:32.375 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:32.476 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T18:30:32.511 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T18:30:32.543 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T18:30:32.574 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T18:30:32.607 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:32.611 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T18:30:32.643 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:32.645 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T18:30:32.684 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T18:30:32.716 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T18:30:32.745 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T18:30:32.823 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T18:30:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T18:30:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T18:30:32.964 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T18:30:32.990 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T18:30:33.056 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:30:33.080 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T18:30:33.170 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T18:30:33.209 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:30:33.221 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T18:30:33.253 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T18:30:33.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T18:30:33.398 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T18:30:33.490 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T18:30:33.604 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:33.634 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:33.733 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:33.888 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:30:33.920 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:33.943 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:33.987 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:33.998 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:34.044 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:30:34.337 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T18:30:34.376 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T18:30:34.410 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T18:30:34.458 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T18:30:34.545 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T18:30:34.603 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T18:30:34.711 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:30:34.779 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T18:30:34.809 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T18:30:34.821 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:30:34.859 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:30:34.886 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T18:30:34.890 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:30:34.924 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:30:34.956 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:30:34.957 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:30:34.978 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:30:34.979 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:34.979 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:30:34.979 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:30:34.979 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:30:34.979 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:35.049 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:30:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:30:35.070 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:35.070 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:30:35.070 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:35.116 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:30:35.138 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:30:35.138 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:35.138 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:30:35.138 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:30:35.139 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:30:35.139 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:35.360 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:30:35.395 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:35.442 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:30:35.464 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:35.517 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:35.520 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T18:30:35.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:30:35.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:30:35.639 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T18:30:36.152 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:30:36.190 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:36.290 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:36.316 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:30:36.866 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T18:30:36.942 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T18:30:36.974 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T18:30:37.005 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T18:30:37.032 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T18:30:37.343 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T18:30:37.371 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:37.462 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T18:30:37.492 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T18:30:37.529 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T18:30:37.994 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T18:30:38.024 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:38.078 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T18:30:38.109 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T18:30:38.962 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:39.222 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:39.860 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:39.897 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:30:39.908 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:39.937 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:30:39.968 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:30:40.003 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:30:40.065 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:40.066 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:30:40.066 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:30:40.066 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:40.100 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:40.181 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:30:40.303 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:30:40.336 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:40.343 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:30:40.373 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:40.382 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:30:40.436 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:30:40.487 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:30:40.520 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:30:40.552 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:30:40.582 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:30:40.636 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:30:40.688 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:30:40.714 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:30:40.909 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T18:30:40.952 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T18:30:41.024 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T18:30:41.024 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T18:30:41.024 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:41.052 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:41.334 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:30:41.370 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:30:41.390 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:30:42.201 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:30:42.283 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:30:42.323 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:30:42.403 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:30:42.455 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:30:42.486 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:30:42.512 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:30:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:30:42.538 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:30:42.634 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:30:42.664 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:30:42.664 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:30:42.749 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:30:42.775 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:30:42.809 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:30:42.842 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:30:42.859 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:30:42.892 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:30:42.931 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:30:42.964 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:30:42.989 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:30:43.014 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:30:43.040 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:30:43.065 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:30:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:30:43.071 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:30:43.251 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:30:43.251 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:30:43.334 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:30:43.359 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:30:43.391 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:30:43.418 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:30:43.444 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:30:43.508 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:30:48.465 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:30:48.466 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:30:48.467 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:30:48.483 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:30:48.484 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:30:48.485 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:30:48.486 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:30:48.487 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:30:48.488 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:30:48.489 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:30:48.490 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:30:48.491 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:30:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:30:48.493 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:30:48.494 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:30:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout:Upgraded: 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:30:51.401 INFO:teuthology.orchestra.run.smithi131.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:30:51.402 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.403 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.404 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:30:51.405 INFO:teuthology.orchestra.run.smithi131.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:30:51.406 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:30:51.407 INFO:teuthology.orchestra.run.smithi131.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:30:51.408 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:30:51.409 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:30:51.410 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:30:51.411 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:30:51.412 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:30:51.413 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:30:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:30:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:30:51.414 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-28T18:30:51.414 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-03-28T18:30:51.895 DEBUG:teuthology.parallel:result is None 2024-03-28T18:30:52.816 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:52.816 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-28T18:30:52.816 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-28T18:30:52.816 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-28T18:30:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-28T18:30:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-28T18:30:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-28T18:30:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-28T18:30:52.817 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T18:30:52.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-03-28T18:30:52.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-03-28T18:30:52.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-03-28T18:30:52.962 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-03-28T18:30:52.963 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-03-28T18:30:52.963 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-03-28T18:30:52.963 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-03-28T18:30:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:52.975 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:52.991 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:52.991 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:52.991 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:30:52.991 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:53.211 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:53.256 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T18:30:53.257 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:53.257 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T18:30:53.257 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:54.468 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:54.494 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:54.494 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:54.494 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:30:54.495 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:54.495 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:54.495 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:54.720 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:54.743 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:54.899 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:55.006 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T18:30:55.006 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:55.006 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T18:30:55.007 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:55.007 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T18:30:55.007 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:55.188 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:55.211 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:55.212 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:55.212 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:30:55.212 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:55.212 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:55.212 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:55.299 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:30:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T18:30:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T18:30:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T18:30:55.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:55.667 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:55.697 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:55.729 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:30:55.790 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:30:55.791 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:55.820 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:55.867 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T18:30:55.867 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:55.867 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T18:30:55.868 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:55.868 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T18:30:55.868 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:55.977 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T18:30:56.055 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:56.104 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:56.605 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:56.635 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T18:30:56.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T18:30:56.812 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T18:30:56.839 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:57.067 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:57.096 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:57.173 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T18:30:57.196 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:57.209 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:57.368 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:30:57.398 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:57.422 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:57.422 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:57.422 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:30:57.422 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:57.422 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:57.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:57.466 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:57.477 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:57.522 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:30:58.322 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T18:30:58.357 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:58.395 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:30:58.426 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:30:58.449 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:30:58.449 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:58.449 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:30:58.449 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:30:58.450 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:30:58.450 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:58.493 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:30:58.500 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T18:30:58.515 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:30:58.516 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:58.516 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:30:58.516 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:58.569 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:30:58.592 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:58.688 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T18:30:58.720 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:58.787 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T18:30:58.787 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:58.787 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T18:30:58.787 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:58.788 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T18:30:58.788 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:30:58.845 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:58.882 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T18:30:58.895 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:30:58.917 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:30:58.917 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:30:58.917 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:30:58.917 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:30:58.918 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:30:58.918 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:30:58.966 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T18:30:59.918 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T18:30:59.947 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T18:31:00.015 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:00.082 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:31:00.148 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T18:31:00.148 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:31:00.148 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T18:31:00.148 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:00.216 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T18:31:00.289 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:00.666 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:31:00.737 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T18:31:00.738 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T18:31:00.738 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T18:31:00.738 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:31:00.738 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T18:31:00.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:03.349 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:31:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:31:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:31:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:31:03.627 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:31:03.739 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:31:03.779 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:31:03.818 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:31:03.879 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:31:03.922 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:31:03.947 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:31:03.987 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:31:04.017 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:31:04.071 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:31:04.133 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:31:04.158 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:31:04.736 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:04.780 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:04.801 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:05.557 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:31:05.612 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T18:31:05.620 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:31:05.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T18:31:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:31:05.710 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T18:31:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:31:05.823 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:31:05.854 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:31:05.879 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:31:05.905 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:31:05.931 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:31:05.931 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:31:05.996 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T18:31:06.003 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:31:06.032 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:31:06.032 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:31:06.117 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:31:06.123 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T18:31:06.143 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:31:06.171 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T18:31:06.177 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:31:06.202 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:31:06.216 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T18:31:06.218 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:31:06.251 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:31:06.286 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T18:31:06.291 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:31:06.322 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T18:31:06.323 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:31:06.349 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T18:31:06.349 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:31:06.374 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:31:06.394 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T18:31:06.399 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:31:06.424 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:31:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:06.431 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:06.447 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T18:31:06.546 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T18:31:06.610 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:06.611 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:31:06.675 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T18:31:06.685 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:31:06.710 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:31:06.714 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T18:31:06.742 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:31:06.769 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:31:06.794 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:31:06.858 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:31:07.609 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:07.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:07.713 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T18:31:09.041 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T18:31:09.131 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T18:31:09.179 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T18:31:09.315 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T18:31:09.350 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T18:31:09.383 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T18:31:09.418 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T18:31:09.479 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T18:31:09.480 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:31:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T18:31:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T18:31:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T18:31:09.871 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T18:31:09.917 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T18:31:09.950 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T18:31:09.991 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T18:31:10.058 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T18:31:10.131 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T18:31:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T18:31:10.237 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T18:31:10.270 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T18:31:10.304 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:31:10.337 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T18:31:10.373 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:10.373 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:10.704 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T18:31:10.704 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:31:10.835 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T18:31:10.865 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:31:10.920 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T18:31:10.960 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T18:31:11.000 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T18:31:11.119 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:31:11.377 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:31:11.377 INFO:teuthology.orchestra.run.smithi119.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-28T18:31:11.377 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:11.712 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T18:31:11.712 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:31:11.712 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:31:11.712 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:31:11.712 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:31:11.713 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:31:11.714 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:31:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:31:11.730 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:31:11.731 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:31:11.732 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:31:11.733 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:31:11.734 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:31:11.735 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:31:11.736 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:31:11.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:31:11.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:31:11.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:31:11.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:31:14.672 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.673 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.674 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.675 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:31:14.676 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:31:14.677 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:31:14.678 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:31:14.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:31:14.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:31:14.681 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.682 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:31:14.683 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:31:14.684 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-28T18:31:15.140 DEBUG:teuthology.parallel:result is None 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T18:31:16.135 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T18:31:16.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T18:31:16.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T18:31:16.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T18:31:16.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T18:31:16.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T18:31:16.137 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T18:31:16.151 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T18:31:16.152 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T18:31:16.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T18:31:16.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T18:31:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T18:31:16.156 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T18:31:16.157 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T18:31:16.158 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T18:31:16.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T18:31:16.160 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T18:31:16.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T18:31:16.162 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T18:31:16.163 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T18:31:16.163 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T18:31:16.163 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T18:31:16.163 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T18:31:16.163 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:18.613 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.614 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.615 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T18:31:18.617 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T18:31:18.618 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T18:31:18.619 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.620 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T18:31:18.621 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T18:31:18.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T18:31:18.623 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T18:31:18.624 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T18:31:18.625 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T18:31:18.626 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T18:31:18.626 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:31:18.626 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-28T18:31:19.308 DEBUG:teuthology.parallel:result is None 2024-03-28T18:31:19.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:19.448 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:31:19.502 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.502 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.502 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:31:19.504 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:19.725 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:31:19.750 INFO:teuthology.orchestra.run.smithi131.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.750 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.750 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:31:19.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:19.969 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T18:31:19.995 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.995 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T18:31:19.995 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T18:31:19.997 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T18:31:19.997 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:31:19.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:31:20.080 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:31:20.080 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:31:20.115 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:31:20.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T18:31:20.149 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T18:31:20.150 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:31:20.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:31:20.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:31:20.339 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:31:20.340 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:31:20.371 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:31:20.440 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:31:20.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T18:31:20.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T18:31:20.541 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T18:31:20.541 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:31:20.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:31:20.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:31:20.728 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:31:20.728 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:31:20.756 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:31:20.822 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:31:20.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T18:31:20.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T18:31:20.923 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T18:31:20.923 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:31:20.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:31:21.007 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:31:21.122 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:31:21.122 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:31:21.149 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:31:21.214 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:31:21.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T18:31:21.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T18:31:21.315 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T18:31:21.431 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-28T18:31:21.431 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:21.432 INFO:tasks.cephadm:Cluster fsid is 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:31:21.432 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T18:31:21.432 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.119', 'mon.b': '172.21.15.131', 'mon.c': '172.21.15.196'} 2024-03-28T18:31:21.432 INFO:tasks.cephadm:First mon is mon.a on smithi119 2024-03-28T18:31:21.432 INFO:tasks.cephadm:First mgr is x 2024-03-28T18:31:21.433 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T18:31:21.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-03-28T18:31:21.509 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2024-03-28T18:31:21.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-03-28T18:31:21.567 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T18:31:21.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:21.710 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T18:31:21.849 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:31:21.850 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T18:31:21.850 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:22.553 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:31 /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:22.554 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.246 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:31 /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.246 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.915 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 18:31 /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.916 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.965 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:23.983 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T18:31:24.017 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T18:31:24.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:31:24.024 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:31:24.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T18:31:24.795 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:31:24.904 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:31:25.639 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "repo_digests": [ 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-03-28T18:31:50.798 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-03-28T18:31:56.645 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-28T18:32:01.201 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:32:01.201 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T18:32:01.202 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T18:32:01.202 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-03-28T18:32:01.202 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T18:32:01.202 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:32:01.202 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:32:01.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-03-28T18:32:01.329 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2024-03-28T18:32:01.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-03-28T18:32:01.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-03-28T18:32:01.490 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2024-03-28T18:32:01.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-03-28T18:32:01.554 INFO:tasks.cephadm:Writing seed config... 2024-03-28T18:32:01.555 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T18:32:01.555 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T18:32:01.555 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T18:32:01.556 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T18:32:01.557 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T18:32:01.558 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-28T18:32:01.559 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:32:01.559 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T18:32:01.608 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5ff7da1e-ed31-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T18:32:01.609 DEBUG:teuthology.orchestra.run.smithi119:mon.a> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T18:32:01.654 DEBUG:teuthology.orchestra.run.smithi119:mgr.x> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T18:32:01.698 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T18:32:01.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.119 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:32:01.969 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-28T18:32:01.969 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', '5ff7da1e-ed31-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.119', '--skip-admin-label'] 2024-03-28T18:32:02.026 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:32:02.026 INFO:teuthology.orchestra.run.smithi119.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T18:32:02.027 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying podman|docker is present... 2024-03-28T18:32:02.077 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:32:02.077 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying lvm2 is present... 2024-03-28T18:32:02.077 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying time synchronization is in place... 2024-03-28T18:32:02.092 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T18:32:02.093 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T18:32:02.110 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T18:32:02.111 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:32:02.131 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-03-28T18:32:02.149 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-03-28T18:32:02.150 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-03-28T18:32:02.150 INFO:teuthology.orchestra.run.smithi119.stdout:Repeating the final host check... 2024-03-28T18:32:02.200 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T18:32:02.200 INFO:teuthology.orchestra.run.smithi119.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T18:32:02.201 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl is present 2024-03-28T18:32:02.201 INFO:teuthology.orchestra.run.smithi119.stdout:lvcreate is present 2024-03-28T18:32:02.218 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T18:32:02.218 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T18:32:02.239 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T18:32:02.239 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:32:02.263 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-03-28T18:32:02.284 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-03-28T18:32:02.284 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-03-28T18:32:02.285 INFO:teuthology.orchestra.run.smithi119.stdout:Host looks OK 2024-03-28T18:32:02.285 INFO:teuthology.orchestra.run.smithi119.stdout:Cluster fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:02.286 INFO:teuthology.orchestra.run.smithi119.stdout:Acquiring lock 140573883686336 on /run/cephadm/5ff7da1e-ed31-11ee-95d0-87774f69a715.lock 2024-03-28T18:32:02.286 INFO:teuthology.orchestra.run.smithi119.stdout:Lock 140573883686336 acquired on /run/cephadm/5ff7da1e-ed31-11ee-95d0-87774f69a715.lock 2024-03-28T18:32:02.287 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 3300 ... 2024-03-28T18:32:02.289 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 6789 ... 2024-03-28T18:32:02.289 INFO:teuthology.orchestra.run.smithi119.stdout:Base mon IP(s) is [172.21.15.119:3300, 172.21.15.119:6789], mon addrv is [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-03-28T18:32:02.304 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.119 metric 100 2024-03-28T18:32:02.304 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.119 metric 100 2024-03-28T18:32:02.317 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T18:32:02.317 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-28T18:32:02.317 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1766sec hoplimit 64 pref medium 2024-03-28T18:32:02.328 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T18:32:02.329 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T18:32:02.329 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T18:32:02.329 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-28T18:32:02.329 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-03-28T18:32:02.329 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T18:32:02.332 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-03-28T18:32:02.332 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-03-28T18:32:02.333 INFO:teuthology.orchestra.run.smithi119.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T18:32:02.335 INFO:teuthology.orchestra.run.smithi119.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T18:32:02.338 INFO:teuthology.orchestra.run.smithi119.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:32:12.097 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T18:32:12.098 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T18:32:13.883 INFO:teuthology.orchestra.run.smithi119.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:32:13.883 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T18:32:13.883 INFO:teuthology.orchestra.run.smithi119.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T18:32:14.946 INFO:teuthology.orchestra.run.smithi119.stdout:stat: stdout 167 167 2024-03-28T18:32:14.947 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial keys... 2024-03-28T18:32:15.990 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQCvtwVmb/UHFxAAy8J3E7RmMREdQ21mXAO/qg== 2024-03-28T18:32:17.120 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQCwtwVmqV98GxAAxYTFKsbrVLEB4EtIckfiwA== 2024-03-28T18:32:18.233 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQCxtwVmvXrDJxAAeqQSbrTL0pRSwAAM1IQQAw== 2024-03-28T18:32:18.235 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial monmap... 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool for a [v2:172.21.15.119:3300,v1:172.21.15.119:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = quincy 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: set fsid to 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T18:32:19.323 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:32:19.324 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mon... 2024-03-28T18:32:20.380 INFO:teuthology.orchestra.run.smithi119.stdout:create mon.a on 2024-03-28T18:32:20.728 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T18:32:20.970 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T18:32:21.240 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target. 2024-03-28T18:32:21.240 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target. 2024-03-28T18:32:21.598 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a 2024-03-28T18:32:21.599 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T18:32:21.871 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target.wants/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@.service. 2024-03-28T18:32:22.497 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 podman[31433]: 2024-03-28 18:32:22.487313175 +0000 UTC m=+0.361728591 container init b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T18:32:22.498 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 podman[31433]: 2024-03-28 18:32:22.496772866 +0000 UTC m=+0.371188281 container start b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:32:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:32:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-03-28T18:32:22.604 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:32:22.604 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:32:22.605 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-03-28T18:32:22.605 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T18:32:22.605 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon to start... 2024-03-28T18:32:22.605 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon... 2024-03-28T18:32:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 bash[31433]: b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 2024-03-28T18:32:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 systemd[1]: Started Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:32:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 ceph-mon[31451]: mkfs 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:22 smithi119 ceph-mon[31451]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T18:32:23.893 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T18:32:23.893 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:32:23.893 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: fsmap 2024-03-28T18:32:23.894 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T18:32:23.894 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: mgrmap e1: no daemons active 2024-03-28T18:32:23.894 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:23 smithi119 ceph-mon[31451]: from='client.? 172.21.15.119:0/3884956728' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T18:32:23.907 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T18:32:23.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout id: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:23.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T18:32:23.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:23.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout services: 2024-03-28T18:32:23.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.738697s) 2024-03-28T18:32:23.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T18:32:23.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T18:32:23.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout data: 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.219+0000 7fbb9f200700 1 Processor -- start 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.219+0000 7fbb9f200700 1 -- start start 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.220+0000 7fbb9f200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:23.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.220+0000 7fbb9f200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba010a500 con 0x7fbba0109b10 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.221+0000 7fbb9de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.221+0000 7fbb9de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40560/0 (socket says 172.21.15.119:40560) 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.221+0000 7fbb9de00700 1 -- 172.21.15.119:0/690967742 learned_addr learned my addr 172.21.15.119:0/690967742 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.221+0000 7fbb9de00700 1 -- 172.21.15.119:0/690967742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba010ad10 con 0x7fbba0109b10 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.222+0000 7fbb9de00700 1 --2- 172.21.15.119:0/690967742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbb900098a0 tx=0x7fbb90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1379fa090f9423 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.222+0000 7fbb9ca00700 1 -- 172.21.15.119:0/690967742 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb90004160 con 0x7fbba0109b10 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.222+0000 7fbb9ca00700 1 -- 172.21.15.119:0/690967742 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90004300 con 0x7fbba0109b10 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.222+0000 7fbb9ca00700 1 -- 172.21.15.119:0/690967742 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb900036a0 con 0x7fbba0109b10 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 -- 172.21.15.119:0/690967742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 msgr2=0x7fbba0109f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 --2- 172.21.15.119:0/690967742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbb900098a0 tx=0x7fbb90009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.913 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 -- 172.21.15.119:0/690967742 shutdown_connections 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 --2- 172.21.15.119:0/690967742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba0109f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 -- 172.21.15.119:0/690967742 >> 172.21.15.119:0/690967742 conn(0x7fbba007cec0 msgr2=0x7fbba007d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 -- 172.21.15.119:0/690967742 shutdown_connections 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.223+0000 7fbb9f200700 1 -- 172.21.15.119:0/690967742 wait complete. 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.224+0000 7fbb9f200700 1 Processor -- start 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.224+0000 7fbb9f200700 1 -- start start 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.225+0000 7fbb9f200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.225+0000 7fbb9f200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba010b240 con 0x7fbba0109b10 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.225+0000 7fbb9de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.225+0000 7fbb9de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40572/0 (socket says 172.21.15.119:40572) 2024-03-28T18:32:23.917 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.225+0000 7fbb9de00700 1 -- 172.21.15.119:0/3884956728 learned_addr learned my addr 172.21.15.119:0/3884956728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb9de00700 1 -- 172.21.15.119:0/3884956728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba019d770 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb9de00700 1 --2- 172.21.15.119:0/3884956728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb90000c00 tx=0x7fbb90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb90003db0 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb90011630 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb90011950 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.226+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba019da00 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.227+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba01a06f0 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.227+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbb90011af0 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.227+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbb9001c450 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.228+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba0069260 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.231+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbb90032020 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.309+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fbba002dbb0 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.310+0000 7fbb96a00700 1 -- 172.21.15.119:0/3884956728 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fbb90004260 con 0x7fbba0109b10 2024-03-28T18:32:23.918 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.311+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 msgr2=0x7fbba019d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.311+0000 7fbb9f200700 1 --2- 172.21.15.119:0/3884956728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbb90000c00 tx=0x7fbb90004750 comp rx=0 tx=0).stop 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.311+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 shutdown_connections 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.311+0000 7fbb9f200700 1 --2- 172.21.15.119:0/3884956728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbba0109b10 0x7fbba019d230 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.311+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 >> 172.21.15.119:0/3884956728 conn(0x7fbba007cec0 msgr2=0x7fbba0193ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.312+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 shutdown_connections 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:23.312+0000 7fbb9f200700 1 -- 172.21.15.119:0/3884956728 wait complete. 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:mon is available 2024-03-28T18:32:23.919 INFO:teuthology.orchestra.run.smithi119.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T18:32:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:24 smithi119 ceph-mon[31451]: from='client.? 172.21.15.119:0/1731909862' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T18:32:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:24 smithi119 ceph-mon[31451]: from='client.? 172.21.15.119:0/1731909862' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T18:32:25.206 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:25.207 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T18:32:25.212 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T18:32:25.212 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.488+0000 7fad34600700 1 Processor -- start 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.488+0000 7fad34600700 1 -- start start 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.489+0000 7fad34600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.489+0000 7fad34600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c1084e0 con 0x7fad2c107af0 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.489+0000 7fad31c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.489+0000 7fad31c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40588/0 (socket says 172.21.15.119:40588) 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.489+0000 7fad31c00700 1 -- 172.21.15.119:0/4284706135 learned_addr learned my addr 172.21.15.119:0/4284706135 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.490+0000 7fad31c00700 1 -- 172.21.15.119:0/4284706135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c108cf0 con 0x7fad2c107af0 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.490+0000 7fad31c00700 1 --2- 172.21.15.119:0/4284706135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fad20009e80 tx=0x7fad2000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8aba73fd8651d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.490+0000 7fad2be00700 1 -- 172.21.15.119:0/4284706135 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad20017070 con 0x7fad2c107af0 2024-03-28T18:32:25.213 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.490+0000 7fad2be00700 1 -- 172.21.15.119:0/4284706135 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20003a20 con 0x7fad2c107af0 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.490+0000 7fad2be00700 1 -- 172.21.15.119:0/4284706135 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad20003d20 con 0x7fad2c107af0 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 -- 172.21.15.119:0/4284706135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 msgr2=0x7fad2c107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 --2- 172.21.15.119:0/4284706135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fad20009e80 tx=0x7fad2000b330 comp rx=0 tx=0).stop 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 -- 172.21.15.119:0/4284706135 shutdown_connections 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 --2- 172.21.15.119:0/4284706135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c107f10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 -- 172.21.15.119:0/4284706135 >> 172.21.15.119:0/4284706135 conn(0x7fad2c1030f0 msgr2=0x7fad2c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 -- 172.21.15.119:0/4284706135 shutdown_connections 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.491+0000 7fad34600700 1 -- 172.21.15.119:0/4284706135 wait complete. 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.492+0000 7fad34600700 1 Processor -- start 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad34600700 1 -- start start 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad34600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad34600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c109220 con 0x7fad2c107af0 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad31c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:25.214 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad31c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40594/0 (socket says 172.21.15.119:40594) 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.493+0000 7fad31c00700 1 -- 172.21.15.119:0/1731909862 learned_addr learned my addr 172.21.15.119:0/1731909862 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad31c00700 1 -- 172.21.15.119:0/1731909862 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c19b7d0 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad31c00700 1 --2- 172.21.15.119:0/1731909862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fad20004000 tx=0x7fad20004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad20022020 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20018040 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c19ba60 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad20017070 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.494+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c19e750 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.495+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fad20025020 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.495+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fad2000f050 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.495+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c069260 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.498+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fad2002c030 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.575+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fad2c02dbf0 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.584+0000 7fad2a000700 1 -- 172.21.15.119:0/1731909862 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fad2001e480 con 0x7fad2c107af0 2024-03-28T18:32:25.215 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.585+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 msgr2=0x7fad2c19b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.585+0000 7fad34600700 1 --2- 172.21.15.119:0/1731909862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fad20004000 tx=0x7fad20004ad0 comp rx=0 tx=0).stop 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.585+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 shutdown_connections 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.585+0000 7fad34600700 1 --2- 172.21.15.119:0/1731909862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad2c107af0 0x7fad2c19b290 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.585+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 >> 172.21.15.119:0/1731909862 conn(0x7fad2c1030f0 msgr2=0x7fad2c191ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.586+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 shutdown_connections 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:24.586+0000 7fad34600700 1 -- 172.21.15.119:0/1731909862 wait complete. 2024-03-28T18:32:25.216 INFO:teuthology.orchestra.run.smithi119.stdout:Generating new minimal ceph.conf... 2024-03-28T18:32:26.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:25 smithi119 ceph-mon[31451]: from='client.? 172.21.15.119:0/4027509897' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:32:26.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.761+0000 7f09a9800700 1 Processor -- start 2024-03-28T18:32:26.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a9800700 1 -- start start 2024-03-28T18:32:26.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a9800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:26.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a9800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4108530 con 0x7f09a4107b40 2024-03-28T18:32:26.466 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40600/0 (socket says 172.21.15.119:40600) 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.762+0000 7f09a2a00700 1 -- 172.21.15.119:0/749117360 learned_addr learned my addr 172.21.15.119:0/749117360 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.763+0000 7f09a2a00700 1 -- 172.21.15.119:0/749117360 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4108cf0 con 0x7f09a4107b40 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.763+0000 7f09a2a00700 1 --2- 172.21.15.119:0/749117360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e80 tx=0x7f099400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03cdc71f6687edc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.763+0000 7f09a1600700 1 -- 172.21.15.119:0/749117360 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0994017070 con 0x7f09a4107b40 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.763+0000 7f09a1600700 1 -- 172.21.15.119:0/749117360 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f099400b8c0 con 0x7f09a4107b40 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.763+0000 7f09a1600700 1 -- 172.21.15.119:0/749117360 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09940045c0 con 0x7f09a4107b40 2024-03-28T18:32:26.467 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.764+0000 7f09a9800700 1 -- 172.21.15.119:0/749117360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 msgr2=0x7f09a4107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.764+0000 7f09a9800700 1 --2- 172.21.15.119:0/749117360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e80 tx=0x7f099400b330 comp rx=0 tx=0).stop 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.765+0000 7f09a9800700 1 -- 172.21.15.119:0/749117360 shutdown_connections 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.765+0000 7f09a9800700 1 --2- 172.21.15.119:0/749117360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a4107f60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.765+0000 7f09a9800700 1 -- 172.21.15.119:0/749117360 >> 172.21.15.119:0/749117360 conn(0x7f09a4103140 msgr2=0x7f09a4105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.765+0000 7f09a9800700 1 -- 172.21.15.119:0/749117360 shutdown_connections 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.765+0000 7f09a9800700 1 -- 172.21.15.119:0/749117360 wait complete. 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.766+0000 7f09a9800700 1 Processor -- start 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.766+0000 7f09a9800700 1 -- start start 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a9800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:26.471 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a9800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a410c0a0 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a2a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40614/0 (socket says 172.21.15.119:40614) 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a2a00700 1 -- 172.21.15.119:0/4027509897 learned_addr learned my addr 172.21.15.119:0/4027509897 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a2a00700 1 -- 172.21.15.119:0/4027509897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a419c000 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.767+0000 7f09a2a00700 1 --2- 172.21.15.119:0/4027509897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e50 tx=0x7f099400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0994020020 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0994003b50 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a419c290 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0994017040 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a419ef80 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0994023020 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.768+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f099400f050 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.769+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a4069260 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.772+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f099402a020 con 0x7f09a4107b40 2024-03-28T18:32:26.472 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.848+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f09a419f150 con 0x7f09a4107b40 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.848+0000 7f099b400700 1 -- 172.21.15.119:0/4027509897 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f0994003d00 con 0x7f09a4107b40 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 msgr2=0x7f09a419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 --2- 172.21.15.119:0/4027509897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0994009e50 tx=0x7f099400bd30 comp rx=0 tx=0).stop 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 shutdown_connections 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 --2- 172.21.15.119:0/4027509897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a4107b40 0x7f09a419bac0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 >> 172.21.15.119:0/4027509897 conn(0x7f09a4103140 msgr2=0x7f09a4103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 shutdown_connections 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:25.850+0000 7f09a9800700 1 -- 172.21.15.119:0/4027509897 wait complete. 2024-03-28T18:32:26.473 INFO:teuthology.orchestra.run.smithi119.stdout:Restarting the monitor... 2024-03-28T18:32:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:26 smithi119 systemd[1]: Stopping Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:32:27.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a[31447]: 2024-03-28T18:32:26.940+0000 7f2a16c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:32:27.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a[31447]: 2024-03-28T18:32:26.940+0000 7f2a16c00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T18:32:27.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:27 smithi119 podman[31692]: 2024-03-28 18:32:27.136923494 +0000 UTC m=+0.313695255 container died b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T18:32:27.775 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:27 smithi119 podman[31692]: 2024-03-28 18:32:27.549740275 +0000 UTC m=+0.726512033 container cleanup b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:32:27.775 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:27 smithi119 bash[31692]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a 2024-03-28T18:32:28.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:27 smithi119 podman[31704]: 2024-03-28 18:32:27.775331748 +0000 UTC m=+0.622108753 container remove b8546ba182ca05cc96a96484995bc149f930679a2db66a49dae4dfe978174188 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T18:32:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T18:32:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 systemd[1]: Stopped Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:32:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service: Consumed 1.857s CPU time. 2024-03-28T18:32:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 systemd[1]: Starting Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:32:28.955 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 podman[31794]: 2024-03-28 18:32:28.576806991 +0000 UTC m=+0.035350653 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:32:28.955 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 podman[31794]: 2024-03-28 18:32:28.743186513 +0000 UTC m=+0.201730202 container create 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:32:28.955 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 podman[31794]: 2024-03-28 18:32:28.955397664 +0000 UTC m=+0.413941393 container init 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:32:29.068 INFO:teuthology.orchestra.run.smithi119.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:28 smithi119 podman[31794]: 2024-03-28 18:32:28.965662513 +0000 UTC m=+0.424206206 container start 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: pidfile_write: ignore empty --pid-file 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: load: jerasure load: lrc 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: RocksDB version: 7.9.2 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Git sha 0 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: DB SUMMARY 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: DB Session ID: ZQ56284521R6FPHOTNBN 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: CURRENT file: CURRENT 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T18:32:29.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100656 ; 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.error_if_exists: 0 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.create_if_missing: 0 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.paranoid_checks: 1 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.env: 0x5646dadcdb20 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.info_log: 0x5646dcc81960 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.statistics: (nil) 2024-03-28T18:32:29.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.use_fsync: 0 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_log_file_size: 0 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T18:32:29.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_fallocate: 1 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.use_direct_reads: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.db_log_dir: 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.wal_dir: 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T18:32:29.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.write_buffer_manager: 0x5646dcc541e0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.unordered_write: 0 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T18:32:29.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.row_cache: None 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.wal_filter: None 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.two_write_queues: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.wal_compression: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.atomic_flush: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.log_readahead_size: 0 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T18:32:29.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_background_jobs: 2 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_background_compactions: -1 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_subcompactions: 1 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T18:32:29.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_open_files: -1 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_background_flushes: -1 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Compression algorithms supported: 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kZSTD supported: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kXpressCompression supported: 0 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kZlibCompression supported: 1 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kSnappyCompression supported: 1 2024-03-28T18:32:29.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kLZ4Compression supported: 1 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: kBZip2Compression supported: 0 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T18:32:29.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.merge_operator: 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_filter: None 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5646dcc81a60) 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-03-28T18:32:29.226 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_type: 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_index_type: 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_shortening: 1 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: checksum: 4 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: no_block_cache: 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache: 0x5646dcc87090 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_options: 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: capacity : 536870912 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_shard_bits : 4 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: strict_capacity_limit : 0 2024-03-28T18:32:29.227 INFO:journalctl@ceph.mon.a.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_compressed: (nil) 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: persistent_cache: (nil) 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size: 4096 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size_deviation: 10 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_restart_interval: 16 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_block_restart_interval: 1 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: metadata_block_size: 4096 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: partition_filters: 0 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: use_delta_encoding: 1 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: filter_policy: bloomfilter 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: whole_key_filtering: 1 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: verify_compression: 0 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: format_version: 5 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_index_compression: 1 2024-03-28T18:32:29.228 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_align: 0 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout: max_auto_readahead_size: 262144 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout: prepopulate_block_cache: 0 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout: initial_auto_readahead_size: 8192 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression: NoCompression 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.num_levels: 7 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T18:32:29.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T18:32:29.230 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T18:32:29.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T18:32:29.232 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.table_properties_collectors: 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.inplace_update_support: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.bloom_locality: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.max_successive_merges: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T18:32:29.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.ttl: 2592000 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enable_blob_files: false 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.min_blob_size: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T18:32:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0871a2cf-64e8-475e-b239-e97343e3eeed 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650749022744, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650749024480, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95777, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93763, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13904, "raw_average_key_size": 50, "raw_value_size": 86223, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711650749, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0871a2cf-64e8-475e-b239-e97343e3eeed", "db_session_id": "ZQ56284521R6FPHOTNBN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711650749024647, "job": 1, "event": "recovery_finished"} 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5646dcd7a000 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: DB pointer 0x5646dcd64000 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T18:32:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** DB Stats ** 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 2/0 95.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 95.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:32:29.236 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 13.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:32:29.237 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 13.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: Block cache BinnedLRUCache@0x5646dcc87090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: starting mon.a rank 0 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???) e1 preinit fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).mds e1 new map 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).mds e1 print_map 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: e1 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: legacy client fscid: -1 2024-03-28T18:32:29.238 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout: No filesystems configured 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 bash[31794]: 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 systemd[1]: Started Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: fsmap 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T18:32:29.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:29 smithi119 ceph-mon[31813]: mgrmap e1: no daemons active 2024-03-28T18:32:30.404 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.691+0000 7fe427000700 1 Processor -- start 2024-03-28T18:32:30.405 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.691+0000 7fe427000700 1 -- start start 2024-03-28T18:32:30.406 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.691+0000 7fe427000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:30.406 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.691+0000 7fe427000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42007c8a0 con 0x7fe42007de70 2024-03-28T18:32:30.406 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.692+0000 7fe41fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:30.407 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.692+0000 7fe41fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43152/0 (socket says 172.21.15.119:43152) 2024-03-28T18:32:30.407 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.692+0000 7fe41fe00700 1 -- 172.21.15.119:0/395114382 learned_addr learned my addr 172.21.15.119:0/395114382 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:30.409 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.693+0000 7fe41fe00700 1 -- 172.21.15.119:0/395114382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42007ca20 con 0x7fe42007de70 2024-03-28T18:32:30.409 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.693+0000 7fe41fe00700 1 --2- 172.21.15.119:0/395114382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4100098a0 tx=0x7fe410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f91b7d47131a56e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:30.409 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.693+0000 7fe41ea00700 1 -- 172.21.15.119:0/395114382 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe410004160 con 0x7fe42007de70 2024-03-28T18:32:30.409 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.693+0000 7fe41ea00700 1 -- 172.21.15.119:0/395114382 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe410004300 con 0x7fe42007de70 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.693+0000 7fe41ea00700 1 -- 172.21.15.119:0/395114382 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe410011b50 con 0x7fe42007de70 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 -- 172.21.15.119:0/395114382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 msgr2=0x7fe42007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 --2- 172.21.15.119:0/395114382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe4100098a0 tx=0x7fe410009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 -- 172.21.15.119:0/395114382 shutdown_connections 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 --2- 172.21.15.119:0/395114382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42007c2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 -- 172.21.15.119:0/395114382 >> 172.21.15.119:0/395114382 conn(0x7fe420103190 msgr2=0x7fe4201055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 -- 172.21.15.119:0/395114382 shutdown_connections 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.694+0000 7fe427000700 1 -- 172.21.15.119:0/395114382 wait complete. 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.695+0000 7fe427000700 1 Processor -- start 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe427000700 1 -- start start 2024-03-28T18:32:30.410 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe427000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:30.411 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe427000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42010e100 con 0x7fe42007de70 2024-03-28T18:32:30.411 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe41fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:30.411 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe41fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43154/0 (socket says 172.21.15.119:43154) 2024-03-28T18:32:30.411 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.696+0000 7fe41fe00700 1 -- 172.21.15.119:0/342300555 learned_addr learned my addr 172.21.15.119:0/342300555 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:30.411 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe41fe00700 1 -- 172.21.15.119:0/342300555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42019e3f0 con 0x7fe42007de70 2024-03-28T18:32:30.413 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe41fe00700 1 --2- 172.21.15.119:0/342300555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fb0 tx=0x7fe410011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:30.413 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4100038b0 con 0x7fe42007de70 2024-03-28T18:32:30.413 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe410011e50 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42019e680 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe410019550 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.697+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42019f060 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.698+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe410019b90 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.698+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe410024730 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.699+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4200637c0 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.702+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe410003a50 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.777+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe42019f350 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.781+0000 7fe41cc00700 1 -- 172.21.15.119:0/342300555 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe410024930 con 0x7fe42007de70 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 msgr2=0x7fe42019deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 --2- 172.21.15.119:0/342300555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe410009fb0 tx=0x7fe410011bf0 comp rx=0 tx=0).stop 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 shutdown_connections 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 --2- 172.21.15.119:0/342300555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe42007de70 0x7fe42019deb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 >> 172.21.15.119:0/342300555 conn(0x7fe420103190 msgr2=0x7fe420103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 shutdown_connections 2024-03-28T18:32:30.415 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:29.783+0000 7fe427000700 1 -- 172.21.15.119:0/342300555 wait complete. 2024-03-28T18:32:30.416 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T18:32:30.434 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:32:30.435 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mgr... 2024-03-28T18:32:30.436 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T18:32:30.438 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T18:32:30.878 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x 2024-03-28T18:32:30.879 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T18:32:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/342300555' entity='client.admin' 2024-03-28T18:32:31.126 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715.target.wants/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@.service. 2024-03-28T18:32:31.494 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 systemd[1]: Starting Ceph mgr.x for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:32:31.745 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 podman[32029]: 2024-03-28 18:32:31.398552126 +0000 UTC m=+0.033192027 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:32:31.746 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 podman[32029]: 2024-03-28 18:32:31.496088246 +0000 UTC m=+0.130728142 container create fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T18:32:31.746 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 podman[32029]: 2024-03-28 18:32:31.736046828 +0000 UTC m=+0.370686768 container init fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:32:31.746 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 podman[32029]: 2024-03-28 18:32:31.745342639 +0000 UTC m=+0.379982552 container start fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:32:31.876 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:32:31.876 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-03-28T18:32:31.892 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:32:31.893 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:32:31.893 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-03-28T18:32:31.893 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T18:32:31.917 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:32:31.917 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-03-28T18:32:31.935 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:32:31.935 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:32:31.935 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-03-28T18:32:31.936 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T18:32:31.936 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr to start... 2024-03-28T18:32:31.936 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr... 2024-03-28T18:32:32.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 bash[32029]: fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d 2024-03-28T18:32:32.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:31 smithi119 systemd[1]: Started Ceph mgr.x for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:32:32.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:32 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3090182382' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:32:32.660 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:32 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:32.520+0000 7fed055d7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:32:32.961 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:32 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:32.659+0000 7fed055d7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:32:33.274 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:33.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:32:33.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:32:33.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:32:33.285 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:32:22.574171+0000", 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.452+0000 7f1c78c00700 1 Processor -- start 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.452+0000 7f1c78c00700 1 -- start start 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.453+0000 7f1c78c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.453+0000 7f1c78c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7407be60 con 0x7f1c74106370 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.453+0000 7f1c71e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:33.286 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.453+0000 7f1c71e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43182/0 (socket says 172.21.15.119:43182) 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.453+0000 7f1c71e00700 1 -- 172.21.15.119:0/2804462702 learned_addr learned my addr 172.21.15.119:0/2804462702 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.454+0000 7f1c71e00700 1 -- 172.21.15.119:0/2804462702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74106cd0 con 0x7f1c74106370 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.454+0000 7f1c71e00700 1 --2- 172.21.15.119:0/2804462702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c640098a0 tx=0x7f1c64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b6150b416978b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.454+0000 7f1c70a00700 1 -- 172.21.15.119:0/2804462702 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c64004160 con 0x7f1c74106370 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.454+0000 7f1c70a00700 1 -- 172.21.15.119:0/2804462702 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c64004300 con 0x7f1c74106370 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.454+0000 7f1c70a00700 1 -- 172.21.15.119:0/2804462702 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c64011b50 con 0x7f1c74106370 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 -- 172.21.15.119:0/2804462702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 msgr2=0x7f1c74106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 --2- 172.21.15.119:0/2804462702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1c640098a0 tx=0x7f1c64009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 -- 172.21.15.119:0/2804462702 shutdown_connections 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 --2- 172.21.15.119:0/2804462702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c74106790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 -- 172.21.15.119:0/2804462702 >> 172.21.15.119:0/2804462702 conn(0x7f1c74101ff0 msgr2=0x7f1c74104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 -- 172.21.15.119:0/2804462702 shutdown_connections 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.455+0000 7f1c78c00700 1 -- 172.21.15.119:0/2804462702 wait complete. 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.456+0000 7f1c78c00700 1 Processor -- start 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c78c00700 1 -- start start 2024-03-28T18:32:33.287 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c78c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:33.288 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c78c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74107d60 con 0x7f1c74106370 2024-03-28T18:32:33.288 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c71e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:33.288 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c71e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43196/0 (socket says 172.21.15.119:43196) 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.457+0000 7f1c71e00700 1 -- 172.21.15.119:0/3090182382 learned_addr learned my addr 172.21.15.119:0/3090182382 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c71e00700 1 -- 172.21.15.119:0/3090182382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7419e220 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c71e00700 1 --2- 172.21.15.119:0/3090182382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c64011b60 tx=0x7f1c64011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c64003970 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c64003b10 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7419e4b0 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c64024610 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.458+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c741a1190 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.459+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1c64024ca0 con 0x7f1c74106370 2024-03-28T18:32:33.290 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.459+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1c64011e00 con 0x7f1c74106370 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.459+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c74069260 con 0x7f1c74106370 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.462+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1c6401a9f0 con 0x7f1c74106370 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.540+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1c7402dc80 con 0x7f1c74106370 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.541+0000 7f1c6aa00700 1 -- 172.21.15.119:0/3090182382 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f1c6401a430 con 0x7f1c74106370 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 msgr2=0x7f1c7419dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 --2- 172.21.15.119:0/3090182382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1c64011b60 tx=0x7f1c64011c40 comp rx=0 tx=0).stop 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 shutdown_connections 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 --2- 172.21.15.119:0/3090182382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c74106370 0x7f1c7419dce0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 >> 172.21.15.119:0/3090182382 conn(0x7f1c74101ff0 msgr2=0x7f1c74194920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.542+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 shutdown_connections 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:32.543+0000 7f1c78c00700 1 -- 172.21.15.119:0/3090182382 wait complete. 2024-03-28T18:32:33.291 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (1/15)... 2024-03-28T18:32:33.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:33 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:33.465+0000 7fed055d7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:32:33.885 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:33 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:33.584+0000 7fed055d7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:32:33.885 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:33 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:33.763+0000 7fed055d7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:32:33.885 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:33 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:33.885+0000 7fed055d7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:32:34.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:33 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:33.993+0000 7fed055d7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:32:34.697 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:34.327+0000 7fed055d7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:32:34.698 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:34.464+0000 7fed055d7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:32:34.698 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:34.574+0000 7fed055d7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:32:35.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:34.697+0000 7fed055d7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:32:35.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:34.911+0000 7fed055d7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:32:36.220 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/628008754' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:32:36.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:36 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:36.219+0000 7fed055d7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:32:36.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:36 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:36.392+0000 7fed055d7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:32:36.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:32:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:32:36.663 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:32:36.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:32:36.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:32:22.574171+0000", 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.857+0000 7f729b600700 1 Processor -- start 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.857+0000 7f729b600700 1 -- start start 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.857+0000 7f729b600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:36.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.858+0000 7f729b600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729407c860 con 0x7f729407de30 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.858+0000 7f7298c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.858+0000 7f7298c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43204/0 (socket says 172.21.15.119:43204) 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.858+0000 7f7298c00700 1 -- 172.21.15.119:0/1867207205 learned_addr learned my addr 172.21.15.119:0/1867207205 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.859+0000 7f7298c00700 1 -- 172.21.15.119:0/1867207205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729407c9e0 con 0x7f729407de30 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.859+0000 7f7298c00700 1 --2- 172.21.15.119:0/1867207205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f72880098a0 tx=0x7f7288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563d30f13dbd195e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.859+0000 7f7293400700 1 -- 172.21.15.119:0/1867207205 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7288004160 con 0x7f729407de30 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.859+0000 7f7293400700 1 -- 172.21.15.119:0/1867207205 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7288004300 con 0x7f729407de30 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.859+0000 7f7293400700 1 -- 172.21.15.119:0/1867207205 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7288011b50 con 0x7f729407de30 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 -- 172.21.15.119:0/1867207205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 msgr2=0x7f729407c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 --2- 172.21.15.119:0/1867207205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f72880098a0 tx=0x7f7288009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 -- 172.21.15.119:0/1867207205 shutdown_connections 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 --2- 172.21.15.119:0/1867207205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729407c290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 -- 172.21.15.119:0/1867207205 >> 172.21.15.119:0/1867207205 conn(0x7f72941030f0 msgr2=0x7f7294105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 -- 172.21.15.119:0/1867207205 shutdown_connections 2024-03-28T18:32:36.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.860+0000 7f729b600700 1 -- 172.21.15.119:0/1867207205 wait complete. 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.861+0000 7f729b600700 1 Processor -- start 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.861+0000 7f729b600700 1 -- start start 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f729b600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f729b600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729410e120 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f7298c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f7298c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43220/0 (socket says 172.21.15.119:43220) 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f7298c00700 1 -- 172.21.15.119:0/628008754 learned_addr learned my addr 172.21.15.119:0/628008754 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.862+0000 7f7298c00700 1 -- 172.21.15.119:0/628008754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729419e220 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7298c00700 1 --2- 172.21.15.119:0/628008754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7288011b60 tx=0x7f7288011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7288003970 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7288003b10 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729419e4b0 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7288024610 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72941a1190 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7288024c50 con 0x7f729407de30 2024-03-28T18:32:36.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.863+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7288011e00 con 0x7f729407de30 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.864+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7294069260 con 0x7f729407de30 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.869+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7288043dd0 con 0x7f729407de30 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.947+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f72941a1360 con 0x7f729407de30 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.948+0000 7f7291600700 1 -- 172.21.15.119:0/628008754 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f7288032030 con 0x7f729407de30 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.949+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 msgr2=0x7f729419dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.949+0000 7f729b600700 1 --2- 172.21.15.119:0/628008754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7288011b60 tx=0x7f7288011c40 comp rx=0 tx=0).stop 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.949+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 shutdown_connections 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.949+0000 7f729b600700 1 --2- 172.21.15.119:0/628008754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f729407de30 0x7f729419dce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.949+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 >> 172.21.15.119:0/628008754 conn(0x7f72941030f0 msgr2=0x7f7294103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.950+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 shutdown_connections 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:35.950+0000 7f729b600700 1 -- 172.21.15.119:0/628008754 wait complete. 2024-03-28T18:32:36.676 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (2/15)... 2024-03-28T18:32:36.924 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:36 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:36.809+0000 7fed055d7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:32:36.924 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:36 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:36.923+0000 7fed055d7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:32:37.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:37 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:37.174+0000 7fed055d7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:32:37.569 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:37 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:37.411+0000 7fed055d7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:32:38.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:37 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:37.782+0000 7fed055d7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:32:38.394 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:38 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:38.094+0000 7fed055d7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:32:38.394 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:38 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:38.392+0000 7fed055d7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:32:39.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:39 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4235293720' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:32:39.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:39 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:39.441+0000 7fed055d7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:32:40.113 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.114 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:32:40.115 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:40.115 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:32:40.115 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:32:40.115 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:32:40.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:32:40.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:32:22.574171+0000", 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.382+0000 7f1b01a00700 1 Processor -- start 2024-03-28T18:32:40.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.383+0000 7f1b01a00700 1 -- start start 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.383+0000 7f1b01a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.383+0000 7f1b01a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc07be60 con 0x7f1afc106370 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.384+0000 7f1afaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.384+0000 7f1afaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36170/0 (socket says 172.21.15.119:36170) 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.384+0000 7f1afaa00700 1 -- 172.21.15.119:0/2350760731 learned_addr learned my addr 172.21.15.119:0/2350760731 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.384+0000 7f1afaa00700 1 -- 172.21.15.119:0/2350760731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc108ce0 con 0x7f1afc106370 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.384+0000 7f1afaa00700 1 --2- 172.21.15.119:0/2350760731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0098a0 tx=0x7f1aec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe24f1c1de67fee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1af9600700 1 -- 172.21.15.119:0/2350760731 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1aec004160 con 0x7f1afc106370 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1af9600700 1 -- 172.21.15.119:0/2350760731 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aec004300 con 0x7f1afc106370 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1af9600700 1 -- 172.21.15.119:0/2350760731 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1aec011b50 con 0x7f1afc106370 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1b01a00700 1 -- 172.21.15.119:0/2350760731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 msgr2=0x7f1afc1087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1b01a00700 1 --2- 172.21.15.119:0/2350760731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0098a0 tx=0x7f1aec009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1b01a00700 1 -- 172.21.15.119:0/2350760731 shutdown_connections 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1b01a00700 1 --2- 172.21.15.119:0/2350760731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1087a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.385+0000 7f1b01a00700 1 -- 172.21.15.119:0/2350760731 >> 172.21.15.119:0/2350760731 conn(0x7f1afc101ff0 msgr2=0x7f1afc104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:40.125 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.386+0000 7f1b01a00700 1 -- 172.21.15.119:0/2350760731 shutdown_connections 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.386+0000 7f1b01a00700 1 -- 172.21.15.119:0/2350760731 wait complete. 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1b01a00700 1 Processor -- start 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1b01a00700 1 -- start start 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1b01a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1b01a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc109d70 con 0x7f1afc106370 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1afaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1afaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36184/0 (socket says 172.21.15.119:36184) 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.387+0000 7f1afaa00700 1 -- 172.21.15.119:0/4235293720 learned_addr learned my addr 172.21.15.119:0/4235293720 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1afaa00700 1 -- 172.21.15.119:0/4235293720 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc1a2650 con 0x7f1afc106370 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1afaa00700 1 --2- 172.21.15.119:0/4235293720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1aec011b60 tx=0x7f1aec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1aec003970 con 0x7f1afc106370 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aec003b10 con 0x7f1afc106370 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1aec024610 con 0x7f1afc106370 2024-03-28T18:32:40.126 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc1a28e0 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.388+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc1a55d0 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.389+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1aec024c50 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.389+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1aec011e00 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.389+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc069260 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.392+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1aec01a9f0 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.471+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1afc19b670 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.471+0000 7f1af3400700 1 -- 172.21.15.119:0/4235293720 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f1afc19b670 con 0x7f1afc106370 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 msgr2=0x7f1afc1a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 --2- 172.21.15.119:0/4235293720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1aec011b60 tx=0x7f1aec011c40 comp rx=0 tx=0).stop 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 shutdown_connections 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 --2- 172.21.15.119:0/4235293720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1afc106370 0x7f1afc1a2110 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 >> 172.21.15.119:0/4235293720 conn(0x7f1afc101ff0 msgr2=0x7f1afc1043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 shutdown_connections 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:39.473+0000 7f1b01a00700 1 -- 172.21.15.119:0/4235293720 wait complete. 2024-03-28T18:32:40.127 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (3/15)... 2024-03-28T18:32:40.546 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:40.144+0000 7fed055d7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:32:40.546 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:40.269+0000 7fed055d7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:32:40.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:40.545+0000 7fed055d7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: Activating manager daemon x 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: mgrmap e2: x(active, starting, since 0.0125697s) 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: Manager daemon x is now available 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' 2024-03-28T18:32:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:40 smithi119 ceph-mon[31813]: from='mgr.14100 172.21.15.119:0/2425886618' entity='mgr.x' 2024-03-28T18:32:42.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:42 smithi119 ceph-mon[31813]: mgrmap e3: x(active, since 1.02273s) 2024-03-28T18:32:43.599 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:43 smithi119 ceph-mon[31813]: mgrmap e4: x(active, since 2s) 2024-03-28T18:32:43.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:43 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2144896906' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-03-28T18:32:43.612 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T18:32:43.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T18:32:43.622 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T18:32:43.622 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T18:32:43.623 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T18:32:43.624 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T18:32:22.574171+0000", 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.714+0000 7f57ea400700 1 Processor -- start 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.714+0000 7f57ea400700 1 -- start start 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.715+0000 7f57ea400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.715+0000 7f57ea400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e407c880 con 0x7f57e407de50 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.715+0000 7f57e9000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.715+0000 7f57e9000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36266/0 (socket says 172.21.15.119:36266) 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.715+0000 7f57e9000700 1 -- 172.21.15.119:0/3975038095 learned_addr learned my addr 172.21.15.119:0/3975038095 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.716+0000 7f57e9000700 1 -- 172.21.15.119:0/3975038095 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e407ca00 con 0x7f57e407de50 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.716+0000 7f57e9000700 1 --2- 172.21.15.119:0/3975038095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f57d40098a0 tx=0x7f57d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50813d6167c5fbdc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:43.625 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.716+0000 7f57e3e00700 1 -- 172.21.15.119:0/3975038095 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57d4004160 con 0x7f57e407de50 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.716+0000 7f57e3e00700 1 -- 172.21.15.119:0/3975038095 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57d4004300 con 0x7f57e407de50 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.716+0000 7f57e3e00700 1 -- 172.21.15.119:0/3975038095 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57d4011b50 con 0x7f57e407de50 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 -- 172.21.15.119:0/3975038095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 msgr2=0x7f57e407c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 --2- 172.21.15.119:0/3975038095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f57d40098a0 tx=0x7f57d4009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 -- 172.21.15.119:0/3975038095 shutdown_connections 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 --2- 172.21.15.119:0/3975038095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e407c2b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 -- 172.21.15.119:0/3975038095 >> 172.21.15.119:0/3975038095 conn(0x7f57e4102ec0 msgr2=0x7f57e41052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 -- 172.21.15.119:0/3975038095 shutdown_connections 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.717+0000 7f57ea400700 1 -- 172.21.15.119:0/3975038095 wait complete. 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.718+0000 7f57ea400700 1 Processor -- start 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57ea400700 1 -- start start 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57ea400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57ea400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e410df10 con 0x7f57e407de50 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57e9000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:43.626 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57e9000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36272/0 (socket says 172.21.15.119:36272) 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57e9000700 1 -- 172.21.15.119:0/2144896906 learned_addr learned my addr 172.21.15.119:0/2144896906 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.719+0000 7f57e9000700 1 -- 172.21.15.119:0/2144896906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e419e010 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57e9000700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011670 tx=0x7f57d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57d40039c0 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57d4011e50 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57ea400700 1 -- 172.21.15.119:0/2144896906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e419e2a0 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57d401a5c0 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.720+0000 7f57ea400700 1 -- 172.21.15.119:0/2144896906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41a0f90 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.721+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f57d4024430 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.721+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e4069260 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.722+0000 7f57e2000700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 0x7f57c403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.722+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f57d404d280 con 0x7f57e407de50 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.722+0000 7f57dbe00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 0x7f57c403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.723+0000 7f57dbe00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 0x7f57c403b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57cc006fd0 tx=0x7f57cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:43.627 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:42.733+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f57d404cca0 con 0x7f57e407de50 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.036+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f57e40625d0 con 0x7f57e407de50 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.037+0000 7f57e2000700 1 -- 172.21.15.119:0/2144896906 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f57d4003b60 con 0x7f57e407de50 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 msgr2=0x7f57c403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 0x7f57c403b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57cc006fd0 tx=0x7f57cc006e40 comp rx=0 tx=0).stop 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 msgr2=0x7f57e419dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57d4011670 tx=0x7f57d4011b60 comp rx=0 tx=0).stop 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 shutdown_connections 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f57c4038ed0 0x7f57c403b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 --2- 172.21.15.119:0/2144896906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57e407de50 0x7f57e419dad0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.039+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 >> 172.21.15.119:0/2144896906 conn(0x7f57e4102ec0 msgr2=0x7f57e41039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.040+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 shutdown_connections 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:43.040+0000 7f57daa00700 1 -- 172.21.15.119:0/2144896906 wait complete. 2024-03-28T18:32:43.628 INFO:teuthology.orchestra.run.smithi119.stdout:mgr is available 2024-03-28T18:32:44.799 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:44 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2016303967' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T18:32:45.272 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:45.273 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T18:32:45.273 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:32:45.273 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T18:32:45.273 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T18:32:45.273 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T18:32:45.275 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T18:32:45.276 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.172+0000 7f1c90600700 1 Processor -- start 2024-03-28T18:32:45.276 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.172+0000 7f1c90600700 1 -- start start 2024-03-28T18:32:45.276 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.173+0000 7f1c90600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:45.276 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.173+0000 7f1c90600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8807bdc0 con 0x7f1c88106410 2024-03-28T18:32:45.276 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.173+0000 7f1c8dc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.173+0000 7f1c8dc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36276/0 (socket says 172.21.15.119:36276) 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.173+0000 7f1c8dc00700 1 -- 172.21.15.119:0/1747356290 learned_addr learned my addr 172.21.15.119:0/1747356290 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.174+0000 7f1c8dc00700 1 -- 172.21.15.119:0/1747356290 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c88106d70 con 0x7f1c88106410 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.174+0000 7f1c8dc00700 1 --2- 172.21.15.119:0/1747356290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0098a0 tx=0x7f1c7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3440e2e4cedd7e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.174+0000 7f1c87e00700 1 -- 172.21.15.119:0/1747356290 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c7c004160 con 0x7f1c88106410 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.174+0000 7f1c87e00700 1 -- 172.21.15.119:0/1747356290 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c7c004300 con 0x7f1c88106410 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.174+0000 7f1c87e00700 1 -- 172.21.15.119:0/1747356290 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c7c011b50 con 0x7f1c88106410 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 -- 172.21.15.119:0/1747356290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 msgr2=0x7f1c88106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 --2- 172.21.15.119:0/1747356290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0098a0 tx=0x7f1c7c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 -- 172.21.15.119:0/1747356290 shutdown_connections 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 --2- 172.21.15.119:0/1747356290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c88106830 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:45.281 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 -- 172.21.15.119:0/1747356290 >> 172.21.15.119:0/1747356290 conn(0x7f1c88101ff0 msgr2=0x7f1c88104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 -- 172.21.15.119:0/1747356290 shutdown_connections 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.175+0000 7f1c90600700 1 -- 172.21.15.119:0/1747356290 wait complete. 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.176+0000 7f1c90600700 1 Processor -- start 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.176+0000 7f1c90600700 1 -- start start 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c90600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c90600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c88107e00 con 0x7f1c88106410 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c8dc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c8dc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36292/0 (socket says 172.21.15.119:36292) 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c8dc00700 1 -- 172.21.15.119:0/2016303967 learned_addr learned my addr 172.21.15.119:0/2016303967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.177+0000 7f1c8dc00700 1 -- 172.21.15.119:0/2016303967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8819e220 con 0x7f1c88106410 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c8dc00700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c011b60 tx=0x7f1c7c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c7c003970 con 0x7f1c88106410 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c7c003b10 con 0x7f1c88106410 2024-03-28T18:32:45.282 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8819e4b0 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c7c024610 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.178+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881a1190 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.179+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f1c7c01a430 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.179+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88069260 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.180+0000 7f1c86000700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 0x7f1c7003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.180+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1c7c011e00 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.180+0000 7f1c8d200700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 0x7f1c7003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.181+0000 7f1c8d200700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 0x7f1c7003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1c74006fd0 tx=0x7f1c74006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.185+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c7c02ce10 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.403+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1c880008d0 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.405+0000 7f1c86000700 1 -- 172.21.15.119:0/2016303967 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f1c7c00e260 con 0x7f1c88106410 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 msgr2=0x7f1c7003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 0x7f1c7003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1c74006fd0 tx=0x7f1c74006e40 comp rx=0 tx=0).stop 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 msgr2=0x7f1c8819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c011b60 tx=0x7f1c7c011c40 comp rx=0 tx=0).stop 2024-03-28T18:32:45.283 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 shutdown_connections 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f1c70038ed0 0x7f1c7003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 --2- 172.21.15.119:0/2016303967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88106410 0x7f1c8819dce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 >> 172.21.15.119:0/2016303967 conn(0x7f1c88101ff0 msgr2=0x7f1c88194920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 shutdown_connections 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:44.409+0000 7f1c90600700 1 -- 172.21.15.119:0/2016303967 wait complete. 2024-03-28T18:32:45.284 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling cephadm module... 2024-03-28T18:32:46.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:46 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4059463627' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T18:32:46.550 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setuser ceph since I am not root 2024-03-28T18:32:46.550 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setgroup ceph since I am not root 2024-03-28T18:32:46.875 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f019a600700 1 Processor -- start 2024-03-28T18:32:46.875 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f019a600700 1 -- start start 2024-03-28T18:32:46.875 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f019a600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:46.875 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f019a600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019410a700 con 0x7f0194109d10 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f0193400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f0193400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36304/0 (socket says 172.21.15.119:36304) 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f0193400700 1 -- 172.21.15.119:0/2238389287 learned_addr learned my addr 172.21.15.119:0/2238389287 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.839+0000 7f0193400700 1 -- 172.21.15.119:0/2238389287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019410af10 con 0x7f0194109d10 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.840+0000 7f0193400700 1 --2- 172.21.15.119:0/2238389287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01840098a0 tx=0x7f0184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c05121951bbfeeb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.840+0000 7f0192000700 1 -- 172.21.15.119:0/2238389287 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0184004160 con 0x7f0194109d10 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.840+0000 7f0192000700 1 -- 172.21.15.119:0/2238389287 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0184004300 con 0x7f0194109d10 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.840+0000 7f0192000700 1 -- 172.21.15.119:0/2238389287 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0184011b50 con 0x7f0194109d10 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 -- 172.21.15.119:0/2238389287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 msgr2=0x7f019410a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 --2- 172.21.15.119:0/2238389287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01840098a0 tx=0x7f0184009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 -- 172.21.15.119:0/2238389287 shutdown_connections 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 --2- 172.21.15.119:0/2238389287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019410a130 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 -- 172.21.15.119:0/2238389287 >> 172.21.15.119:0/2238389287 conn(0x7f019407d100 msgr2=0x7f019407d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 -- 172.21.15.119:0/2238389287 shutdown_connections 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.841+0000 7f019a600700 1 -- 172.21.15.119:0/2238389287 wait complete. 2024-03-28T18:32:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.842+0000 7f019a600700 1 Processor -- start 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.842+0000 7f019a600700 1 -- start start 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.842+0000 7f019a600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.842+0000 7f019a600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019410e2c0 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f0193400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f0193400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36308/0 (socket says 172.21.15.119:36308) 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f0193400700 1 -- 172.21.15.119:0/4059463627 learned_addr learned my addr 172.21.15.119:0/4059463627 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f0193400700 1 -- 172.21.15.119:0/4059463627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019419e390 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f0193400700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0184011850 tx=0x7f0184011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0184003af0 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0184020070 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.843+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01840195a0 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.844+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019419e620 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.844+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019419f090 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.845+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f0184019850 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.845+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01940637c0 con 0x7f0194109d10 2024-03-28T18:32:46.877 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.845+0000 7f018be00700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 0x7f017803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.845+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0184011e00 con 0x7f0194109d10 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.847+0000 7f0192a00700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 0x7f017803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.847+0000 7f0192a00700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 0x7f017803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:45.850+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0184003c90 con 0x7f0194109d10 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.122+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f019419f650 con 0x7f0194109d10 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.175+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f018404cc20 con 0x7f0194109d10 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.179+0000 7f018be00700 1 -- 172.21.15.119:0/4059463627 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f018402bd00 con 0x7f0194109d10 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 msgr2=0x7f017803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 0x7f017803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0180006fd0 tx=0x7f0180006e40 comp rx=0 tx=0).stop 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 msgr2=0x7f019419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0184011850 tx=0x7f0184011c10 comp rx=0 tx=0).stop 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 shutdown_connections 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f0178038f20 0x7f017803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:46.878 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 --2- 172.21.15.119:0/4059463627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0194109d10 0x7f019419de50 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0184011850 tx=0x7f0184011c10 comp rx=0 tx=0).stop 2024-03-28T18:32:46.879 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 >> 172.21.15.119:0/4059463627 conn(0x7f019407d100 msgr2=0x7f01941074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:46.879 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 shutdown_connections 2024-03-28T18:32:46.879 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:46.180+0000 7f019a600700 1 -- 172.21.15.119:0/4059463627 wait complete. 2024-03-28T18:32:47.172 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:46.955+0000 7f74f7082200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:32:47.172 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:47.077+0000 7f74f7082200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:32:47.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:47 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4059463627' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T18:32:47.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:47 smithi119 ceph-mon[31813]: mgrmap e5: x(active, since 5s) 2024-03-28T18:32:48.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:47.884+0000 7f74f7082200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:32:48.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.006+0000 7f74f7082200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:32:48.427 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1292996316' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T18:32:48.427 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.200+0000 7f74f7082200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:32:48.428 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.337+0000 7f74f7082200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.470+0000 7f22c1e00700 1 Processor -- start 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.470+0000 7f22c1e00700 1 -- start start 2024-03-28T18:32:48.441 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.471+0000 7f22c1e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.471+0000 7f22c1e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc07c8a0 con 0x7f22bc07de70 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.471+0000 7f22baa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.471+0000 7f22baa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36324/0 (socket says 172.21.15.119:36324) 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.471+0000 7f22baa00700 1 -- 172.21.15.119:0/359015211 learned_addr learned my addr 172.21.15.119:0/359015211 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.472+0000 7f22baa00700 1 -- 172.21.15.119:0/359015211 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc07ca20 con 0x7f22bc07de70 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.472+0000 7f22baa00700 1 --2- 172.21.15.119:0/359015211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098a0 tx=0x7f22ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5572fbd40a1ba587 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.472+0000 7f22b9600700 1 -- 172.21.15.119:0/359015211 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac004160 con 0x7f22bc07de70 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.472+0000 7f22b9600700 1 -- 172.21.15.119:0/359015211 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22ac004300 con 0x7f22bc07de70 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.472+0000 7f22b9600700 1 -- 172.21.15.119:0/359015211 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac011b50 con 0x7f22bc07de70 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 -- 172.21.15.119:0/359015211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 msgr2=0x7f22bc07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 --2- 172.21.15.119:0/359015211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22ac0098a0 tx=0x7f22ac009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 -- 172.21.15.119:0/359015211 shutdown_connections 2024-03-28T18:32:48.442 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 --2- 172.21.15.119:0/359015211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc07c2d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 -- 172.21.15.119:0/359015211 >> 172.21.15.119:0/359015211 conn(0x7f22bc1030f0 msgr2=0x7f22bc105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 -- 172.21.15.119:0/359015211 shutdown_connections 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.473+0000 7f22c1e00700 1 -- 172.21.15.119:0/359015211 wait complete. 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.474+0000 7f22c1e00700 1 Processor -- start 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.474+0000 7f22c1e00700 1 -- start start 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22c1e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22c1e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc10e0e0 con 0x7f22bc07de70 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22baa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22baa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36338/0 (socket says 172.21.15.119:36338) 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22baa00700 1 -- 172.21.15.119:0/1292996316 learned_addr learned my addr 172.21.15.119:0/1292996316 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.475+0000 7f22baa00700 1 -- 172.21.15.119:0/1292996316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc19e250 con 0x7f22bc07de70 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22baa00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22ac011850 tx=0x7f22ac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac003af0 con 0x7f22bc07de70 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22ac020070 con 0x7f22bc07de70 2024-03-28T18:32:48.443 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc19e4e0 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22ac0195a0 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.476+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc1a11c0 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.477+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc069260 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.477+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f22ac019850 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.478+0000 7f22c0a00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 0x7f22a003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.478+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f22ac011e00 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.478+0000 7f22ba000700 1 -- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 msgr2=0x7f22a003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.478+0000 7f22ba000700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 0x7f22a003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.484+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f22ac00b480 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.679+0000 7f22ba000700 1 -- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 msgr2=0x7f22a003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.679+0000 7f22ba000700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 0x7f22a003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.766+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f22bc1a1390 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.767+0000 7f22c0a00700 1 -- 172.21.15.119:0/1292996316 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f22bc1a1390 con 0x7f22bc07de70 2024-03-28T18:32:48.444 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 msgr2=0x7f22a003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 0x7f22a003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 msgr2=0x7f22bc19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22ac011850 tx=0x7f22ac011c10 comp rx=0 tx=0).stop 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 shutdown_connections 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f22a0038f70 0x7f22a003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 --2- 172.21.15.119:0/1292996316 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22bc07de70 0x7f22bc19dd10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.771+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 >> 172.21.15.119:0/1292996316 conn(0x7f22bc1030f0 msgr2=0x7f22bc103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.772+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 shutdown_connections 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:47.772+0000 7f22c1e00700 1 -- 172.21.15.119:0/1292996316 wait complete. 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-03-28T18:32:48.445 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 5... 2024-03-28T18:32:48.724 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.450+0000 7f74f7082200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:32:48.984 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.795+0000 7f74f7082200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:32:48.985 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:48.937+0000 7f74f7082200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:32:49.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:49.050+0000 7f74f7082200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:32:49.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:49.171+0000 7f74f7082200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:32:49.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:49.387+0000 7f74f7082200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:32:50.807 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:50.637+0000 7f74f7082200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:32:50.807 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:50.806+0000 7f74f7082200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:32:51.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:51.221+0000 7f74f7082200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:32:51.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:51.338+0000 7f74f7082200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:32:51.998 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:51.592+0000 7f74f7082200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:32:51.998 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:51.834+0000 7f74f7082200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:32:52.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:52.205+0000 7f74f7082200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:32:52.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:52.521+0000 7f74f7082200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:32:53.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:52.819+0000 7f74f7082200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:32:54.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:53.831+0000 7f74f7082200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:32:54.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:54.519+0000 7f74f7082200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:32:54.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:54.648+0000 7f74f7082200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:32:55.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:32:54.926+0000 7f74f7082200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: Active manager daemon x restarted 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: Activating manager daemon x 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: mgrmap e6: x(active, starting, since 0.00655365s) 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: Manager daemon x is now available 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:32:55.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:54 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: Found migration_current of "None". Setting to last migration. 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:32:56.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:55 smithi119 ceph-mon[31813]: mgrmap e7: x(active, since 1.01709s) 2024-03-28T18:32:56.601 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.984+0000 7f92c0400700 1 Processor -- start 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.984+0000 7f92c0400700 1 -- start start 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.985+0000 7f92c0400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.985+0000 7f92c0400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b807c8a0 con 0x7f92b807de70 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.985+0000 7f92bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.985+0000 7f92bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42606/0 (socket says 172.21.15.119:42606) 2024-03-28T18:32:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.985+0000 7f92bda00700 1 -- 172.21.15.119:0/739995988 learned_addr learned my addr 172.21.15.119:0/739995988 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.986+0000 7f92bda00700 1 -- 172.21.15.119:0/739995988 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b807ca20 con 0x7f92b807de70 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.986+0000 7f92bda00700 1 --2- 172.21.15.119:0/739995988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098a0 tx=0x7f92ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94baa676812525db server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.986+0000 7f92b7e00700 1 -- 172.21.15.119:0/739995988 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92ac004160 con 0x7f92b807de70 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.986+0000 7f92b7e00700 1 -- 172.21.15.119:0/739995988 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92ac004300 con 0x7f92b807de70 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.986+0000 7f92b7e00700 1 -- 172.21.15.119:0/739995988 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92ac011b50 con 0x7f92b807de70 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 -- 172.21.15.119:0/739995988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 msgr2=0x7f92b807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 --2- 172.21.15.119:0/739995988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0098a0 tx=0x7f92ac009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 -- 172.21.15.119:0/739995988 shutdown_connections 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 --2- 172.21.15.119:0/739995988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b807c2d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 -- 172.21.15.119:0/739995988 >> 172.21.15.119:0/739995988 conn(0x7f92b81030f0 msgr2=0x7f92b8105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 -- 172.21.15.119:0/739995988 shutdown_connections 2024-03-28T18:32:56.607 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.987+0000 7f92c0400700 1 -- 172.21.15.119:0/739995988 wait complete. 2024-03-28T18:32:56.607 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.988+0000 7f92c0400700 1 Processor -- start 2024-03-28T18:32:56.607 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.988+0000 7f92c0400700 1 -- start start 2024-03-28T18:32:56.607 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92c0400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92c0400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b810e0e0 con 0x7f92b807de70 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42614/0 (socket says 172.21.15.119:42614) 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92bda00700 1 -- 172.21.15.119:0/3788391804 learned_addr learned my addr 172.21.15.119:0/3788391804 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.989+0000 7f92bda00700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b819e250 con 0x7f92b807de70 2024-03-28T18:32:56.613 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92bda00700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f92ac011850 tx=0x7f92ac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92ac003af0 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92ac020070 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92ac0195a0 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b819e4e0 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.990+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b81a11c0 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.991+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f92ac0198a0 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.992+0000 7f92b6000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.992+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a003baf0 con 0x7f92a0038f20 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.992+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f92ac011e00 con 0x7f92b807de70 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.992+0000 7f92bd000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:48.992+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:49.193+0000 7f92bd000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:49.193+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:49.594+0000 7f92bd000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:56.614 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:49.594+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:50.395+0000 7f92bd000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:50.395+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:51.997+0000 7f92bd000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3103230972 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:51.997+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:54.934+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f92ac019be0 con 0x7f92b807de70 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:54.935+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 msgr2=0x7f92a003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:54.935+0000 7f92b6000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.940+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f92ac04ebb0 con 0x7f92b807de70 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.941+0000 7f92b6000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 0x7f92a003ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.941+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92ac03bb70 con 0x7f92a003c810 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.942+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 0x7f92a003ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.943+0000 7f92bd000700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 0x7f92a003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f92a4003a10 tx=0x7f92a4009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.945+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f92ac03bb70 con 0x7f92a003c810 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.953+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f92b8069260 con 0x7f92a003c810 2024-03-28T18:32:56.615 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.954+0000 7f92b6000700 1 -- 172.21.15.119:0/3788391804 <== mgr.14120 v2:172.21.15.119:6800/3359592084 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f92b8069260 con 0x7f92a003c810 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 msgr2=0x7f92a003ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 0x7f92a003ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f92a4003a10 tx=0x7f92a4009290 comp rx=0 tx=0).stop 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 msgr2=0x7f92b819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f92ac011850 tx=0x7f92ac011c10 comp rx=0 tx=0).stop 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 shutdown_connections 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f92a003c810 0x7f92a003ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b807de70 0x7f92b819dd10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 --2- 172.21.15.119:0/3788391804 >> [v2:172.21.15.119:6800/3103230972,v1:172.21.15.119:6801/3103230972] conn(0x7f92a0038f20 0x7f92a003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 >> 172.21.15.119:0/3788391804 conn(0x7f92b81030f0 msgr2=0x7f92b8103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 shutdown_connections 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:55.955+0000 7f92c0400700 1 -- 172.21.15.119:0/3788391804 wait complete. 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 5 is available 2024-03-28T18:32:56.616 INFO:teuthology.orchestra.run.smithi119.stdout:Setting orchestrator backend to cephadm... 2024-03-28T18:32:57.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:56 smithi119 ceph-mon[31813]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T18:32:57.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:56 smithi119 ceph-mon[31813]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T18:32:57.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:56 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: [28/Mar/2024:18:32:56] ENGINE Bus STARTING 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: [28/Mar/2024:18:32:56] ENGINE Serving on http://172.21.15.119:8765 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: [28/Mar/2024:18:32:56] ENGINE Serving on https://172.21.15.119:7150 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: [28/Mar/2024:18:32:56] ENGINE Bus STARTED 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: mgrmap e8: x(active, since 2s) 2024-03-28T18:32:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:32:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:57 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:32:58.095 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.216+0000 7f3d3c600700 1 Processor -- start 2024-03-28T18:32:58.095 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.217+0000 7f3d3c600700 1 -- start start 2024-03-28T18:32:58.095 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.217+0000 7f3d3c600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.217+0000 7f3d3c600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3410a700 con 0x7f3d34109d10 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.217+0000 7f3d39c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.218+0000 7f3d39c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42682/0 (socket says 172.21.15.119:42682) 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.218+0000 7f3d39c00700 1 -- 172.21.15.119:0/3440297369 learned_addr learned my addr 172.21.15.119:0/3440297369 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.218+0000 7f3d39c00700 1 -- 172.21.15.119:0/3440297369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3410af10 con 0x7f3d34109d10 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.218+0000 7f3d39c00700 1 --2- 172.21.15.119:0/3440297369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009e80 tx=0x7f3d2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3496d65563486e75 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d33e00700 1 -- 172.21.15.119:0/3440297369 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d28017070 con 0x7f3d34109d10 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d33e00700 1 -- 172.21.15.119:0/3440297369 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d2800b8c0 con 0x7f3d34109d10 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d33e00700 1 -- 172.21.15.119:0/3440297369 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d280045c0 con 0x7f3d34109d10 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d3c600700 1 -- 172.21.15.119:0/3440297369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 msgr2=0x7f3d3410a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d3c600700 1 --2- 172.21.15.119:0/3440297369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009e80 tx=0x7f3d2800b330 comp rx=0 tx=0).stop 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d3c600700 1 -- 172.21.15.119:0/3440297369 shutdown_connections 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d3c600700 1 --2- 172.21.15.119:0/3440297369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3410a130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:58.096 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.219+0000 7f3d3c600700 1 -- 172.21.15.119:0/3440297369 >> 172.21.15.119:0/3440297369 conn(0x7f3d3407d100 msgr2=0x7f3d3407d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.220+0000 7f3d3c600700 1 -- 172.21.15.119:0/3440297369 shutdown_connections 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.220+0000 7f3d3c600700 1 -- 172.21.15.119:0/3440297369 wait complete. 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.220+0000 7f3d3c600700 1 Processor -- start 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d3c600700 1 -- start start 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d3c600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d3c600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3410e2c0 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d39c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d39c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42696/0 (socket says 172.21.15.119:42696) 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.221+0000 7f3d39c00700 1 -- 172.21.15.119:0/942228801 learned_addr learned my addr 172.21.15.119:0/942228801 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d39c00700 1 -- 172.21.15.119:0/942228801 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3419e340 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d39c00700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009530 tx=0x7f3d2800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d28020020 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d28003b50 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d28017040 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3419e5d0 con 0x7f3d34109d10 2024-03-28T18:32:58.097 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.222+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3419f200 con 0x7f3d34109d10 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.223+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3d28023020 con 0x7f3d34109d10 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.223+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d340637c0 con 0x7f3d34109d10 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.224+0000 7f3d32000700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 0x7f3d1c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.224+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3d2800f050 con 0x7f3d34109d10 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.224+0000 7f3d39200700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 0x7f3d1c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.225+0000 7f3d39200700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 0x7f3d1c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d20006fd0 tx=0x7f3d20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.229+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d2802a020 con 0x7f3d34109d10 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.466+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d34107ba0 con 0x7f3d1c038f70 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.482+0000 7f3d32000700 1 -- 172.21.15.119:0/942228801 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3d34107ba0 con 0x7f3d1c038f70 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.485+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 msgr2=0x7f3d1c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.485+0000 7f3d3c600700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 0x7f3d1c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3d20006fd0 tx=0x7f3d20006e40 comp rx=0 tx=0).stop 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.485+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 msgr2=0x7f3d3419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:58.098 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.485+0000 7f3d3c600700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009530 tx=0x7f3d2800bd50 comp rx=0 tx=0).stop 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 shutdown_connections 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f3d1c038f70 0x7f3d1c03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 --2- 172.21.15.119:0/942228801 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d34109d10 0x7f3d3419de00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 >> 172.21.15.119:0/942228801 conn(0x7f3d3407d100 msgr2=0x7f3d34107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 shutdown_connections 2024-03-28T18:32:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:57.486+0000 7f3d3c600700 1 -- 172.21.15.119:0/942228801 wait complete. 2024-03-28T18:32:59.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:58 smithi119 ceph-mon[31813]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:32:59.564 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.648+0000 7f13bf800700 1 Processor -- start 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bf800700 1 -- start start 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bf800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bf800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b807c8a0 con 0x7f13b807de20 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42706/0 (socket says 172.21.15.119:42706) 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.649+0000 7f13bce00700 1 -- 172.21.15.119:0/1750307420 learned_addr learned my addr 172.21.15.119:0/1750307420 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.650+0000 7f13bce00700 1 -- 172.21.15.119:0/1750307420 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b807ca20 con 0x7f13b807de20 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.650+0000 7f13bce00700 1 --2- 172.21.15.119:0/1750307420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0098a0 tx=0x7f13ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e3d4dbabc987a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.650+0000 7f13b7400700 1 -- 172.21.15.119:0/1750307420 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13ac004160 con 0x7f13b807de20 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13b7400700 1 -- 172.21.15.119:0/1750307420 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13ac004300 con 0x7f13b807de20 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13b7400700 1 -- 172.21.15.119:0/1750307420 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13ac011b50 con 0x7f13b807de20 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13bf800700 1 -- 172.21.15.119:0/1750307420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 msgr2=0x7f13b807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:59.565 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13bf800700 1 --2- 172.21.15.119:0/1750307420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13ac0098a0 tx=0x7f13ac009bb0 comp rx=0 tx=0).stop 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13bf800700 1 -- 172.21.15.119:0/1750307420 shutdown_connections 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13bf800700 1 --2- 172.21.15.119:0/1750307420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b807c2d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.651+0000 7f13bf800700 1 -- 172.21.15.119:0/1750307420 >> 172.21.15.119:0/1750307420 conn(0x7f13b8103140 msgr2=0x7f13b8105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.652+0000 7f13bf800700 1 -- 172.21.15.119:0/1750307420 shutdown_connections 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.652+0000 7f13bf800700 1 -- 172.21.15.119:0/1750307420 wait complete. 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bf800700 1 Processor -- start 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bf800700 1 -- start start 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bf800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bf800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b810e0d0 con 0x7f13b807de20 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.653+0000 7f13bce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42718/0 (socket says 172.21.15.119:42718) 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13bce00700 1 -- 172.21.15.119:0/1637759895 learned_addr learned my addr 172.21.15.119:0/1637759895 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13bce00700 1 -- 172.21.15.119:0/1637759895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b819e240 con 0x7f13b807de20 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13bce00700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13ac011b60 tx=0x7f13ac011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:59.566 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13ac003970 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13ac003b10 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b819e4d0 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.654+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13ac024610 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.655+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b81a11b0 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.656+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f13ac01a9f0 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.656+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b8069260 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.656+0000 7f13b5600700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 0x7f13a003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.656+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f13ac011e00 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.656+0000 7f13b7e00700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 0x7f13a003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.657+0000 7f13b7e00700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 0x7f13a003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.663+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f13ac019e10 con 0x7f13b807de20 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.885+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f13b81043c0 con 0x7f13a0038f70 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.886+0000 7f13b5600700 1 -- 172.21.15.119:0/1637759895 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f13b81043c0 con 0x7f13a0038f70 2024-03-28T18:32:59.567 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 msgr2=0x7f13a003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 0x7f13a003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4006e40 comp rx=0 tx=0).stop 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 msgr2=0x7f13b819dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13ac011b60 tx=0x7f13ac011c40 comp rx=0 tx=0).stop 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 shutdown_connections 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f13a0038f70 0x7f13a003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 --2- 172.21.15.119:0/1637759895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13b807de20 0x7f13b819dd00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.889+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 >> 172.21.15.119:0/1637759895 conn(0x7f13b8103140 msgr2=0x7f13b8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.890+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 shutdown_connections 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:32:58.890+0000 7f13bf800700 1 -- 172.21.15.119:0/1637759895 wait complete. 2024-03-28T18:32:59.568 INFO:teuthology.orchestra.run.smithi119.stdout:Generating ssh key... 2024-03-28T18:33:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:32:59 smithi119 ceph-mon[31813]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:00.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: Generating public/private rsa key pair. 2024-03-28T18:33:00.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: Your identification has been saved in /tmp/tmp2nq_6x74/key. 2024-03-28T18:33:00.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: Your public key has been saved in /tmp/tmp2nq_6x74/key.pub. 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: The key fingerprint is: 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: SHA256:eXqYquRNLuM2Bd/fmz8Tnyy4IcsmzQFZFGloQlVdjUs ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: The key's randomart image is: 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: +---[RSA 3072]----+ 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | ....+++ ..o | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | . o + . E . | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | o + . . | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | . o . . | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | o .S . | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | o .* . | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | ... *ooo. .o.| 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | o=+ o.=+.oooo.| 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: | o+=+ oo .+ooo | 2024-03-28T18:33:00.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: +----[SHA256]-----+ 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.101+0000 7f6f4fe00700 1 Processor -- start 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.101+0000 7f6f4fe00700 1 -- start start 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.102+0000 7f6f4fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.102+0000 7f6f4fe00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4807c8a0 con 0x7f6f4807de70 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.102+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.102+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37452/0 (socket says 172.21.15.119:37452) 2024-03-28T18:33:01.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.102+0000 7f6f4d400700 1 -- 172.21.15.119:0/1758360379 learned_addr learned my addr 172.21.15.119:0/1758360379 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.103+0000 7f6f4d400700 1 -- 172.21.15.119:0/1758360379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4807ca20 con 0x7f6f4807de70 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.103+0000 7f6f4d400700 1 --2- 172.21.15.119:0/1758360379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0098a0 tx=0x7f6f3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f217ac4a182bab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.103+0000 7f6f47e00700 1 -- 172.21.15.119:0/1758360379 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f3c004160 con 0x7f6f4807de70 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.103+0000 7f6f47e00700 1 -- 172.21.15.119:0/1758360379 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f3c004300 con 0x7f6f4807de70 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.103+0000 7f6f47e00700 1 -- 172.21.15.119:0/1758360379 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f3c011b50 con 0x7f6f4807de70 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 -- 172.21.15.119:0/1758360379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 msgr2=0x7f6f4807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/1758360379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0098a0 tx=0x7f6f3c009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 -- 172.21.15.119:0/1758360379 shutdown_connections 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/1758360379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4807c2d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 -- 172.21.15.119:0/1758360379 >> 172.21.15.119:0/1758360379 conn(0x7f6f481030f0 msgr2=0x7f6f48105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 -- 172.21.15.119:0/1758360379 shutdown_connections 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.104+0000 7f6f4fe00700 1 -- 172.21.15.119:0/1758360379 wait complete. 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.105+0000 7f6f4fe00700 1 Processor -- start 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.105+0000 7f6f4fe00700 1 -- start start 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4fe00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4810e0e0 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37466/0 (socket says 172.21.15.119:37466) 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4d400700 1 -- 172.21.15.119:0/640468595 learned_addr learned my addr 172.21.15.119:0/640468595 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.106+0000 7f6f4d400700 1 -- 172.21.15.119:0/640468595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4819e250 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f4d400700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c011850 tx=0x7f6f3c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f3c003af0 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f3c020070 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4819e4e0 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f3c0195a0 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.107+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f481a11c0 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.108+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f48069260 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.108+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6f3c019850 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.109+0000 7f6f46000700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 0x7f6f3003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.109+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6f3c011e00 con 0x7f6f4807de70 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.109+0000 7f6f4ca00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 0x7f6f3003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:01.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.110+0000 7f6f4ca00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 0x7f6f3003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006fd0 tx=0x7f6f34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.114+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f3c019b40 con 0x7f6f4807de70 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.339+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f48104370 con 0x7f6f30038f70 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.518+0000 7f6f46000700 1 -- 172.21.15.119:0/640468595 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f48104370 con 0x7f6f30038f70 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 msgr2=0x7f6f3003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 0x7f6f3003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006fd0 tx=0x7f6f34006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 msgr2=0x7f6f4819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c011850 tx=0x7f6f3c011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 shutdown_connections 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f6f30038f70 0x7f6f3003b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 --2- 172.21.15.119:0/640468595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4807de70 0x7f6f4819dd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 >> 172.21.15.119:0/640468595 conn(0x7f6f481030f0 msgr2=0x7f6f48103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 shutdown_connections 2024-03-28T18:33:01.262 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:00.522+0000 7f6f4fe00700 1 -- 172.21.15.119:0/640468595 wait complete. 2024-03-28T18:33:01.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:01 smithi119 ceph-mon[31813]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:01.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:01 smithi119 ceph-mon[31813]: Generating ssh key... 2024-03-28T18:33:01.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:01 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:01.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:01 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:02.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:02 smithi119 ceph-mon[31813]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:02.806 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.809+0000 7f4502800700 1 Processor -- start 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.809+0000 7f4502800700 1 -- start start 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.809+0000 7f4502800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.809+0000 7f4502800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc07c8a0 con 0x7f44fc07de70 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.810+0000 7f44fb400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.810+0000 7f44fb400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37480/0 (socket says 172.21.15.119:37480) 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.810+0000 7f44fb400700 1 -- 172.21.15.119:0/3338370333 learned_addr learned my addr 172.21.15.119:0/3338370333 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.811+0000 7f44fb400700 1 -- 172.21.15.119:0/3338370333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc07ca20 con 0x7f44fc07de70 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.811+0000 7f44fb400700 1 --2- 172.21.15.119:0/3338370333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0098a0 tx=0x7f44ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5775b28883b0fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.811+0000 7f44fa000700 1 -- 172.21.15.119:0/3338370333 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44ec004160 con 0x7f44fc07de70 2024-03-28T18:33:02.813 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.811+0000 7f44fa000700 1 -- 172.21.15.119:0/3338370333 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ec004300 con 0x7f44fc07de70 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.811+0000 7f44fa000700 1 -- 172.21.15.119:0/3338370333 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44ec011b50 con 0x7f44fc07de70 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 -- 172.21.15.119:0/3338370333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 msgr2=0x7f44fc07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 --2- 172.21.15.119:0/3338370333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0098a0 tx=0x7f44ec009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 -- 172.21.15.119:0/3338370333 shutdown_connections 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 --2- 172.21.15.119:0/3338370333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc07c2d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 -- 172.21.15.119:0/3338370333 >> 172.21.15.119:0/3338370333 conn(0x7f44fc1030f0 msgr2=0x7f44fc105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 -- 172.21.15.119:0/3338370333 shutdown_connections 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.812+0000 7f4502800700 1 -- 172.21.15.119:0/3338370333 wait complete. 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.813+0000 7f4502800700 1 Processor -- start 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.813+0000 7f4502800700 1 -- start start 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f4502800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:02.814 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f4502800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc10e0e0 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f44fb400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f44fb400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37492/0 (socket says 172.21.15.119:37492) 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f44fb400700 1 -- 172.21.15.119:0/1634550367 learned_addr learned my addr 172.21.15.119:0/1634550367 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.814+0000 7f44fb400700 1 -- 172.21.15.119:0/1634550367 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc19e250 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f44fb400700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44ec011850 tx=0x7f44ec011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44ec003af0 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44ec020070 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc19e4e0 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44ec0195a0 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.815+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc1a11c0 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.816+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc069260 con 0x7f44fc07de70 2024-03-28T18:33:02.815 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.817+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f44ec0198a0 con 0x7f44fc07de70 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.817+0000 7f44f3e00700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 0x7f44e003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.817+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f44ec011e00 con 0x7f44fc07de70 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.817+0000 7f44faa00700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 0x7f44e003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.818+0000 7f44faa00700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 0x7f44e003b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f44e8006fd0 tx=0x7f44e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:01.822+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44ec019b90 con 0x7f44fc07de70 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.044+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f44fc104370 con 0x7f44e0038f20 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.045+0000 7f44f3e00700 1 -- 172.21.15.119:0/1634550367 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f44fc104370 con 0x7f44e0038f20 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 msgr2=0x7f44e003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 0x7f44e003b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f44e8006fd0 tx=0x7f44e8006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 msgr2=0x7f44fc19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44ec011850 tx=0x7f44ec011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:02.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 shutdown_connections 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f44e0038f20 0x7f44e003b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 --2- 172.21.15.119:0/1634550367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44fc07de70 0x7f44fc19dd10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 >> 172.21.15.119:0/1634550367 conn(0x7f44fc1030f0 msgr2=0x7f44fc103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 shutdown_connections 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:02.048+0000 7f4502800700 1 -- 172.21.15.119:0/1634550367 wait complete. 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T18:33:02.817 INFO:teuthology.orchestra.run.smithi119.stdout:Adding host smithi119... 2024-03-28T18:33:04.759 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:04 smithi119 ceph-mon[31813]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:05.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:05 smithi119 ceph-mon[31813]: Deploying cephadm binary to smithi119 2024-03-28T18:33:10.640 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Added host 'smithi119' with addr '172.21.15.119' 2024-03-28T18:33:10.640 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.403+0000 7fe0f6800700 1 Processor -- start 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.403+0000 7fe0f6800700 1 -- start start 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.404+0000 7fe0f6800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.404+0000 7fe0f6800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f0107420 con 0x7fe0f0106a30 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.404+0000 7fe0ef400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.404+0000 7fe0ef400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37500/0 (socket says 172.21.15.119:37500) 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.404+0000 7fe0ef400700 1 -- 172.21.15.119:0/3544961141 learned_addr learned my addr 172.21.15.119:0/3544961141 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.405+0000 7fe0ef400700 1 -- 172.21.15.119:0/3544961141 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0107c30 con 0x7fe0f0106a30 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.405+0000 7fe0ef400700 1 --2- 172.21.15.119:0/3544961141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00098a0 tx=0x7fe0e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b28620d03d95c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.405+0000 7fe0ee000700 1 -- 172.21.15.119:0/3544961141 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e0004160 con 0x7fe0f0106a30 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.405+0000 7fe0ee000700 1 -- 172.21.15.119:0/3544961141 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0e0004300 con 0x7fe0f0106a30 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.405+0000 7fe0ee000700 1 -- 172.21.15.119:0/3544961141 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e0011b50 con 0x7fe0f0106a30 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 -- 172.21.15.119:0/3544961141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 msgr2=0x7fe0f0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 --2- 172.21.15.119:0/3544961141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00098a0 tx=0x7fe0e0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:10.641 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 -- 172.21.15.119:0/3544961141 shutdown_connections 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 --2- 172.21.15.119:0/3544961141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0106e50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 -- 172.21.15.119:0/3544961141 >> 172.21.15.119:0/3544961141 conn(0x7fe0f0101ff0 msgr2=0x7fe0f0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 -- 172.21.15.119:0/3544961141 shutdown_connections 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.406+0000 7fe0f6800700 1 -- 172.21.15.119:0/3544961141 wait complete. 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.407+0000 7fe0f6800700 1 Processor -- start 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.407+0000 7fe0f6800700 1 -- start start 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0f6800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0f6800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f010afe0 con 0x7fe0f0106a30 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0ef400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0ef400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37506/0 (socket says 172.21.15.119:37506) 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0ef400700 1 -- 172.21.15.119:0/148895870 learned_addr learned my addr 172.21.15.119:0/148895870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.408+0000 7fe0ef400700 1 -- 172.21.15.119:0/148895870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0190310 con 0x7fe0f0106a30 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0ef400700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0011670 tx=0x7fe0e0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e00039c0 con 0x7fe0f0106a30 2024-03-28T18:33:10.642 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0e0011e50 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0e001a5c0 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f01905a0 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.409+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f0190a80 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.410+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe0e0024430 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.410+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f00637c0 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.411+0000 7fe0e7e00700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 0x7fe0d403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.411+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe0e004d220 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.411+0000 7fe0eea00700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 0x7fe0d403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.412+0000 7fe0eea00700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 0x7fe0d403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006fd0 tx=0x7fe0dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.416+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe0e002d3f0 con 0x7fe0f0106a30 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:03.640+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0f0190f00 con 0x7fe0d4038f20 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.943+0000 7fe0e7e00700 1 -- 172.21.15.119:0/148895870 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe0f0190f00 con 0x7fe0d4038f20 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 msgr2=0x7fe0d403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:10.643 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 0x7fe0d403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc006fd0 tx=0x7fe0dc006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 msgr2=0x7fe0f0191dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0011670 tx=0x7fe0e0011b60 comp rx=0 tx=0).stop 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 shutdown_connections 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe0d4038f20 0x7fe0d403b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 --2- 172.21.15.119:0/148895870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0f0106a30 0x7fe0f0191dc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.947+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 >> 172.21.15.119:0/148895870 conn(0x7fe0f0101ff0 msgr2=0x7fe0f0102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.948+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 shutdown_connections 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:09.948+0000 7fe0f6800700 1 -- 172.21.15.119:0/148895870 wait complete. 2024-03-28T18:33:10.644 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mon service... 2024-03-28T18:33:11.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:10 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:11.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:10 smithi119 ceph-mon[31813]: Added host smithi119 2024-03-28T18:33:11.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:10 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:12.553 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:12 smithi119 ceph-mon[31813]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:12.553 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:12 smithi119 ceph-mon[31813]: Saving service mon spec with placement count:5 2024-03-28T18:33:12.553 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:12 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:12.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.273+0000 7fa895600700 1 Processor -- start 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.274+0000 7fa895600700 1 -- start start 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.274+0000 7fa895600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.274+0000 7fa895600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89007c8c0 con 0x7fa89007de90 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.275+0000 7fa88ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.275+0000 7fa88ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41300/0 (socket says 172.21.15.119:41300) 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.275+0000 7fa88ea00700 1 -- 172.21.15.119:0/1470066096 learned_addr learned my addr 172.21.15.119:0/1470066096 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.275+0000 7fa88ea00700 1 -- 172.21.15.119:0/1470066096 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89007ca40 con 0x7fa89007de90 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.275+0000 7fa88ea00700 1 --2- 172.21.15.119:0/1470066096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8800098a0 tx=0x7fa880009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad6c164e70c5c71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa88d600700 1 -- 172.21.15.119:0/1470066096 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa880004160 con 0x7fa89007de90 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa88d600700 1 -- 172.21.15.119:0/1470066096 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa880004300 con 0x7fa89007de90 2024-03-28T18:33:12.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa88d600700 1 -- 172.21.15.119:0/1470066096 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa880011b50 con 0x7fa89007de90 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa895600700 1 -- 172.21.15.119:0/1470066096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 msgr2=0x7fa89007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa895600700 1 --2- 172.21.15.119:0/1470066096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa8800098a0 tx=0x7fa880009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.276+0000 7fa895600700 1 -- 172.21.15.119:0/1470066096 shutdown_connections 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.277+0000 7fa895600700 1 --2- 172.21.15.119:0/1470066096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa89007c2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.277+0000 7fa895600700 1 -- 172.21.15.119:0/1470066096 >> 172.21.15.119:0/1470066096 conn(0x7fa890103190 msgr2=0x7fa8901055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.277+0000 7fa895600700 1 -- 172.21.15.119:0/1470066096 shutdown_connections 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.277+0000 7fa895600700 1 -- 172.21.15.119:0/1470066096 wait complete. 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa895600700 1 Processor -- start 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa895600700 1 -- start start 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa895600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa895600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa890109d20 con 0x7fa89007de90 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa88ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:12.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa88ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41310/0 (socket says 172.21.15.119:41310) 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.278+0000 7fa88ea00700 1 -- 172.21.15.119:0/1298935556 learned_addr learned my addr 172.21.15.119:0/1298935556 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa88ea00700 1 -- 172.21.15.119:0/1298935556 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8901a2740 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa88ea00700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa880011850 tx=0x7fa880011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa880003af0 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa880020070 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8901a29d0 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8800195a0 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.279+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8901a34f0 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.280+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8900637c0 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.280+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa880019850 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.281+0000 7fa887400700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 0x7fa87403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.281+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa880011e00 con 0x7fa89007de90 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.281+0000 7fa88e000700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 0x7fa87403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.282+0000 7fa88e000700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 0x7fa87403b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:12.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.286+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa880019b40 con 0x7fa89007de90 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.544+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa8901a4240 con 0x7fa874038f70 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.553+0000 7fa887400700 1 -- 172.21.15.119:0/1298935556 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa8901a4240 con 0x7fa874038f70 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 msgr2=0x7fa87403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 0x7fa87403b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa87c006fd0 tx=0x7fa87c006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 msgr2=0x7fa8901a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa880011850 tx=0x7fa880011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 shutdown_connections 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa874038f70 0x7fa87403b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 --2- 172.21.15.119:0/1298935556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89007de90 0x7fa8901a2200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.555+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 >> 172.21.15.119:0/1298935556 conn(0x7fa890103190 msgr2=0x7fa890198e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.556+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 shutdown_connections 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:11.556+0000 7fa895600700 1 -- 172.21.15.119:0/1298935556 wait complete. 2024-03-28T18:33:12.841 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mgr service... 2024-03-28T18:33:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:14 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:14 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:14 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:14.756 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.783+0000 7fe955a00700 1 Processor -- start 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.783+0000 7fe955a00700 1 -- start start 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.783+0000 7fe955a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.783+0000 7fe955a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9501084c0 con 0x7fe950107ad0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.784+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.784+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41322/0 (socket says 172.21.15.119:41322) 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.784+0000 7fe94ea00700 1 -- 172.21.15.119:0/760178080 learned_addr learned my addr 172.21.15.119:0/760178080 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.784+0000 7fe94ea00700 1 -- 172.21.15.119:0/760178080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950108cd0 con 0x7fe950107ad0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.784+0000 7fe94ea00700 1 --2- 172.21.15.119:0/760178080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9400098a0 tx=0x7fe940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6dbcac0b475a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.785+0000 7fe94d600700 1 -- 172.21.15.119:0/760178080 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe940004160 con 0x7fe950107ad0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.785+0000 7fe94d600700 1 -- 172.21.15.119:0/760178080 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe940004300 con 0x7fe950107ad0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.785+0000 7fe94d600700 1 -- 172.21.15.119:0/760178080 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe940011b20 con 0x7fe950107ad0 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 -- 172.21.15.119:0/760178080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 msgr2=0x7fe950107ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 --2- 172.21.15.119:0/760178080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe9400098a0 tx=0x7fe940009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.757 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 -- 172.21.15.119:0/760178080 shutdown_connections 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 --2- 172.21.15.119:0/760178080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe950107ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 -- 172.21.15.119:0/760178080 >> 172.21.15.119:0/760178080 conn(0x7fe9501030f0 msgr2=0x7fe950105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 -- 172.21.15.119:0/760178080 shutdown_connections 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.786+0000 7fe955a00700 1 -- 172.21.15.119:0/760178080 wait complete. 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.787+0000 7fe955a00700 1 Processor -- start 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.787+0000 7fe955a00700 1 -- start start 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.787+0000 7fe955a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.787+0000 7fe955a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95010c080 con 0x7fe950107ad0 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.788+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.788+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41326/0 (socket says 172.21.15.119:41326) 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.788+0000 7fe94ea00700 1 -- 172.21.15.119:0/3344389011 learned_addr learned my addr 172.21.15.119:0/3344389011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.788+0000 7fe94ea00700 1 -- 172.21.15.119:0/3344389011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95019e370 con 0x7fe950107ad0 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.788+0000 7fe94ea00700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe9400117e0 tx=0x7fe940011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:14.758 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.789+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe940003bf0 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.789+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe94001a910 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.789+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95019e600 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.789+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9400249a0 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.789+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95019efe0 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.790+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe940022020 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.790+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9500637c0 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.790+0000 7fe947400700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 0x7fe93403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.791+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe94004d1f0 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.791+0000 7fe94e000700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 0x7fe93403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.791+0000 7fe94e000700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 0x7fe93403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:13.796+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe940011e20 con 0x7fe950107ad0 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.057+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe950104290 con 0x7fe934038f20 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.069+0000 7fe947400700 1 -- 172.21.15.119:0/3344389011 <== mgr.14120 v2:172.21.15.119:6800/3359592084 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe950104290 con 0x7fe934038f20 2024-03-28T18:33:14.759 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 msgr2=0x7fe93403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 0x7fe93403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe93c006fd0 tx=0x7fe93c006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 msgr2=0x7fe95019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe9400117e0 tx=0x7fe940011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 shutdown_connections 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fe934038f20 0x7fe93403b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 --2- 172.21.15.119:0/3344389011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe950107ad0 0x7fe95019de30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 >> 172.21.15.119:0/3344389011 conn(0x7fe9501030f0 msgr2=0x7fe950103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.072+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 shutdown_connections 2024-03-28T18:33:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:14.073+0000 7fe955a00700 1 -- 172.21.15.119:0/3344389011 wait complete. 2024-03-28T18:33:15.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:15 smithi119 ceph-mon[31813]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:15.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:15 smithi119 ceph-mon[31813]: Saving service mgr spec with placement count:2 2024-03-28T18:33:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:16 smithi119 ceph-mon[31813]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:16 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4252873311' entity='client.admin' 2024-03-28T18:33:16.370 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.368+0000 7fa29a800700 1 Processor -- start 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.368+0000 7fa29a800700 1 -- start start 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.368+0000 7fa29a800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.368+0000 7fa29a800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2941084e0 con 0x7fa294107af0 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.369+0000 7fa293400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.369+0000 7fa293400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41332/0 (socket says 172.21.15.119:41332) 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.369+0000 7fa293400700 1 -- 172.21.15.119:0/1008316312 learned_addr learned my addr 172.21.15.119:0/1008316312 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.369+0000 7fa293400700 1 -- 172.21.15.119:0/1008316312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa294108cf0 con 0x7fa294107af0 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.370+0000 7fa293400700 1 --2- 172.21.15.119:0/1008316312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2840098a0 tx=0x7fa284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea7aa3b0896ca85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:16.374 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.370+0000 7fa292000700 1 -- 172.21.15.119:0/1008316312 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa284004160 con 0x7fa294107af0 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.370+0000 7fa292000700 1 -- 172.21.15.119:0/1008316312 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa284004300 con 0x7fa294107af0 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.370+0000 7fa292000700 1 -- 172.21.15.119:0/1008316312 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa284011b20 con 0x7fa294107af0 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 -- 172.21.15.119:0/1008316312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 msgr2=0x7fa294107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 --2- 172.21.15.119:0/1008316312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa2840098a0 tx=0x7fa284009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 -- 172.21.15.119:0/1008316312 shutdown_connections 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 --2- 172.21.15.119:0/1008316312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa294107f10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 -- 172.21.15.119:0/1008316312 >> 172.21.15.119:0/1008316312 conn(0x7fa2941030f0 msgr2=0x7fa294105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 -- 172.21.15.119:0/1008316312 shutdown_connections 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.371+0000 7fa29a800700 1 -- 172.21.15.119:0/1008316312 wait complete. 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.372+0000 7fa29a800700 1 Processor -- start 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.372+0000 7fa29a800700 1 -- start start 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.372+0000 7fa29a800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.372+0000 7fa29a800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29410c0a0 con 0x7fa294107af0 2024-03-28T18:33:16.375 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.373+0000 7fa293400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.373+0000 7fa293400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41348/0 (socket says 172.21.15.119:41348) 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.373+0000 7fa293400700 1 -- 172.21.15.119:0/4252873311 learned_addr learned my addr 172.21.15.119:0/4252873311 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.373+0000 7fa293400700 1 -- 172.21.15.119:0/4252873311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29419e210 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.373+0000 7fa293400700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa284000c00 tx=0x7fa284011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.374+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa284003b60 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.374+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa28401a910 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.374+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa284024930 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.374+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29419e4a0 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.374+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29419e900 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.375+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa284022020 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.375+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa294069260 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.376+0000 7fa28be00700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 0x7fa27803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.376+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa28404d070 con 0x7fa294107af0 2024-03-28T18:33:16.376 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.376+0000 7fa292a00700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 0x7fa27803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:16.377 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.377+0000 7fa292a00700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 0x7fa27803b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa280006fd0 tx=0x7fa280006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.386+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa28402ad60 con 0x7fa294107af0 2024-03-28T18:33:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.620+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa2941a1180 con 0x7fa294107af0 2024-03-28T18:33:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.625+0000 7fa28be00700 1 -- 172.21.15.119:0/4252873311 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa284021d60 con 0x7fa294107af0 2024-03-28T18:33:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 msgr2=0x7fa27803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 0x7fa27803b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa280006fd0 tx=0x7fa280006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 msgr2=0x7fa29419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa284000c00 tx=0x7fa284011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 shutdown_connections 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa278038f20 0x7fa27803b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 --2- 172.21.15.119:0/4252873311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa294107af0 0x7fa29419dcd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.627+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 >> 172.21.15.119:0/4252873311 conn(0x7fa2941030f0 msgr2=0x7fa294103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.628+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 shutdown_connections 2024-03-28T18:33:16.379 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:15.628+0000 7fa29a800700 1 -- 172.21.15.119:0/4252873311 wait complete. 2024-03-28T18:33:18.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:17 smithi119 ceph-mon[31813]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:18.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:17 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:18.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:17 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1361828219' entity='client.admin' 2024-03-28T18:33:18.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.225+0000 7f7a72200700 1 Processor -- start 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.225+0000 7f7a72200700 1 -- start start 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.226+0000 7f7a72200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.226+0000 7f7a72200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c07c8c0 con 0x7f7a6c07de90 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.226+0000 7f7a6b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.226+0000 7f7a6b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41364/0 (socket says 172.21.15.119:41364) 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.226+0000 7f7a6b400700 1 -- 172.21.15.119:0/155703831 learned_addr learned my addr 172.21.15.119:0/155703831 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.227+0000 7f7a6b400700 1 -- 172.21.15.119:0/155703831 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c07ca40 con 0x7f7a6c07de90 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.227+0000 7f7a6b400700 1 --2- 172.21.15.119:0/155703831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009e80 tx=0x7f7a5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab42681c7c8b928 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.227+0000 7f7a6aa00700 1 -- 172.21.15.119:0/155703831 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a5c017070 con 0x7f7a6c07de90 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.227+0000 7f7a6aa00700 1 -- 172.21.15.119:0/155703831 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c00b8c0 con 0x7f7a6c07de90 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.227+0000 7f7a6aa00700 1 -- 172.21.15.119:0/155703831 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a5c0045c0 con 0x7f7a6c07de90 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 -- 172.21.15.119:0/155703831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 msgr2=0x7f7a6c07c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:18.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 --2- 172.21.15.119:0/155703831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009e80 tx=0x7f7a5c00b330 comp rx=0 tx=0).stop 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 -- 172.21.15.119:0/155703831 shutdown_connections 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 --2- 172.21.15.119:0/155703831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c07c2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 -- 172.21.15.119:0/155703831 >> 172.21.15.119:0/155703831 conn(0x7f7a6c1030f0 msgr2=0x7f7a6c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 -- 172.21.15.119:0/155703831 shutdown_connections 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.228+0000 7f7a72200700 1 -- 172.21.15.119:0/155703831 wait complete. 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.229+0000 7f7a72200700 1 Processor -- start 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.229+0000 7f7a72200700 1 -- start start 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a72200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a72200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c109d90 con 0x7f7a6c07de90 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a6b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a6b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41376/0 (socket says 172.21.15.119:41376) 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a6b400700 1 -- 172.21.15.119:0/1361828219 learned_addr learned my addr 172.21.15.119:0/1361828219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.230+0000 7f7a6b400700 1 -- 172.21.15.119:0/1361828219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c1a2700 con 0x7f7a6c07de90 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a6b400700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009fd0 tx=0x7f7a5c00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:18.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a5c020020 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c012430 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a5c017040 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c1a2990 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.231+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c1a34b0 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.232+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c0637c0 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.232+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7a5c023020 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.233+0000 7f7a68c00700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 0x7f7a5003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.233+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7a5c00f050 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.233+0000 7f7a62a00700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 0x7f7a5003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.234+0000 7f7a62a00700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 0x7f7a5003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006fd0 tx=0x7f7a54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.238+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a5c02a020 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.459+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7a6c1a4210 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.466+0000 7f7a68c00700 1 -- 172.21.15.119:0/1361828219 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7a5c011e70 con 0x7f7a6c07de90 2024-03-28T18:33:18.130 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.469+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 msgr2=0x7f7a5003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.469+0000 7f7a72200700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 0x7f7a5003b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7a54006fd0 tx=0x7f7a54006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 msgr2=0x7f7a6c1a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009fd0 tx=0x7f7a5c00bde0 comp rx=0 tx=0).stop 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 shutdown_connections 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7f7a50038f70 0x7f7a5003b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 --2- 172.21.15.119:0/1361828219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a6c07de90 0x7f7a6c1a21c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 >> 172.21.15.119:0/1361828219 conn(0x7f7a6c1030f0 msgr2=0x7f7a6c198e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 shutdown_connections 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:17.470+0000 7f7a72200700 1 -- 172.21.15.119:0/1361828219 wait complete. 2024-03-28T18:33:18.131 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling the dashboard module... 2024-03-28T18:33:20.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:19 smithi119 ceph-mon[31813]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:20.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:19 smithi119 ceph-mon[31813]: from='mgr.14120 172.21.15.119:0/1357233771' entity='mgr.x' 2024-03-28T18:33:20.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:19 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/989290535' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T18:33:20.503 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setuser ceph since I am not root 2024-03-28T18:33:20.503 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setgroup ceph since I am not root 2024-03-28T18:33:21.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/989290535' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T18:33:21.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-mon[31813]: mgrmap e9: x(active, since 25s) 2024-03-28T18:33:21.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:20.812+0000 7f3cec8f3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:33:21.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:20.933+0000 7f3cec8f3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.733+0000 7fa9ea400700 1 Processor -- start 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.733+0000 7fa9ea400700 1 -- start start 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.733+0000 7fa9ea400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.733+0000 7fa9ea400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e407c860 con 0x7fa9e407de30 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.734+0000 7fa9e3400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.734+0000 7fa9e3400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41386/0 (socket says 172.21.15.119:41386) 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.734+0000 7fa9e3400700 1 -- 172.21.15.119:0/893522146 learned_addr learned my addr 172.21.15.119:0/893522146 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:21.449 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.734+0000 7fa9e3400700 1 -- 172.21.15.119:0/893522146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e407c9e0 con 0x7fa9e407de30 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.734+0000 7fa9e3400700 1 --2- 172.21.15.119:0/893522146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40098a0 tx=0x7fa9d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79797b48c735c2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.735+0000 7fa9e2000700 1 -- 172.21.15.119:0/893522146 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9d4004160 con 0x7fa9e407de30 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.735+0000 7fa9e2000700 1 -- 172.21.15.119:0/893522146 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4004300 con 0x7fa9e407de30 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.735+0000 7fa9e2000700 1 -- 172.21.15.119:0/893522146 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9d4011b20 con 0x7fa9e407de30 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 -- 172.21.15.119:0/893522146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 msgr2=0x7fa9e407c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 --2- 172.21.15.119:0/893522146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40098a0 tx=0x7fa9d4009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 -- 172.21.15.119:0/893522146 shutdown_connections 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 --2- 172.21.15.119:0/893522146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e407c290 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 -- 172.21.15.119:0/893522146 >> 172.21.15.119:0/893522146 conn(0x7fa9e41030f0 msgr2=0x7fa9e4105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 -- 172.21.15.119:0/893522146 shutdown_connections 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.736+0000 7fa9ea400700 1 -- 172.21.15.119:0/893522146 wait complete. 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.737+0000 7fa9ea400700 1 Processor -- start 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.737+0000 7fa9ea400700 1 -- start start 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.737+0000 7fa9ea400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:21.450 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.737+0000 7fa9ea400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e410e1b0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.738+0000 7fa9e3400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.738+0000 7fa9e3400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41392/0 (socket says 172.21.15.119:41392) 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.738+0000 7fa9e3400700 1 -- 172.21.15.119:0/989290535 learned_addr learned my addr 172.21.15.119:0/989290535 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.738+0000 7fa9e3400700 1 -- 172.21.15.119:0/989290535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e419e310 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.738+0000 7fa9e3400700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40117e0 tx=0x7fa9d4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.739+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9d4003ba0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.739+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9d401a910 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.739+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9d40248f0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.739+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e419e5a0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.739+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e419ef80 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.740+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e40637c0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.741+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa9d4022020 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.741+0000 7fa9dbe00700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 0x7fa9c803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.741+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa9d404d2a0 con 0x7fa9e407de30 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.742+0000 7fa9e2a00700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 0x7fa9c803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:21.451 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.742+0000 7fa9e2a00700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 0x7fa9c803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0006fd0 tx=0x7fa9d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:18.747+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa9d401ab70 con 0x7fa9e407de30 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.045+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa9e419f470 con 0x7fa9e407de30 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.991+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa9d4017ca0 con 0x7fa9e407de30 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.997+0000 7fa9dbe00700 1 -- 172.21.15.119:0/989290535 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fa9d402d3f0 con 0x7fa9e407de30 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.997+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 msgr2=0x7fa9c803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.997+0000 7fa9ea400700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 0x7fa9c803b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0006fd0 tx=0x7fa9d0006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.997+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 msgr2=0x7fa9e419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.997+0000 7fa9ea400700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40117e0 tx=0x7fa9d4011ba0 comp rx=0 tx=0).stop 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 shutdown_connections 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fa9c8038f70 0x7fa9c803b430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 --2- 172.21.15.119:0/989290535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9e407de30 0x7fa9e419ddd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 >> 172.21.15.119:0/989290535 conn(0x7fa9e41030f0 msgr2=0x7fa9e4103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 shutdown_connections 2024-03-28T18:33:21.452 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:19.998+0000 7fa9ea400700 1 -- 172.21.15.119:0/989290535 wait complete. 2024-03-28T18:33:21.976 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:21.728+0000 7f3cec8f3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:33:21.976 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:21.851+0000 7f3cec8f3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:33:22.278 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.034+0000 7f3cec8f3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:33:22.279 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.165+0000 7f3cec8f3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:33:22.279 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.277+0000 7f3cec8f3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:33:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3353656569' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T18:33:22.886 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.622+0000 7f3cec8f3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:33:22.887 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.774+0000 7f3cec8f3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:33:23.037 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:33:23.037 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T18:33:23.037 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T18:33:23.037 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.125+0000 7ff438600700 1 Processor -- start 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.126+0000 7ff438600700 1 -- start start 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.126+0000 7ff438600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.126+0000 7ff438600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43007c7d0 con 0x7ff43007dde0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.127+0000 7ff435c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.127+0000 7ff435c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41862/0 (socket says 172.21.15.119:41862) 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.127+0000 7ff435c00700 1 -- 172.21.15.119:0/886676654 learned_addr learned my addr 172.21.15.119:0/886676654 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.127+0000 7ff435c00700 1 -- 172.21.15.119:0/886676654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43007c950 con 0x7ff43007dde0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.127+0000 7ff435c00700 1 --2- 172.21.15.119:0/886676654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4240098a0 tx=0x7ff424009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ba0e66a92962ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.128+0000 7ff42fe00700 1 -- 172.21.15.119:0/886676654 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff424004160 con 0x7ff43007dde0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.128+0000 7ff42fe00700 1 -- 172.21.15.119:0/886676654 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff424004300 con 0x7ff43007dde0 2024-03-28T18:33:23.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.128+0000 7ff42fe00700 1 -- 172.21.15.119:0/886676654 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff424011b20 con 0x7ff43007dde0 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 -- 172.21.15.119:0/886676654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 msgr2=0x7ff43007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 --2- 172.21.15.119:0/886676654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4240098a0 tx=0x7ff424009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 -- 172.21.15.119:0/886676654 shutdown_connections 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 --2- 172.21.15.119:0/886676654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43007c290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 -- 172.21.15.119:0/886676654 >> 172.21.15.119:0/886676654 conn(0x7ff430103140 msgr2=0x7ff430105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 -- 172.21.15.119:0/886676654 shutdown_connections 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.129+0000 7ff438600700 1 -- 172.21.15.119:0/886676654 wait complete. 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.130+0000 7ff438600700 1 Processor -- start 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.130+0000 7ff438600700 1 -- start start 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.130+0000 7ff438600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.130+0000 7ff438600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43010e100 con 0x7ff43007dde0 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.131+0000 7ff435c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:23.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.131+0000 7ff435c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41878/0 (socket says 172.21.15.119:41878) 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.131+0000 7ff435c00700 1 -- 172.21.15.119:0/3353656569 learned_addr learned my addr 172.21.15.119:0/3353656569 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.131+0000 7ff435c00700 1 -- 172.21.15.119:0/3353656569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43019e1a0 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.131+0000 7ff435c00700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff424004660 tx=0x7ff424011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.132+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff424003ca0 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.132+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff42401a910 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.132+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43019e430 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.132+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff424024970 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.132+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43019e890 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.133+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff430069260 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.133+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ff424022020 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.134+0000 7ff42e000700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 0x7ff41803b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.134+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff42404d250 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.134+0000 7ff435200700 1 -- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 msgr2=0x7ff41803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.134+0000 7ff435200700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 0x7ff41803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.139+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff42402ddb0 con 0x7ff43007dde0 2024-03-28T18:33:23.040 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.335+0000 7ff435200700 1 -- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 msgr2=0x7ff41803b630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.335+0000 7ff435200700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 0x7ff41803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.416+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff4301a16d0 con 0x7ff43007dde0 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.417+0000 7ff42e000700 1 -- 172.21.15.119:0/3353656569 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff4240212c0 con 0x7ff43007dde0 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 msgr2=0x7ff41803b630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 0x7ff41803b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 msgr2=0x7ff43019dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff424004660 tx=0x7ff424011ce0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 shutdown_connections 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7ff418039170 0x7ff41803b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 --2- 172.21.15.119:0/3353656569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff43007dde0 0x7ff43019dc60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 >> 172.21.15.119:0/3353656569 conn(0x7ff430103140 msgr2=0x7ff430103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.419+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 shutdown_connections 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:22.420+0000 7ff438600700 1 -- 172.21.15.119:0/3353656569 wait complete. 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-03-28T18:33:23.041 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 9... 2024-03-28T18:33:23.193 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:22.885+0000 7f3cec8f3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:33:23.193 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:23.008+0000 7f3cec8f3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:33:23.456 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:23.244+0000 7f3cec8f3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:33:24.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:24.515+0000 7f3cec8f3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:33:24.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:24.685+0000 7f3cec8f3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:33:25.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:25.098+0000 7f3cec8f3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:33:25.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:25.216+0000 7f3cec8f3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:33:25.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:25.470+0000 7f3cec8f3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:33:25.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:25.712+0000 7f3cec8f3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:33:26.404 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:26.086+0000 7f3cec8f3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:33:26.404 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:26.402+0000 7f3cec8f3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:33:27.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:26.703+0000 7f3cec8f3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:33:28.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:27.716+0000 7f3cec8f3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:33:28.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:28.409+0000 7f3cec8f3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:33:28.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:28.537+0000 7f3cec8f3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:33:29.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:33:28.821+0000 7f3cec8f3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:33:29.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: Active manager daemon x restarted 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: Activating manager daemon x 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: mgrmap e10: x(active, starting, since 0.00885688s) 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: Manager daemon x is now available 2024-03-28T18:33:29.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:30.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:30.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:33:30.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:33:30.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:29 smithi119 ceph-mon[31813]: mgrmap e11: x(active, since 1.0102s) 2024-03-28T18:33:30.501 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-03-28T18:33:30.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T18:33:30.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T18:33:30.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-03-28T18:33:30.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.612+0000 7fc7f4a00700 1 Processor -- start 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.612+0000 7fc7f4a00700 1 -- start start 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.612+0000 7fc7f4a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.612+0000 7fc7f4a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f007c860 con 0x7fc7f007de30 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.613+0000 7fc7ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.613+0000 7fc7ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41882/0 (socket says 172.21.15.119:41882) 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.613+0000 7fc7ede00700 1 -- 172.21.15.119:0/811970418 learned_addr learned my addr 172.21.15.119:0/811970418 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.613+0000 7fc7ede00700 1 -- 172.21.15.119:0/811970418 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f007c9e0 con 0x7fc7f007de30 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.613+0000 7fc7ede00700 1 --2- 172.21.15.119:0/811970418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00098a0 tx=0x7fc7e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cea3e3539c4a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.614+0000 7fc7eca00700 1 -- 172.21.15.119:0/811970418 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e0004160 con 0x7fc7f007de30 2024-03-28T18:33:30.505 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.614+0000 7fc7eca00700 1 -- 172.21.15.119:0/811970418 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7e0004300 con 0x7fc7f007de30 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.614+0000 7fc7eca00700 1 -- 172.21.15.119:0/811970418 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e0011b20 con 0x7fc7f007de30 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 -- 172.21.15.119:0/811970418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 msgr2=0x7fc7f007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 --2- 172.21.15.119:0/811970418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00098a0 tx=0x7fc7e0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 -- 172.21.15.119:0/811970418 shutdown_connections 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 --2- 172.21.15.119:0/811970418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f007c290 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 -- 172.21.15.119:0/811970418 >> 172.21.15.119:0/811970418 conn(0x7fc7f01030f0 msgr2=0x7fc7f0105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 -- 172.21.15.119:0/811970418 shutdown_connections 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.615+0000 7fc7f4a00700 1 -- 172.21.15.119:0/811970418 wait complete. 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.616+0000 7fc7f4a00700 1 Processor -- start 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7f4a00700 1 -- start start 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7f4a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7f4a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f010e170 con 0x7fc7f007de30 2024-03-28T18:33:30.506 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:30.507 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41898/0 (socket says 172.21.15.119:41898) 2024-03-28T18:33:30.507 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.617+0000 7fc7ede00700 1 -- 172.21.15.119:0/1561760277 learned_addr learned my addr 172.21.15.119:0/1561760277 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:30.507 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7ede00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f019e190 con 0x7fc7f007de30 2024-03-28T18:33:30.507 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7ede00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0011b60 tx=0x7fc7e0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e0003b60 con 0x7fc7f007de30 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7e001a910 con 0x7fc7f007de30 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e0024930 con 0x7fc7f007de30 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.618+0000 7fc7f4a00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f019e420 con 0x7fc7f007de30 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.619+0000 7fc7f4a00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f019e880 con 0x7fc7f007de30 2024-03-28T18:33:30.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fc7e0022020 con 0x7fc7f007de30 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7e6a00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7d403bb40 con 0x7fc7d4038f70 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc7e004d1c0 con 0x7fc7f007de30 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7ed400700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.620+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.821+0000 7fc7ed400700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:23.821+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:24.222+0000 7fc7ed400700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:24.222+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:25.023+0000 7fc7ed400700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:30.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:25.023+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:26.625+0000 7fc7ed400700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/3359592084 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:26.625+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:28.829+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7fc7e001a430 con 0x7fc7f007de30 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:28.829+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 msgr2=0x7fc7d403b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:28.829+0000 7fc7e6a00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.831+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fc7e004eb00 con 0x7fc7f007de30 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.831+0000 7fc7e6a00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 0x7fc7d403ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.831+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7e0001d30 con 0x7fc7d403c9f0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.832+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 0x7fc7d403ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.835+0000 7fc7ed400700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 0x7fc7d403ede0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc003a10 tx=0x7fc7dc0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.836+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fc7e0001d30 con 0x7fc7d403c9f0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.844+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc7c8002600 con 0x7fc7d403c9f0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.844+0000 7fc7e6a00700 1 -- 172.21.15.119:0/1561760277 <== mgr.14152 v2:172.21.15.119:6800/2493078203 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc7c8002600 con 0x7fc7d403c9f0 2024-03-28T18:33:30.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 msgr2=0x7fc7d403ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 0x7fc7d403ede0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc003a10 tx=0x7fc7dc0092b0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 msgr2=0x7fc7f019dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0011b60 tx=0x7fc7e0011c40 comp rx=0 tx=0).stop 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 shutdown_connections 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc7d403c9f0 0x7fc7d403ede0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7f007de30 0x7fc7f019dc50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 --2- 172.21.15.119:0/1561760277 >> [v2:172.21.15.119:6800/3359592084,v1:172.21.15.119:6801/3359592084] conn(0x7fc7d4038f70 0x7fc7d403b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 >> 172.21.15.119:0/1561760277 conn(0x7fc7f01030f0 msgr2=0x7fc7f0103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 shutdown_connections 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:29.845+0000 7fc7dbe00700 1 -- 172.21.15.119:0/1561760277 wait complete. 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 9 is available 2024-03-28T18:33:30.514 INFO:teuthology.orchestra.run.smithi119.stdout:Generating a dashboard self-signed certificate... 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: [28/Mar/2024:18:33:29] ENGINE Bus STARTING 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: [28/Mar/2024:18:33:29] ENGINE Serving on http://172.21.15.119:8765 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: [28/Mar/2024:18:33:29] ENGINE Serving on https://172.21.15.119:7150 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: [28/Mar/2024:18:33:29] ENGINE Bus STARTED 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T18:33:31.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:30 smithi119 ceph-mon[31813]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T18:33:32.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:31 smithi119 ceph-mon[31813]: mgrmap e12: x(active, since 2s) 2024-03-28T18:33:32.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:32.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:32.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.249+0000 7fba37400700 1 Processor -- start 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.249+0000 7fba37400700 1 -- start start 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.250+0000 7fba37400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.250+0000 7fba37400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3010a700 con 0x7fba30109d10 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.250+0000 7fba34a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.250+0000 7fba34a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35344/0 (socket says 172.21.15.119:35344) 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.250+0000 7fba34a00700 1 -- 172.21.15.119:0/4115772924 learned_addr learned my addr 172.21.15.119:0/4115772924 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.251+0000 7fba34a00700 1 -- 172.21.15.119:0/4115772924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3010af10 con 0x7fba30109d10 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.251+0000 7fba34a00700 1 --2- 172.21.15.119:0/4115772924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba240098a0 tx=0x7fba24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaf92326e916d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:32.324 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.251+0000 7fba2f400700 1 -- 172.21.15.119:0/4115772924 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba24004160 con 0x7fba30109d10 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.251+0000 7fba2f400700 1 -- 172.21.15.119:0/4115772924 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba24004300 con 0x7fba30109d10 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.251+0000 7fba2f400700 1 -- 172.21.15.119:0/4115772924 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba24011b20 con 0x7fba30109d10 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 -- 172.21.15.119:0/4115772924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 msgr2=0x7fba3010a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 --2- 172.21.15.119:0/4115772924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba240098a0 tx=0x7fba24009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 -- 172.21.15.119:0/4115772924 shutdown_connections 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 --2- 172.21.15.119:0/4115772924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3010a130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 -- 172.21.15.119:0/4115772924 >> 172.21.15.119:0/4115772924 conn(0x7fba3007d100 msgr2=0x7fba3007d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 -- 172.21.15.119:0/4115772924 shutdown_connections 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.252+0000 7fba37400700 1 -- 172.21.15.119:0/4115772924 wait complete. 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.253+0000 7fba37400700 1 Processor -- start 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba37400700 1 -- start start 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba37400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba37400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3010e2c0 con 0x7fba30109d10 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba34a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba34a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35350/0 (socket says 172.21.15.119:35350) 2024-03-28T18:33:32.325 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba34a00700 1 -- 172.21.15.119:0/2643399577 learned_addr learned my addr 172.21.15.119:0/2643399577 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.254+0000 7fba34a00700 1 -- 172.21.15.119:0/2643399577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3019e1c0 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba34a00700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba240117e0 tx=0x7fba24011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba24003bf0 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba2401a910 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba24024970 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3019e450 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.255+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3019e8b0 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.256+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba30069260 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.257+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fba24022020 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.257+0000 7fba2d600700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 0x7fba1803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.257+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fba2404d390 con 0x7fba30109d10 2024-03-28T18:33:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.257+0000 7fba2fe00700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 0x7fba1803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.258+0000 7fba2fe00700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 0x7fba1803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba1c006fd0 tx=0x7fba1c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.263+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba2401ab70 con 0x7fba30109d10 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.519+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fba30107b20 con 0x7fba18038fd0 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.683+0000 7fba2d600700 1 -- 172.21.15.119:0/2643399577 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fba30107b20 con 0x7fba18038fd0 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 msgr2=0x7fba1803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 0x7fba1803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba1c006fd0 tx=0x7fba1c006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 msgr2=0x7fba3019dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba240117e0 tx=0x7fba24011ba0 comp rx=0 tx=0).stop 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 shutdown_connections 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba18038fd0 0x7fba1803b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 --2- 172.21.15.119:0/2643399577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba30109d10 0x7fba3019dc80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.686+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 >> 172.21.15.119:0/2643399577 conn(0x7fba3007d100 msgr2=0x7fba30107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.687+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 shutdown_connections 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:31.687+0000 7fba37400700 1 -- 172.21.15.119:0/2643399577 wait complete. 2024-03-28T18:33:32.327 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial admin user... 2024-03-28T18:33:33.158 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:32 smithi119 ceph-mon[31813]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:34.122 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$dV8THee8EVmr5fl19f98teIGharfQFx6XarL3uH.VLMcs/LZ8u8bK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711650813, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.898+0000 7f5f8d000700 1 Processor -- start 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.898+0000 7f5f8d000700 1 -- start start 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.899+0000 7f5f8d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.899+0000 7f5f8d000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8807c840 con 0x7f5f8807de10 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.899+0000 7f5f85e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.899+0000 7f5f85e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35354/0 (socket says 172.21.15.119:35354) 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.899+0000 7f5f85e00700 1 -- 172.21.15.119:0/1472962471 learned_addr learned my addr 172.21.15.119:0/1472962471 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.900+0000 7f5f85e00700 1 -- 172.21.15.119:0/1472962471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8807c9c0 con 0x7f5f8807de10 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.900+0000 7f5f85e00700 1 --2- 172.21.15.119:0/1472962471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098a0 tx=0x7f5f78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fab6cf9a075d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.900+0000 7f5f85400700 1 -- 172.21.15.119:0/1472962471 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f78004160 con 0x7f5f8807de10 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.900+0000 7f5f85400700 1 -- 172.21.15.119:0/1472962471 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f78004300 con 0x7f5f8807de10 2024-03-28T18:33:34.123 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.900+0000 7f5f85400700 1 -- 172.21.15.119:0/1472962471 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f78011b20 con 0x7f5f8807de10 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.901+0000 7f5f8d000700 1 -- 172.21.15.119:0/1472962471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 msgr2=0x7f5f8807c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.901+0000 7f5f8d000700 1 --2- 172.21.15.119:0/1472962471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5f780098a0 tx=0x7f5f78009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.901+0000 7f5f8d000700 1 -- 172.21.15.119:0/1472962471 shutdown_connections 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.901+0000 7f5f8d000700 1 --2- 172.21.15.119:0/1472962471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8807c270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.901+0000 7f5f8d000700 1 -- 172.21.15.119:0/1472962471 >> 172.21.15.119:0/1472962471 conn(0x7f5f881030f0 msgr2=0x7f5f88105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.902+0000 7f5f8d000700 1 -- 172.21.15.119:0/1472962471 shutdown_connections 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.902+0000 7f5f8d000700 1 -- 172.21.15.119:0/1472962471 wait complete. 2024-03-28T18:33:34.124 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.903+0000 7f5f8d000700 1 Processor -- start 2024-03-28T18:33:34.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.903+0000 7f5f8d000700 1 -- start start 2024-03-28T18:33:34.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f8d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:34.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f8d000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8810e130 con 0x7f5f8807de10 2024-03-28T18:33:34.127 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f85e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f85e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35368/0 (socket says 172.21.15.119:35368) 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f85e00700 1 -- 172.21.15.119:0/3657423810 learned_addr learned my addr 172.21.15.119:0/3657423810 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.904+0000 7f5f85e00700 1 -- 172.21.15.119:0/3657423810 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8819e1a0 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f85e00700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f78011b60 tx=0x7f5f78011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f78003b60 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f7801a910 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f78024930 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8819e430 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.905+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8819e890 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.907+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f88069260 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.907+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5f78022020 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.907+0000 7f5f7ea00700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 0x7f5f6c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.907+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f7804d230 con 0x7f5f8807de10 2024-03-28T18:33:34.128 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.908+0000 7f5f7fe00700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 0x7f5f6c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.909+0000 7f5f7fe00700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 0x7f5f6c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:32.917+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f7800b3e0 con 0x7f5f8807de10 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.157+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5f881a11e0 con 0x7f5f6c038fd0 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.396+0000 7f5f7ea00700 1 -- 172.21.15.119:0/3657423810 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f5f881a11e0 con 0x7f5f6c038fd0 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 msgr2=0x7f5f6c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 0x7f5f6c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5f70006fd0 tx=0x7f5f70006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 msgr2=0x7f5f8819dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f78011b60 tx=0x7f5f78011c40 comp rx=0 tx=0).stop 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 shutdown_connections 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f6c038fd0 0x7f5f6c03b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 --2- 172.21.15.119:0/3657423810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f8807de10 0x7f5f8819dc60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 >> 172.21.15.119:0/3657423810 conn(0x7f5f881030f0 msgr2=0x7f5f88103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 shutdown_connections 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:33.399+0000 7f5f8d000700 1 -- 172.21.15.119:0/3657423810 wait complete. 2024-03-28T18:33:34.129 INFO:teuthology.orchestra.run.smithi119.stdout:Fetching dashboard port number... 2024-03-28T18:33:34.685 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:34 smithi119 ceph-mon[31813]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:34.685 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:34.685 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:35.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:35 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/927724423' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T18:33:36.453 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.684+0000 7f4cd6000700 1 Processor -- start 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.684+0000 7f4cd6000700 1 -- start start 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.684+0000 7f4cd6000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.684+0000 7f4cd6000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd007be60 con 0x7f4cd01085a0 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.685+0000 7f4ccf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.685+0000 7f4ccf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35376/0 (socket says 172.21.15.119:35376) 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.685+0000 7f4ccf400700 1 -- 172.21.15.119:0/3429727906 learned_addr learned my addr 172.21.15.119:0/3429727906 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.685+0000 7f4ccf400700 1 -- 172.21.15.119:0/3429727906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd010aed0 con 0x7f4cd01085a0 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.686+0000 7f4ccf400700 1 --2- 172.21.15.119:0/3429727906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00098a0 tx=0x7f4cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6ff25ca16528e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.686+0000 7f4cce000700 1 -- 172.21.15.119:0/3429727906 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc0004160 con 0x7f4cd01085a0 2024-03-28T18:33:36.454 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.686+0000 7f4cce000700 1 -- 172.21.15.119:0/3429727906 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cc0004300 con 0x7f4cd01085a0 2024-03-28T18:33:36.458 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.686+0000 7f4cce000700 1 -- 172.21.15.119:0/3429727906 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc0011b20 con 0x7f4cd01085a0 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.687+0000 7f4cd6000700 1 -- 172.21.15.119:0/3429727906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 msgr2=0x7f4cd010a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.687+0000 7f4cd6000700 1 --2- 172.21.15.119:0/3429727906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00098a0 tx=0x7f4cc0009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.687+0000 7f4cd6000700 1 -- 172.21.15.119:0/3429727906 shutdown_connections 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.687+0000 7f4cd6000700 1 --2- 172.21.15.119:0/3429727906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd010a990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.687+0000 7f4cd6000700 1 -- 172.21.15.119:0/3429727906 >> 172.21.15.119:0/3429727906 conn(0x7f4cd0101ff0 msgr2=0x7f4cd0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.688+0000 7f4cd6000700 1 -- 172.21.15.119:0/3429727906 shutdown_connections 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.688+0000 7f4cd6000700 1 -- 172.21.15.119:0/3429727906 wait complete. 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.689+0000 7f4cd6000700 1 Processor -- start 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4cd6000700 1 -- start start 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4cd6000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4cd6000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd010bf40 con 0x7f4cd01085a0 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4ccf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:36.459 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4ccf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35378/0 (socket says 172.21.15.119:35378) 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.690+0000 7f4ccf400700 1 -- 172.21.15.119:0/927724423 learned_addr learned my addr 172.21.15.119:0/927724423 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4ccf400700 1 -- 172.21.15.119:0/927724423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd019c050 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4ccf400700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00117e0 tx=0x7f4cc0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc0003ba0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cc001a910 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd019c2e0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.691+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4cc00248f0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.692+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd019ce00 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.693+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f4cc0022020 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.693+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd00637c0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.693+0000 7f4cc7e00700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 0x7f4cb403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.693+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4cc004d7b0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.694+0000 7f4ccea00700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 0x7f4cb403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.694+0000 7f4ccea00700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 0x7f4cb403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006fd0 tx=0x7f4cbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.699+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cc000b3e0 con 0x7f4cd01085a0 2024-03-28T18:33:36.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.916+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4cd019d210 con 0x7f4cd01085a0 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.917+0000 7f4cc7e00700 1 -- 172.21.15.119:0/927724423 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4cc00172b0 con 0x7f4cd01085a0 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.919+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 msgr2=0x7f4cb403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.919+0000 7f4cd6000700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 0x7f4cb403b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006fd0 tx=0x7f4cbc006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.919+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 msgr2=0x7f4cd019bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.919+0000 7f4cd6000700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00117e0 tx=0x7f4cc0011ba0 comp rx=0 tx=0).stop 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.919+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 shutdown_connections 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.920+0000 7f4cd6000700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4cb4038f70 0x7f4cb403b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.920+0000 7f4cd6000700 1 --2- 172.21.15.119:0/927724423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4cd01085a0 0x7f4cd019bb10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.920+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 >> 172.21.15.119:0/927724423 conn(0x7f4cd0101ff0 msgr2=0x7f4cd0102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.920+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 shutdown_connections 2024-03-28T18:33:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:34.920+0000 7f4cd6000700 1 -- 172.21.15.119:0/927724423 wait complete. 2024-03-28T18:33:36.485 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T18:33:36.485 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-03-28T18:33:36.507 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T18:33:36.507 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-03-28T18:33:36.508 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-03-28T18:33:36.508 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph Dashboard is now available at: 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout: URL: https://smithi119.front.sepia.ceph.com:8443/ 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout: User: admin 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout: Password: mg9pp8bje7 2024-03-28T18:33:36.515 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:36.516 INFO:teuthology.orchestra.run.smithi119.stdout:Saving cluster configuration to /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config directory 2024-03-28T18:33:36.518 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling autotune for osd_memory_target 2024-03-28T18:33:36.657 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:36 smithi119 ceph-mon[31813]: mgrmap e13: x(active, since 6s) 2024-03-28T18:33:37.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.096+0000 7f97fa800700 1 Processor -- start 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.096+0000 7f97fa800700 1 -- start start 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.097+0000 7f97fa800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.097+0000 7f97fa800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41082f0 con 0x7f97f4107900 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.097+0000 7f97f9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.097+0000 7f97f9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35380/0 (socket says 172.21.15.119:35380) 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.097+0000 7f97f9400700 1 -- 172.21.15.119:0/2581826949 learned_addr learned my addr 172.21.15.119:0/2581826949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.098+0000 7f97f9400700 1 -- 172.21.15.119:0/2581826949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4108b00 con 0x7f97f4107900 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.098+0000 7f97f9400700 1 --2- 172.21.15.119:0/2581826949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098a0 tx=0x7f97e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd59609ab848b158 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.098+0000 7f97f3e00700 1 -- 172.21.15.119:0/2581826949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97e8004160 con 0x7f97f4107900 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.098+0000 7f97f3e00700 1 -- 172.21.15.119:0/2581826949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97e8004300 con 0x7f97f4107900 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.098+0000 7f97f3e00700 1 -- 172.21.15.119:0/2581826949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97e8011b20 con 0x7f97f4107900 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 -- 172.21.15.119:0/2581826949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 msgr2=0x7f97f4107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:37.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 --2- 172.21.15.119:0/2581826949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098a0 tx=0x7f97e8009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 -- 172.21.15.119:0/2581826949 shutdown_connections 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 --2- 172.21.15.119:0/2581826949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f4107d20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 -- 172.21.15.119:0/2581826949 >> 172.21.15.119:0/2581826949 conn(0x7f97f4102f00 msgr2=0x7f97f4105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 -- 172.21.15.119:0/2581826949 shutdown_connections 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.099+0000 7f97fa800700 1 -- 172.21.15.119:0/2581826949 wait complete. 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.100+0000 7f97fa800700 1 Processor -- start 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.100+0000 7f97fa800700 1 -- start start 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97fa800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97fa800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f410beb0 con 0x7f97f4107900 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97f9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97f9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35388/0 (socket says 172.21.15.119:35388) 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97f9400700 1 -- 172.21.15.119:0/251970035 learned_addr learned my addr 172.21.15.119:0/251970035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.101+0000 7f97f9400700 1 -- 172.21.15.119:0/251970035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f419dfd0 con 0x7f97f4107900 2024-03-28T18:33:37.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97f9400700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97e80117e0 tx=0x7f97e8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97e8003bf0 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97e801a910 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f419e260 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97e80249a0 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.102+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f419e6c0 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.103+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f97e8022020 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.103+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f4069260 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.104+0000 7f97f2000700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 0x7f97dc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.104+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97e804d360 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.104+0000 7f97f8a00700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 0x7f97dc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.105+0000 7f97f8a00700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 0x7f97dc03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.109+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97e8011e20 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.328+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f97f41a1090 con 0x7f97f4107900 2024-03-28T18:33:37.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.329+0000 7f97f2000700 1 -- 172.21.15.119:0/251970035 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f97e8004300 con 0x7f97f4107900 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 msgr2=0x7f97dc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 0x7f97dc03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f97e0006fd0 tx=0x7f97e0006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 msgr2=0x7f97f419da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f97e80117e0 tx=0x7f97e8011c10 comp rx=0 tx=0).stop 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 shutdown_connections 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f97dc039020 0x7f97dc03b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 --2- 172.21.15.119:0/251970035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f4107900 0x7f97f419da90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 >> 172.21.15.119:0/251970035 conn(0x7f97f4102f00 msgr2=0x7f97f41039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 shutdown_connections 2024-03-28T18:33:37.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:37.332+0000 7f97fa800700 1 -- 172.21.15.119:0/251970035 wait complete. 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.513+0000 7fc607a00700 1 Processor -- start 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.513+0000 7fc607a00700 1 -- start start 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.514+0000 7fc607a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.514+0000 7fc607a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60007be60 con 0x7fc6001085a0 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.515+0000 7fc605000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:39.436 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.515+0000 7fc605000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35392/0 (socket says 172.21.15.119:35392) 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.515+0000 7fc605000700 1 -- 172.21.15.119:0/3969643325 learned_addr learned my addr 172.21.15.119:0/3969643325 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.515+0000 7fc605000700 1 -- 172.21.15.119:0/3969643325 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60010aed0 con 0x7fc6001085a0 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.516+0000 7fc605000700 1 --2- 172.21.15.119:0/3969643325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40098a0 tx=0x7fc5f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f65caa6aa260f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.516+0000 7fc5ff400700 1 -- 172.21.15.119:0/3969643325 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f4004160 con 0x7fc6001085a0 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.516+0000 7fc5ff400700 1 -- 172.21.15.119:0/3969643325 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5f4004300 con 0x7fc6001085a0 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.516+0000 7fc5ff400700 1 -- 172.21.15.119:0/3969643325 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f4011b20 con 0x7fc6001085a0 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 -- 172.21.15.119:0/3969643325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 msgr2=0x7fc60010a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 --2- 172.21.15.119:0/3969643325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40098a0 tx=0x7fc5f4009bb0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 -- 172.21.15.119:0/3969643325 shutdown_connections 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 --2- 172.21.15.119:0/3969643325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc60010a990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 -- 172.21.15.119:0/3969643325 >> 172.21.15.119:0/3969643325 conn(0x7fc600101ff0 msgr2=0x7fc600104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 -- 172.21.15.119:0/3969643325 shutdown_connections 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.517+0000 7fc607a00700 1 -- 172.21.15.119:0/3969643325 wait complete. 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.518+0000 7fc607a00700 1 Processor -- start 2024-03-28T18:33:39.437 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc607a00700 1 -- start start 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc607a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc607a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60010bf40 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc605000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc605000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35408/0 (socket says 172.21.15.119:35408) 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.519+0000 7fc605000700 1 -- 172.21.15.119:0/3309584600 learned_addr learned my addr 172.21.15.119:0/3309584600 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc605000700 1 -- 172.21.15.119:0/3309584600 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600106550 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc605000700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40117e0 tx=0x7fc5f4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f4003ba0 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5f401a910 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6001067e0 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.520+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f40248f0 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.521+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc600106c40 con 0x7fc6001085a0 2024-03-28T18:33:39.438 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.522+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6000637c0 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.522+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc5f4022020 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.522+0000 7fc5fd600700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 0x7fc5e803b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.523+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5f404d5a0 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.523+0000 7fc5ffe00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 0x7fc5e803b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.523+0000 7fc5ffe00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 0x7fc5e803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec006fd0 tx=0x7fc5ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.528+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5f401ab70 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.864+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fc600107140 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.868+0000 7fc5fd600700 1 -- 172.21.15.119:0/3309584600 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fc5f4010380 con 0x7fc6001085a0 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 msgr2=0x7fc5e803b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 0x7fc5e803b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec006fd0 tx=0x7fc5ec006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 msgr2=0x7fc600108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc5f40117e0 tx=0x7fc5f4011ba0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 shutdown_connections 2024-03-28T18:33:39.439 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc5e8039070 0x7fc5e803b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 --2- 172.21.15.119:0/3309584600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6001085a0 0x7fc600108000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 >> 172.21.15.119:0/3309584600 conn(0x7fc600101ff0 msgr2=0x7fc600102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 shutdown_connections 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-03-28T18:33:38.871+0000 7fc607a00700 1 -- 172.21.15.119:0/3309584600 wait complete. 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.440 INFO:teuthology.orchestra.run.smithi119.stdout: ceph telemetry on 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout:For more information see: 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:39.441 INFO:teuthology.orchestra.run.smithi119.stdout:Bootstrap complete. 2024-03-28T18:33:39.535 INFO:tasks.cephadm:Fetching config... 2024-03-28T18:33:39.535 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:33:39.536 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T18:33:39.591 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T18:33:39.591 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:33:39.591 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T18:33:39.630 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T18:33:39.630 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:33:39.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T18:33:39.753 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T18:33:39.753 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:33:39.753 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T18:33:39.848 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T18:33:39.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:33:39.970 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:33:40.013 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:39 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3309584600' entity='client.admin' 2024-03-28T18:33:40.032 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:33:40.078 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:33:40.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T18:33:40.140 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9yXbnIcQVN9+x+GmC1q20opJwI+eb1OBTLwfnwb5jIn5l6HSvhzIzI7HBkbQDbp6r7iqmX3GJAawQ/jAxOFqdcJgFDz6tvFckuMXaMCaVK/3/p0otLLJpVyMRUYntqZPEzlIlIWAB2H9JOIwuggWIGpGY1L+ObyhQFk+DSnRII8x/hG8X6VDyS9sLuOmQbiWdMV3CGXYMWPrnLcWgZxhvMY5QyITNehMlXJC3UxR6M4mzn4GObyG/gnEGnxlrtd49s+4vnqOtIVwHh5msvwwnyWGyS2YGQeQGNmEQah3KOtAqH9a24hCLBLSLBVtScbvdaozteCHQaNjHgM4L8V0zK5xkpROqdIW10Wuu4ILu+zIapu2XfBu7iu9RHGUgTOYZ1o+PAIOmTldWIcbdbIxwcWikPP68DmcMjYRKahfh+DddOmdegpGab9zY1jxn+zNR79QulaB5FrLpXtymXW9Lv0ydYLPiY04D7BeYzsvBiMUJa+r4Uzs3bzujkD8xjEU= ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:33:40.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T18:33:40.593 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:43.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.364+0000 7f5c2b600700 1 -- 172.21.15.119:0/1308308383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 msgr2=0x7f5c24074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.364+0000 7f5c2b600700 1 --2- 172.21.15.119:0/1308308383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c24074620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5c18009910 tx=0x7f5c18009c20 comp rx=0 tx=0).stop 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.364+0000 7f5c2b600700 1 -- 172.21.15.119:0/1308308383 shutdown_connections 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.364+0000 7f5c2b600700 1 --2- 172.21.15.119:0/1308308383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c24074620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.364+0000 7f5c2b600700 1 -- 172.21.15.119:0/1308308383 >> 172.21.15.119:0/1308308383 conn(0x7f5c240fd150 msgr2=0x7f5c240ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.365+0000 7f5c2b600700 1 -- 172.21.15.119:0/1308308383 shutdown_connections 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.365+0000 7f5c2b600700 1 -- 172.21.15.119:0/1308308383 wait complete. 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.366+0000 7f5c2b600700 1 Processor -- start 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.366+0000 7f5c2b600700 1 -- start start 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.367+0000 7f5c2b600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.367+0000 7f5c2b600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c24109e40 con 0x7f5c240761c0 2024-03-28T18:33:43.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.367+0000 7f5c28c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.367+0000 7f5c28c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44352/0 (socket says 172.21.15.119:44352) 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.367+0000 7f5c28c00700 1 -- 172.21.15.119:0/2003812923 learned_addr learned my addr 172.21.15.119:0/2003812923 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c28c00700 1 -- 172.21.15.119:0/2003812923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24199a00 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c28c00700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5c18005850 tx=0x7f5c18005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c18026070 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c180044a0 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c18003ab0 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c24199c90 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.368+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2419c980 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.370+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2410a5c0 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.370+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5c18029070 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.370+0000 7f5c21600700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 0x7f5c0c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.370+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5c1801e070 con 0x7f5c240761c0 2024-03-28T18:33:43.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.371+0000 7f5c23e00700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 0x7f5c0c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:43.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.371+0000 7f5c23e00700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 0x7f5c0c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5c10006fd0 tx=0x7f5c10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:43.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.376+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c18013070 con 0x7f5c240761c0 2024-03-28T18:33:43.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.597+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5c2402db40 con 0x7f5c240761c0 2024-03-28T18:33:43.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.601+0000 7f5c21600700 1 -- 172.21.15.119:0/2003812923 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5c18030020 con 0x7f5c240761c0 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.604+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 msgr2=0x7f5c0c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.604+0000 7f5c2b600700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 0x7f5c0c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5c10006fd0 tx=0x7f5c10006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.604+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 msgr2=0x7f5c241994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.604+0000 7f5c2b600700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5c18005850 tx=0x7f5c18005030 comp rx=0 tx=0).stop 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 shutdown_connections 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5c0c038fc0 0x7f5c0c03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 --2- 172.21.15.119:0/2003812923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c240761c0 0x7f5c241994c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:43.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 >> 172.21.15.119:0/2003812923 conn(0x7f5c240fd150 msgr2=0x7f5c240fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:43.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 shutdown_connections 2024-03-28T18:33:43.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:43.605+0000 7f5c2b600700 1 -- 172.21.15.119:0/2003812923 wait complete. 2024-03-28T18:33:44.381 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T18:33:44.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T18:33:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:44 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2003812923' entity='client.admin' 2024-03-28T18:33:47.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:47.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:47.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:33:47.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:47.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.241+0000 7f6750800700 1 -- 172.21.15.119:0/1278899904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 msgr2=0x7f6748104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.241+0000 7f6750800700 1 --2- 172.21.15.119:0/1278899904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748104200 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f673c009910 tx=0x7f673c009c20 comp rx=0 tx=0).stop 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.241+0000 7f6750800700 1 -- 172.21.15.119:0/1278899904 shutdown_connections 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.241+0000 7f6750800700 1 --2- 172.21.15.119:0/1278899904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748104200 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.241+0000 7f6750800700 1 -- 172.21.15.119:0/1278899904 >> 172.21.15.119:0/1278899904 conn(0x7f67480ff380 msgr2=0x7f67481017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.242+0000 7f6750800700 1 -- 172.21.15.119:0/1278899904 shutdown_connections 2024-03-28T18:33:47.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.242+0000 7f6750800700 1 -- 172.21.15.119:0/1278899904 wait complete. 2024-03-28T18:33:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.242+0000 7f6750800700 1 Processor -- start 2024-03-28T18:33:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f6750800700 1 -- start start 2024-03-28T18:33:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f6750800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f6750800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6748109e70 con 0x7f6748103de0 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f674de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f674de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44366/0 (socket says 172.21.15.119:44366) 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.243+0000 7f674de00700 1 -- 172.21.15.119:0/3137610723 learned_addr learned my addr 172.21.15.119:0/3137610723 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f674de00700 1 -- 172.21.15.119:0/3137610723 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6748199a50 con 0x7f6748103de0 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f674de00700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f673c005850 tx=0x7f673c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f673c026070 con 0x7f6748103de0 2024-03-28T18:33:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f673c0044a0 con 0x7f6748103de0 2024-03-28T18:33:47.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f673c003ab0 con 0x7f6748103de0 2024-03-28T18:33:47.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6748199ce0 con 0x7f6748103de0 2024-03-28T18:33:47.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.244+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674819c9d0 con 0x7f6748103de0 2024-03-28T18:33:47.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.246+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f673c029070 con 0x7f6748103de0 2024-03-28T18:33:47.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.246+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674810a5f0 con 0x7f6748103de0 2024-03-28T18:33:47.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.246+0000 7f6742a00700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 0x7f673003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:47.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.246+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f673c01e070 con 0x7f6748103de0 2024-03-28T18:33:47.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.247+0000 7f674d400700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 0x7f673003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:47.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.247+0000 7f674d400700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 0x7f673003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6734006fd0 tx=0x7f6734006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:47.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.252+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f673c013070 con 0x7f6748103de0 2024-03-28T18:33:47.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.476+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f674802db40 con 0x7f6730038fc0 2024-03-28T18:33:47.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.482+0000 7f6742a00700 1 -- 172.21.15.119:0/3137610723 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f674802db40 con 0x7f6730038fc0 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 msgr2=0x7f673003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 0x7f673003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6734006fd0 tx=0x7f6734006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 msgr2=0x7f6748199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f673c005850 tx=0x7f673c005030 comp rx=0 tx=0).stop 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 shutdown_connections 2024-03-28T18:33:47.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6730038fc0 0x7f673003b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:47.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 --2- 172.21.15.119:0/3137610723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6748103de0 0x7f6748199510 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:47.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 >> 172.21.15.119:0/3137610723 conn(0x7f67480ff380 msgr2=0x7f67480ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:47.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 shutdown_connections 2024-03-28T18:33:47.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:47.485+0000 7f6750800700 1 -- 172.21.15.119:0/3137610723 wait complete. 2024-03-28T18:33:48.257 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi131 2024-03-28T18:33:48.257 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:33:48.257 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:33:48.276 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:33:48.276 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:33:48.332 INFO:tasks.cephadm:Adding host smithi131 to orchestrator... 2024-03-28T18:33:48.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch host add smithi131 2024-03-28T18:33:48.732 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:33:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:48 smithi119 ceph-mon[31813]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:33:49.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:49 smithi119 ceph-mon[31813]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:33:49.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:49 smithi119 ceph-mon[31813]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:50.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 -- 172.21.15.119:0/3289083647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 msgr2=0x7f3f981041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3289083647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981041e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009910 tx=0x7f3f8c009c20 comp rx=0 tx=0).stop 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 -- 172.21.15.119:0/3289083647 shutdown_connections 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3289083647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981041e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 -- 172.21.15.119:0/3289083647 >> 172.21.15.119:0/3289083647 conn(0x7f3f980ff6d0 msgr2=0x7f3f98101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 -- 172.21.15.119:0/3289083647 shutdown_connections 2024-03-28T18:33:50.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.674+0000 7f3f9e400700 1 -- 172.21.15.119:0/3289083647 wait complete. 2024-03-28T18:33:50.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.675+0000 7f3f9e400700 1 Processor -- start 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.675+0000 7f3f9e400700 1 -- start start 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.675+0000 7f3f9e400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.675+0000 7f3f9e400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9810e1d0 con 0x7f3f98103e00 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.676+0000 7f3f9d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.676+0000 7f3f9d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51104/0 (socket says 172.21.15.119:51104) 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.676+0000 7f3f9d000700 1 -- 172.21.15.119:0/3330820080 learned_addr learned my addr 172.21.15.119:0/3330820080 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.676+0000 7f3f9d000700 1 -- 172.21.15.119:0/3330820080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f98199bf0 con 0x7f3f98103e00 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.676+0000 7f3f9d000700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006010 tx=0x7f3f8c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.677+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c026070 con 0x7f3f98103e00 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.677+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f8c00bbd0 con 0x7f3f98103e00 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.677+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c003910 con 0x7f3f98103e00 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.677+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f98199ee0 con 0x7f3f98103e00 2024-03-28T18:33:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.677+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9819df80 con 0x7f3f98103e00 2024-03-28T18:33:50.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.678+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f3f8c029070 con 0x7f3f98103e00 2024-03-28T18:33:50.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.678+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f981052e0 con 0x7f3f98103e00 2024-03-28T18:33:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.678+0000 7f3f95600700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 0x7f3f8003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.679+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3f8c01e070 con 0x7f3f98103e00 2024-03-28T18:33:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.679+0000 7f3f97e00700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 0x7f3f8003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.679+0000 7f3f97e00700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 0x7f3f8003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:50.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.684+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f8c013070 con 0x7f3f98103e00 2024-03-28T18:33:50.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:50.909+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f9819aa10 con 0x7f3f80039020 2024-03-28T18:33:51.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:50 smithi119 ceph-mon[31813]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:33:51.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:51.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:51.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:51 smithi119 ceph-mon[31813]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:51 smithi119 ceph-mon[31813]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:33:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:52 smithi119 ceph-mon[31813]: Deploying cephadm binary to smithi131 2024-03-28T18:33:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:53 smithi119 ceph-mon[31813]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:55.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.533+0000 7f3f95600700 1 -- 172.21.15.119:0/3330820080 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3f9819aa10 con 0x7f3f80039020 2024-03-28T18:33:55.533 INFO:teuthology.orchestra.run.smithi119.stdout:Added host 'smithi131' with addr '172.21.15.131' 2024-03-28T18:33:55.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 msgr2=0x7f3f8003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:55.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 0x7f3f8003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006fd0 tx=0x7f3f84006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:55.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 msgr2=0x7f3f981996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:55.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006010 tx=0x7f3f8c005030 comp rx=0 tx=0).stop 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 shutdown_connections 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3f80039020 0x7f3f8003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 --2- 172.21.15.119:0/3330820080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f98103e00 0x7f3f981996b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 >> 172.21.15.119:0/3330820080 conn(0x7f3f980ff6d0 msgr2=0x7f3f981057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 shutdown_connections 2024-03-28T18:33:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:55.537+0000 7f3f9e400700 1 -- 172.21.15.119:0/3330820080 wait complete. 2024-03-28T18:33:56.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:55 smithi119 ceph-mon[31813]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:56.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:56.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:33:56.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:33:56.814 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:33:57.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:57 smithi119 ceph-mon[31813]: Added host smithi131 2024-03-28T18:33:57.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:57.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:58.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:58 smithi119 ceph-mon[31813]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 -- 172.21.15.119:0/1401846185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 msgr2=0x7f5a10104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 --2- 172.21.15.119:0/1401846185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a10104390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009910 tx=0x7f5a04009c20 comp rx=0 tx=0).stop 2024-03-28T18:33:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 -- 172.21.15.119:0/1401846185 shutdown_connections 2024-03-28T18:33:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 --2- 172.21.15.119:0/1401846185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a10104390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 -- 172.21.15.119:0/1401846185 >> 172.21.15.119:0/1401846185 conn(0x7f5a100ff880 msgr2=0x7f5a10101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:58.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 -- 172.21.15.119:0/1401846185 shutdown_connections 2024-03-28T18:33:58.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.733+0000 7f5a17800700 1 -- 172.21.15.119:0/1401846185 wait complete. 2024-03-28T18:33:58.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.734+0000 7f5a17800700 1 Processor -- start 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a17800700 1 -- start start 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a17800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a17800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1010e380 con 0x7f5a10103fb0 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a14e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a14e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51116/0 (socket says 172.21.15.119:51116) 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.735+0000 7f5a14e00700 1 -- 172.21.15.119:0/3808760930 learned_addr learned my addr 172.21.15.119:0/3808760930 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a14e00700 1 -- 172.21.15.119:0/3808760930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1019fbc0 con 0x7f5a10103fb0 2024-03-28T18:33:58.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a14e00700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005850 tx=0x7f5a04005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a04026070 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a0400bbd0 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a04003970 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1019feb0 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.736+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a10199920 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.738+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5a04029070 con 0x7f5a10103fb0 2024-03-28T18:33:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.738+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a10105490 con 0x7f5a10103fb0 2024-03-28T18:33:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.738+0000 7f5a0d600700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 0x7f59f803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:33:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.738+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a0401e070 con 0x7f5a10103fb0 2024-03-28T18:33:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.738+0000 7f5a0fe00700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 0x7f59f803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:33:58.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.739+0000 7f5a0fe00700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 0x7f59f803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59fc006fd0 tx=0x7f59fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:33:58.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.744+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5a04013070 con 0x7f5a10103fb0 2024-03-28T18:33:58.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.974+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a1019ac80 con 0x7f59f8039020 2024-03-28T18:33:58.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.976+0000 7f5a0d600700 1 -- 172.21.15.119:0/3808760930 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f5a1019ac80 con 0x7f59f8039020 2024-03-28T18:33:58.976 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:33:58.976 INFO:teuthology.orchestra.run.smithi119.stdout:[{"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}] 2024-03-28T18:33:58.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.978+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 msgr2=0x7f59f803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:58.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.978+0000 7f5a17800700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 0x7f59f803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59fc006fd0 tx=0x7f59fc006e40 comp rx=0 tx=0).stop 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 msgr2=0x7f5a1019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005850 tx=0x7f5a04005030 comp rx=0 tx=0).stop 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 shutdown_connections 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f59f8039020 0x7f59f803b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 --2- 172.21.15.119:0/3808760930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a10103fb0 0x7f5a1019f680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 >> 172.21.15.119:0/3808760930 conn(0x7f5a100ff880 msgr2=0x7f5a10106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:33:58.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 shutdown_connections 2024-03-28T18:33:58.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:33:58.979+0000 7f5a17800700 1 -- 172.21.15.119:0/3808760930 wait complete. 2024-03-28T18:33:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:33:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:59 smithi119 ceph-mon[31813]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:33:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:33:59 smithi119 ceph-mon[31813]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:33:59.731 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-03-28T18:33:59.731 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:33:59.731 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-03-28T18:33:59.749 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:33:59.749 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:33:59.806 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-03-28T18:33:59.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch host add smithi196 2024-03-28T18:34:00.240 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.056+0000 7f4e3ce00700 1 -- 172.21.15.119:0/4136989088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 msgr2=0x7f4e38102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.056+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/4136989088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e38102190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009910 tx=0x7f4e28009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.057+0000 7f4e3ce00700 1 -- 172.21.15.119:0/4136989088 shutdown_connections 2024-03-28T18:34:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.057+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/4136989088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e38102190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:02.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.057+0000 7f4e3ce00700 1 -- 172.21.15.119:0/4136989088 >> 172.21.15.119:0/4136989088 conn(0x7f4e380fd660 msgr2=0x7f4e380ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:02.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.057+0000 7f4e3ce00700 1 -- 172.21.15.119:0/4136989088 shutdown_connections 2024-03-28T18:34:02.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.057+0000 7f4e3ce00700 1 -- 172.21.15.119:0/4136989088 wait complete. 2024-03-28T18:34:02.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.058+0000 7f4e3ce00700 1 Processor -- start 2024-03-28T18:34:02.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.058+0000 7f4e3ce00700 1 -- start start 2024-03-28T18:34:02.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.058+0000 7f4e3ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:02.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.058+0000 7f4e3ce00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3810c180 con 0x7f4e38101db0 2024-03-28T18:34:02.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.059+0000 7f4e35e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:02.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.059+0000 7f4e35e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55660/0 (socket says 172.21.15.119:55660) 2024-03-28T18:34:02.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.059+0000 7f4e35e00700 1 -- 172.21.15.119:0/3052641212 learned_addr learned my addr 172.21.15.119:0/3052641212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.059+0000 7f4e35e00700 1 -- 172.21.15.119:0/3052641212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3819fb30 con 0x7f4e38101db0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.059+0000 7f4e35e00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006010 tx=0x7f4e28005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.060+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e28026070 con 0x7f4e38101db0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.060+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e2800bbd0 con 0x7f4e38101db0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.060+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e28003910 con 0x7f4e38101db0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.060+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3819fe20 con 0x7f4e38101db0 2024-03-28T18:34:02.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.060+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e38199890 con 0x7f4e38101db0 2024-03-28T18:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.061+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f4e28029070 con 0x7f4e38101db0 2024-03-28T18:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.061+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e38103290 con 0x7f4e38101db0 2024-03-28T18:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.061+0000 7f4e2ea00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 0x7f4e1c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.061+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e2801e070 con 0x7f4e38101db0 2024-03-28T18:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.062+0000 7f4e35400700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 0x7f4e1c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.062+0000 7f4e35400700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 0x7f4e1c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006fd0 tx=0x7f4e24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.067+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e28013070 con 0x7f4e38101db0 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:02.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:02.166 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:01 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:02.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:02.294+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e3819abf0 con 0x7f4e1c038fc0 2024-03-28T18:34:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:34:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:34:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:03.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:02 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:03 smithi119 ceph-mon[31813]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:03 smithi119 ceph-mon[31813]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:34:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:03 smithi119 ceph-mon[31813]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:03 smithi119 ceph-mon[31813]: Deploying cephadm binary to smithi196 2024-03-28T18:34:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:05 smithi119 ceph-mon[31813]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:06.838 INFO:teuthology.orchestra.run.smithi119.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-03-28T18:34:06.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.835+0000 7f4e2ea00700 1 -- 172.21.15.119:0/3052641212 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f4e3819abf0 con 0x7f4e1c038fc0 2024-03-28T18:34:06.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.839+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 msgr2=0x7f4e1c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:06.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 0x7f4e1c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e24006fd0 tx=0x7f4e24006e40 comp rx=0 tx=0).stop 2024-03-28T18:34:06.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 msgr2=0x7f4e3819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:06.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006010 tx=0x7f4e28005030 comp rx=0 tx=0).stop 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 shutdown_connections 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4e1c038fc0 0x7f4e1c03b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 --2- 172.21.15.119:0/3052641212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e38101db0 0x7f4e3819f5f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 >> 172.21.15.119:0/3052641212 conn(0x7f4e380fd660 msgr2=0x7f4e380743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 shutdown_connections 2024-03-28T18:34:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:06.840+0000 7f4e3ce00700 1 -- 172.21.15.119:0/3052641212 wait complete. 2024-03-28T18:34:07.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T18:34:07.911 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:08.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:08.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:07 smithi119 ceph-mon[31813]: Added host smithi196 2024-03-28T18:34:08.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:07 smithi119 ceph-mon[31813]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:08.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:08.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:09.191 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:08 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:09.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 -- 172.21.15.119:0/2159684286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 msgr2=0x7f4d60102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 --2- 172.21.15.119:0/2159684286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60102190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009910 tx=0x7f4d50009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 -- 172.21.15.119:0/2159684286 shutdown_connections 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 --2- 172.21.15.119:0/2159684286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60102190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 -- 172.21.15.119:0/2159684286 >> 172.21.15.119:0/2159684286 conn(0x7f4d600fd660 msgr2=0x7f4d600ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 -- 172.21.15.119:0/2159684286 shutdown_connections 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.892+0000 7f4d66400700 1 -- 172.21.15.119:0/2159684286 wait complete. 2024-03-28T18:34:09.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.894+0000 7f4d66400700 1 Processor -- start 2024-03-28T18:34:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.894+0000 7f4d66400700 1 -- start start 2024-03-28T18:34:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.894+0000 7f4d66400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.894+0000 7f4d66400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6010c180 con 0x7f4d60101db0 2024-03-28T18:34:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.894+0000 7f4d5f400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.895+0000 7f4d5f400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52460/0 (socket says 172.21.15.119:52460) 2024-03-28T18:34:09.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.895+0000 7f4d5f400700 1 -- 172.21.15.119:0/3584491019 learned_addr learned my addr 172.21.15.119:0/3584491019 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.895+0000 7f4d5f400700 1 -- 172.21.15.119:0/3584491019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60076380 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.895+0000 7f4d5f400700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006010 tx=0x7f4d50005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.895+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d50026070 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.896+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d5000bc30 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.896+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60074a30 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.896+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d500039d0 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.896+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d60074ef0 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.897+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60103290 con 0x7f4d60101db0 2024-03-28T18:34:09.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.897+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f4d50029070 con 0x7f4d60101db0 2024-03-28T18:34:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.897+0000 7f4d5cc00700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 0x7f4d4403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.897+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d5001e070 con 0x7f4d60101db0 2024-03-28T18:34:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.898+0000 7f4d56a00700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 0x7f4d4403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.899+0000 7f4d56a00700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 0x7f4d4403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d48006fd0 tx=0x7f4d48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:09.903+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d50013070 con 0x7f4d60101db0 2024-03-28T18:34:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:09 smithi119 ceph-mon[31813]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:10.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.132+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d600759c0 con 0x7f4d44038fc0 2024-03-28T18:34:10.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.134+0000 7f4d5cc00700 1 -- 172.21.15.119:0/3584491019 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f4d600759c0 con 0x7f4d44038fc0 2024-03-28T18:34:10.135 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:34:10.135 INFO:teuthology.orchestra.run.smithi119.stdout:[{"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.137+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 msgr2=0x7f4d4403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.137+0000 7f4d66400700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 0x7f4d4403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4d48006fd0 tx=0x7f4d48006e40 comp rx=0 tx=0).stop 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.137+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 msgr2=0x7f4d60074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.137+0000 7f4d66400700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006010 tx=0x7f4d50005030 comp rx=0 tx=0).stop 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 shutdown_connections 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4d44038fc0 0x7f4d4403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 --2- 172.21.15.119:0/3584491019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60101db0 0x7f4d60074490 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 >> 172.21.15.119:0/3584491019 conn(0x7f4d600fd660 msgr2=0x7f4d60104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 shutdown_connections 2024-03-28T18:34:10.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:10.138+0000 7f4d66400700 1 -- 172.21.15.119:0/3584491019 wait complete. 2024-03-28T18:34:10.981 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:10 smithi119 ceph-mon[31813]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:34:11.048 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T18:34:11.048 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T18:34:11.477 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:11.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:11 smithi119 ceph-mon[31813]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:13.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 -- 172.21.15.119:0/3524776210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 msgr2=0x7f1f58100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/3524776210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f58100cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009910 tx=0x7f1f48009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 -- 172.21.15.119:0/3524776210 shutdown_connections 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/3524776210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f58100cc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 -- 172.21.15.119:0/3524776210 >> 172.21.15.119:0/3524776210 conn(0x7f1f58077830 msgr2=0x7f1f580febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 -- 172.21.15.119:0/3524776210 shutdown_connections 2024-03-28T18:34:13.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.411+0000 7f1f5ce00700 1 -- 172.21.15.119:0/3524776210 wait complete. 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.412+0000 7f1f5ce00700 1 Processor -- start 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.412+0000 7f1f5ce00700 1 -- start start 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f5ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f5ce00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5810e3d0 con 0x7f1f581008e0 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f55e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f55e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52486/0 (socket says 172.21.15.119:52486) 2024-03-28T18:34:13.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f55e00700 1 -- 172.21.15.119:0/572057182 learned_addr learned my addr 172.21.15.119:0/572057182 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.413+0000 7f1f55e00700 1 -- 172.21.15.119:0/572057182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5819fb30 con 0x7f1f581008e0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f55e00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005850 tx=0x7f1f48005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f48026070 con 0x7f1f581008e0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f4800bc30 con 0x7f1f581008e0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f48003a20 con 0x7f1f581008e0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5819fdc0 con 0x7f1f581008e0 2024-03-28T18:34:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.414+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f581996c0 con 0x7f1f581008e0 2024-03-28T18:34:13.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.415+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f58069e20 con 0x7f1f581008e0 2024-03-28T18:34:13.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.415+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1f48029070 con 0x7f1f581008e0 2024-03-28T18:34:13.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.416+0000 7f1f4ea00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 0x7f1f3c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:13.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.416+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1f4801e070 con 0x7f1f581008e0 2024-03-28T18:34:13.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.416+0000 7f1f4fe00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 0x7f1f3c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:13.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.417+0000 7f1f4fe00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 0x7f1f3c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1f40006fd0 tx=0x7f1f40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:13.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.421+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f48013070 con 0x7f1f581008e0 2024-03-28T18:34:13.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:13.683+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1f580637c0 con 0x7f1f581008e0 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:34:13.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:13.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:13 smithi119 ceph-mon[31813]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:14.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.549+0000 7f1f4ea00700 1 -- 172.21.15.119:0/572057182 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1f4802ade0 con 0x7f1f581008e0 2024-03-28T18:34:14.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 msgr2=0x7f1f3c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:14.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 0x7f1f3c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1f40006fd0 tx=0x7f1f40006e40 comp rx=0 tx=0).stop 2024-03-28T18:34:14.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 msgr2=0x7f1f5819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:14.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005850 tx=0x7f1f48005030 comp rx=0 tx=0).stop 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 shutdown_connections 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f3c039020 0x7f1f3c03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 --2- 172.21.15.119:0/572057182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f581008e0 0x7f1f5819f5f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 >> 172.21.15.119:0/572057182 conn(0x7f1f58077830 msgr2=0x7f1f580fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.553+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 shutdown_connections 2024-03-28T18:34:14.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:14.554+0000 7f1f5ce00700 1 -- 172.21.15.119:0/572057182 wait complete. 2024-03-28T18:34:14.555 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-03-28T18:34:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:34:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:34:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/572057182' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T18:34:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:14.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:14.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:15.227 INFO:tasks.cephadm:Adding mon.a on smithi119 2024-03-28T18:34:15.228 INFO:tasks.cephadm:Adding mon.b on smithi131 2024-03-28T18:34:15.228 INFO:tasks.cephadm:Adding mon.c on smithi196 2024-03-28T18:34:15.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi119:172.21.15.119=a;smithi131:172.21.15.131=b;smithi196:172.21.15.196=c' 2024-03-28T18:34:15.465 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:15 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/572057182' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T18:34:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:15 smithi119 ceph-mon[31813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:15 smithi119 ceph-mon[31813]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:16.863 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.862+0000 7fc630684700 1 -- 172.21.15.196:0/160746191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 msgr2=0x7fc628106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 --2- 172.21.15.196:0/160746191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc628106e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc624009910 tx=0x7fc624009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 -- 172.21.15.196:0/160746191 shutdown_connections 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 --2- 172.21.15.196:0/160746191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc628106e30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 -- 172.21.15.196:0/160746191 >> 172.21.15.196:0/160746191 conn(0x7fc6280fc580 msgr2=0x7fc6280fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 -- 172.21.15.196:0/160746191 shutdown_connections 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 -- 172.21.15.196:0/160746191 wait complete. 2024-03-28T18:34:16.864 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 Processor -- start 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.863+0000 7fc630684700 1 -- start start 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.864+0000 7fc630684700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.864+0000 7fc630684700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62810b090 con 0x7fc628106a50 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.864+0000 7fc62e420700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.864+0000 7fc62e420700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.196:56730/0 (socket says 172.21.15.196:56730) 2024-03-28T18:34:16.865 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.864+0000 7fc62e420700 1 -- 172.21.15.196:0/1902941372 learned_addr learned my addr 172.21.15.196:0/1902941372 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:34:16.866 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.865+0000 7fc62e420700 1 -- 172.21.15.196:0/1902941372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62819fb50 con 0x7fc628106a50 2024-03-28T18:34:16.866 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.865+0000 7fc62e420700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc624005850 tx=0x7fc624005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc624026070 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62819fe40 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc628199790 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62400bbd0 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc6240039b0 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.866+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c005330 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.867+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fc624029070 con 0x7fc628106a50 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.867+0000 7fc61f7fe700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 0x7fc61403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:16.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.867+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc62401e070 con 0x7fc628106a50 2024-03-28T18:34:16.868 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.868+0000 7fc62dc1f700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 0x7fc61403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:16.868 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.869+0000 7fc62dc1f700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 0x7fc61403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc618006fd0 tx=0x7fc618006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:16.882 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:16.883+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc624013070 con 0x7fc628106a50 2024-03-28T18:34:17.089 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.090+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi131:172.21.15.131=b;smithi196:172.21.15.196=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fc60c000c80 con 0x7fc614038fc0 2024-03-28T18:34:17.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.104+0000 7fc61f7fe700 1 -- 172.21.15.196:0/1902941372 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc60c000c80 con 0x7fc614038fc0 2024-03-28T18:34:17.103 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-03-28T18:34:17.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 msgr2=0x7fc61403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:17.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 0x7fc61403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc618006fd0 tx=0x7fc618006e40 comp rx=0 tx=0).stop 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 msgr2=0x7fc62819f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc624005850 tx=0x7fc624005030 comp rx=0 tx=0).stop 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 shutdown_connections 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc614038fc0 0x7fc61403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 --2- 172.21.15.196:0/1902941372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc628106a50 0x7fc62819f610 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 >> 172.21.15.196:0/1902941372 conn(0x7fc6280fc580 msgr2=0x7fc628190520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 shutdown_connections 2024-03-28T18:34:17.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:17.106+0000 7fc630684700 1 -- 172.21.15.196:0/1902941372 wait complete. 2024-03-28T18:34:17.724 DEBUG:teuthology.orchestra.run.smithi131:mon.b> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T18:34:17.727 DEBUG:teuthology.orchestra.run.smithi196:mon.c> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T18:34:17.729 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:34:17.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi131:172.21.15.131=b;smithi196:172.21.15.196=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi131:172.21.15.131=b;smithi196:172.21.15.196=c;count:3 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:18.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:18.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:18.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:18.788 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:19.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:19 smithi119 ceph-mon[31813]: Deploying daemon mon.c on smithi196 2024-03-28T18:34:20.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:20 smithi119 ceph-mon[31813]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:21.447 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 -- 172.21.15.196:0/1676516343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 msgr2=0x7ff0e8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 --2- 172.21.15.196:0/1676516343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8102190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009910 tx=0x7ff0d8009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 -- 172.21.15.196:0/1676516343 shutdown_connections 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 --2- 172.21.15.196:0/1676516343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8102190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 -- 172.21.15.196:0/1676516343 >> 172.21.15.196:0/1676516343 conn(0x7ff0e80fd660 msgr2=0x7ff0e80ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 -- 172.21.15.196:0/1676516343 shutdown_connections 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 -- 172.21.15.196:0/1676516343 wait complete. 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.446+0000 7ff0ed811700 1 Processor -- start 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0ed811700 1 -- start start 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0ed811700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0ed811700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e810c180 con 0x7ff0e8101db0 2024-03-28T18:34:21.448 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0e6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:21.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0e6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.196:56754/0 (socket says 172.21.15.196:56754) 2024-03-28T18:34:21.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.447+0000 7ff0e6ffd700 1 -- 172.21.15.196:0/4260869358 learned_addr learned my addr 172.21.15.196:0/4260869358 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:34:21.449 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.448+0000 7ff0e6ffd700 1 -- 172.21.15.196:0/4260869358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8197bc0 con 0x7ff0e8101db0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.448+0000 7ff0e6ffd700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8005850 tx=0x7ff0d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.449+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0d8026070 con 0x7ff0e8101db0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.449+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0d800bbd0 con 0x7ff0e8101db0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.449+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff0d8003970 con 0x7ff0e8101db0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.449+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e8197eb0 con 0x7ff0e8101db0 2024-03-28T18:34:21.450 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.449+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e819bf50 con 0x7ff0e8101db0 2024-03-28T18:34:21.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.450+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e8103290 con 0x7ff0e8101db0 2024-03-28T18:34:21.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.450+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7ff0d8029070 con 0x7ff0e8101db0 2024-03-28T18:34:21.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.450+0000 7ff0ec80f700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 0x7ff0d003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:21.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.451+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff0d801e070 con 0x7ff0e8101db0 2024-03-28T18:34:21.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.451+0000 7ff0e67fc700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 0x7ff0d003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:21.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.452+0000 7ff0e67fc700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 0x7ff0d003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006fd0 tx=0x7ff0dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:21.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.466+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff0d8013070 con 0x7ff0e8101db0 2024-03-28T18:34:21.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.760+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0e81989e0 con 0x7ff0e8101db0 2024-03-28T18:34:21.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.761+0000 7ff0ec80f700 1 -- 172.21.15.196:0/4260869358 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7ff0d8030090 con 0x7ff0e8101db0 2024-03-28T18:34:21.762 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:34:21.762 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","modified":"2024-03-28T18:32:18.696710Z","created":"2024-03-28T18:32:18.696710Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T18:34:21.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 msgr2=0x7ff0d003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 0x7ff0d003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006fd0 tx=0x7ff0dc006e40 comp rx=0 tx=0).stop 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 msgr2=0x7ff0e8197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8005850 tx=0x7ff0d8005030 comp rx=0 tx=0).stop 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 shutdown_connections 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0d0038fc0 0x7ff0d003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 --2- 172.21.15.196:0/4260869358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0e8101db0 0x7ff0e8197680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.763+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 >> 172.21.15.196:0/4260869358 conn(0x7ff0e80fd660 msgr2=0x7ff0e8104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.764+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 shutdown_connections 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:21.764+0000 7ff0ed811700 1 -- 172.21.15.196:0/4260869358 wait complete. 2024-03-28T18:34:21.764 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-28T18:34:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:22 smithi119 ceph-mon[31813]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:22 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/4260869358' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:34:22.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:22 smithi196 systemd[1]: Started Ceph mon.c for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:34:23.337 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:34:23.338 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:34:23.573 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:34:27.875 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: mon.a calling monitor election 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: mon.c calling monitor election 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:27.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: fsmap 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: mgrmap e13: x(active, since 58s) 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: overall HEALTH_OK 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:27.877 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.052+0000 7f7e7fa11700 1 -- 172.21.15.196:0/219022002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 msgr2=0x7f7e64005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.052+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/219022002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e64005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e78101ba0 tx=0x7f7e7400feb0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.052+0000 7f7e7fa11700 1 -- 172.21.15.196:0/219022002 shutdown_connections 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.052+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/219022002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e64005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.052+0000 7f7e7fa11700 1 -- 172.21.15.196:0/219022002 >> 172.21.15.196:0/219022002 conn(0x7f7e780fd760 msgr2=0x7f7e780ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.053+0000 7f7e7fa11700 1 -- 172.21.15.196:0/219022002 shutdown_connections 2024-03-28T18:34:28.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.053+0000 7f7e7fa11700 1 -- 172.21.15.196:0/219022002 wait complete. 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 Processor -- start 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 -- start start 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e7819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78109d00 con 0x7f7e640035c0 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7fa11700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78109b80 con 0x7f7e7819fbf0 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:54368/0 (socket says 172.21.15.196:54368) 2024-03-28T18:34:28.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 -- 172.21.15.196:0/3936247313 learned_addr learned my addr 172.21.15.196:0/3936247313 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:34:28.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 -- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 msgr2=0x7f7e78199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T18:34:28.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 -- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 msgr2=0x7f7e78199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:34:28.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:34:28.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.054+0000 7f7e7cfac700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:34:28.056 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.055+0000 7f7e7d7ad700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e7819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:28.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.055+0000 7f7e7d7ad700 1 -- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 msgr2=0x7f7e78199880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:34:28.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.055+0000 7f7e7d7ad700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.055+0000 7f7e7d7ad700 1 -- 172.21.15.196:0/3936247313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e781a0230 con 0x7f7e640035c0 2024-03-28T18:34:28.057 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e7d7ad700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e7819f6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e740059d0 tx=0x7f7e740057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7e74004aa0 con 0x7f7e640035c0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e78199e20 con 0x7f7e640035c0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e74013e40 con 0x7f7e640035c0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7e74011c10 con 0x7f7e640035c0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.056+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7819a380 con 0x7f7e640035c0 2024-03-28T18:34:28.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.057+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e781033b0 con 0x7f7e640035c0 2024-03-28T18:34:28.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.058+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7e74023070 con 0x7f7e640035c0 2024-03-28T18:34:28.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.058+0000 7f7e6e7fc700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 0x7f7e5c03b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:28.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.058+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7e74020050 con 0x7f7e640035c0 2024-03-28T18:34:28.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.059+0000 7f7e7cfac700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 0x7f7e5c03b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:28.060 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.060+0000 7f7e7cfac700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 0x7f7e5c03b2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009180 tx=0x7f7e68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: mon.a calling monitor election 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: mon.c calling monitor election 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: fsmap 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: mgrmap e13: x(active, since 58s) 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: overall HEALTH_OK 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:28.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:28.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:28.072 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.072+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7e74031020 con 0x7f7e640035c0 2024-03-28T18:34:28.353 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.352+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e7819a6f0 con 0x7f7e640035c0 2024-03-28T18:34:28.354 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.353+0000 7f7e6e7fc700 1 -- 172.21.15.196:0/3936247313 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f7e74028030 con 0x7f7e640035c0 2024-03-28T18:34:28.354 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:34:28.354 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","modified":"2024-03-28T18:34:22.502265Z","created":"2024-03-28T18:32:18.696710Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 msgr2=0x7f7e5c03b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 0x7f7e5c03b2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009180 tx=0x7f7e68008040 comp rx=0 tx=0).stop 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 msgr2=0x7f7e7819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e7819f6b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e740059d0 tx=0x7f7e740057d0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 shutdown_connections 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7e5c038e20 0x7f7e5c03b2e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e640035c0 0x7f7e7819f6b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 --2- 172.21.15.196:0/3936247313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e7819fbf0 0x7f7e78199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 >> 172.21.15.196:0/3936247313 conn(0x7f7e780fd760 msgr2=0x7f7e780feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:28.356 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 shutdown_connections 2024-03-28T18:34:28.357 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:28.356+0000 7f7e7fa11700 1 -- 172.21.15.196:0/3936247313 wait complete. 2024-03-28T18:34:28.357 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-03-28T18:34:28.610 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:28 smithi196 ceph-mon[36300]: Deploying daemon mon.b on smithi131 2024-03-28T18:34:28.610 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:28 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/3936247313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:34:28.610 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:28 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:29.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:28 smithi119 ceph-mon[31813]: Deploying daemon mon.b on smithi131 2024-03-28T18:34:29.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:28 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/3936247313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T18:34:29.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:29.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:29 smithi119 ceph-mon[31813]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:29.820 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:34:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:34:29.503+0000 7f3ca1c00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T18:34:29.887 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T18:34:29.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T18:34:29.917 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:29 smithi196 ceph-mon[36300]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:30.130 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:34:31.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:31 smithi131 systemd[1]: Started Ceph mon.b for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:34:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: mon.c calling monitor election 2024-03-28T18:34:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:34:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: mon.a calling monitor election 2024-03-28T18:34:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: fsmap 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: mgrmap e13: x(active, since 67s) 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:34:36.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-28T18:34:36.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:36.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:36 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: mon.c calling monitor election 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: mon.a calling monitor election 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:37.025 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: fsmap 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: mgrmap e13: x(active, since 67s) 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.026 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:36 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 -- 172.21.15.196:0/2447757597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9440062a0 msgr2=0x7fb944006680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2447757597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9440062a0 0x7fb944006680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb95c102d30 tx=0x7fb94c00fe90 comp rx=0 tx=0).stop 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 -- 172.21.15.196:0/2447757597 shutdown_connections 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2447757597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9440062a0 0x7fb944006680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 -- 172.21.15.196:0/2447757597 >> 172.21.15.196:0/2447757597 conn(0x7fb95c0fd780 msgr2=0x7fb95c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 -- 172.21.15.196:0/2447757597 shutdown_connections 2024-03-28T18:34:37.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.589+0000 7fb960d4d700 1 -- 172.21.15.196:0/2447757597 wait complete. 2024-03-28T18:34:37.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 Processor -- start 2024-03-28T18:34:37.593 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 -- start start 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 0x7fb95c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c109c90 con 0x7fb95c197d70 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c109b10 con 0x7fb9440062a0 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb960d4d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c109e10 con 0x7fb95c19dba0 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb959d9b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb959d9b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:56132/0 (socket says 172.21.15.196:56132) 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb959d9b700 1 -- 172.21.15.196:0/2583193843 learned_addr learned my addr 172.21.15.196:0/2583193843 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb95a59c700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.591+0000 7fb95ad9d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 0x7fb95c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb95a59c700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 msgr2=0x7fb95c19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T18:34:37.594 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb95a59c700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 msgr2=0x7fb95c19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb95a59c700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb95a59c700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 msgr2=0x7fb95c19d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 msgr2=0x7fb95c198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 0x7fb95c198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 -- 172.21.15.196:0/2583193843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c1a44c0 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb95ad9d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 0x7fb95c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb959d9b700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb95000b780 tx=0x7fb95000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950004230 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9500043d0 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c1a4750 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950011c60 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.592+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c1a4cb0 con 0x7fb95c19dba0 2024-03-28T18:34:37.595 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.593+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb95001a400 con 0x7fb95c19dba0 2024-03-28T18:34:37.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.593+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c1033b0 con 0x7fb95c19dba0 2024-03-28T18:34:37.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.593+0000 7fb94b7fe700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 0x7fb93c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:37.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.593+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb95004bc30 con 0x7fb95c19dba0 2024-03-28T18:34:37.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.594+0000 7fb95a59c700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 0x7fb93c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:37.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.594+0000 7fb95a59c700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 0x7fb93c03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00f550 tx=0x7fb94c00f4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:37.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.599+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb95001a7f0 con 0x7fb95c19dba0 2024-03-28T18:34:37.701 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.701 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.701 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:37 smithi119 ceph-mon[31813]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:37.701 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:37.701 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.880 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.880 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:37.880 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:37 smithi196 ceph-mon[36300]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:37.881 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:37.881 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:37.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.899+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb95c02da30 con 0x7fb95c19dba0 2024-03-28T18:34:37.899 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.899+0000 7fb94b7fe700 1 -- 172.21.15.196:0/2583193843 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7fb950003c80 con 0x7fb95c19dba0 2024-03-28T18:34:37.900 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-28T18:34:37.900 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","modified":"2024-03-28T18:34:31.510240Z","created":"2024-03-28T18:32:18.696710Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 msgr2=0x7fb93c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 0x7fb93c03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00f550 tx=0x7fb94c00f4c0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 msgr2=0x7fb95c197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb95000b780 tx=0x7fb95000bb40 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 shutdown_connections 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb93c039450 0x7fb93c03b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb95c197d70 0x7fb95c198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb9440062a0 0x7fb95c19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 --2- 172.21.15.196:0/2583193843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb95c19dba0 0x7fb95c197830 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.902+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 >> 172.21.15.196:0/2583193843 conn(0x7fb95c0fd780 msgr2=0x7fb95c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.903+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 shutdown_connections 2024-03-28T18:34:37.903 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:37.903+0000 7fb960d4d700 1 -- 172.21.15.196:0/2583193843 wait complete. 2024-03-28T18:34:37.904 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: mon.c calling monitor election 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: mon.a calling monitor election 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: fsmap 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: mgrmap e13: x(active, since 67s) 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-28T18:34:38.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:38.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:39.119 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T18:34:39.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mon.b calling monitor election 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mon.b calling monitor election 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mon.c calling monitor election 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mon.a calling monitor election 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: fsmap 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:39.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: mgrmap e13: x(active, since 69s) 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: overall HEALTH_OK 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:39 smithi196 ceph-mon[36300]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.510 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mon.b calling monitor election 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mon.b calling monitor election 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mon.c calling monitor election 2024-03-28T18:34:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mon.a calling monitor election 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: fsmap 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: mgrmap e13: x(active, since 69s) 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: overall HEALTH_OK 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:39.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:39 smithi119 ceph-mon[31813]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mon.b calling monitor election 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mon.b calling monitor election 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mon.c calling monitor election 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mon.a calling monitor election 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: fsmap 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: mgrmap e13: x(active, since 69s) 2024-03-28T18:34:39.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: overall HEALTH_OK 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:39.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:39 smithi131 ceph-mon[36065]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:34:40.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:34:40.510+0000 7f3ca1c00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T18:34:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:40.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:40 smithi119 ceph-mon[31813]: Reconfiguring daemon mon.a on smithi119 2024-03-28T18:34:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:40 smithi196 ceph-mon[36300]: Reconfiguring daemon mon.a on smithi119 2024-03-28T18:34:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:40 smithi131 ceph-mon[36065]: Reconfiguring daemon mon.a on smithi119 2024-03-28T18:34:41.426 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:41 smithi119 ceph-mon[31813]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:41.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:41 smithi196 ceph-mon[36300]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:41 smithi131 ceph-mon[36065]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:42.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 -- 172.21.15.119:0/132013278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 msgr2=0x7f567410e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:42.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 --2- 172.21.15.119:0/132013278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f567410e450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5664009910 tx=0x7f5664009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 -- 172.21.15.119:0/132013278 shutdown_connections 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 --2- 172.21.15.119:0/132013278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f567410e450 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 -- 172.21.15.119:0/132013278 >> 172.21.15.119:0/132013278 conn(0x7f5674077760 msgr2=0x7f5674079b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.426+0000 7f5678e00700 1 -- 172.21.15.119:0/132013278 shutdown_connections 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.427+0000 7f5678e00700 1 -- 172.21.15.119:0/132013278 wait complete. 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 Processor -- start 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 -- start start 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56741a3e00 0x7f567419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 0x7f567419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56741125c0 con 0x7f567410e070 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5674112440 con 0x7f56741a3e00 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5678e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5674112740 con 0x7f567419dfc0 2024-03-28T18:34:42.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.428+0000 7f5673400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34714/0 (socket says 172.21.15.119:34714) 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 -- 172.21.15.119:0/3557874879 learned_addr learned my addr 172.21.15.119:0/3557874879 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5672a00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56741a3e00 0x7f567419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 0x7f567419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 -- 172.21.15.119:0/3557874879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 msgr2=0x7f567419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 0x7f567419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 -- 172.21.15.119:0/3557874879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56741a3e00 msgr2=0x7f567419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56741a3e00 0x7f567419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 -- 172.21.15.119:0/3557874879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56741aa790 con 0x7f567410e070 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 0x7f567419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f5673400700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56640092e0 tx=0x7f5664004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.429+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664026070 con 0x7f567410e070 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.430+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f566400b6f0 con 0x7f567410e070 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.430+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56741aa9c0 con 0x7f567410e070 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.430+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640038e0 con 0x7f567410e070 2024-03-28T18:34:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.430+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56741aafd0 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.431+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f5664029070 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.431+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56740637c0 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.431+0000 7f566be00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 0x7f565803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.431+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f566404cbf0 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.431+0000 7f5672a00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 0x7f565803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.432+0000 7f5672a00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 0x7f565803b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5660009ad0 tx=0x7f5660009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.437+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5664013070 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.661+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f56741ab300 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.662+0000 7f566be00700 1 -- 172.21.15.119:0/3557874879 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f5664030090 con 0x7f567410e070 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stdout:# minimal ceph.conf for 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-03-28T18:34:42.120 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stdout: mon_host = [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 msgr2=0x7f565803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 0x7f565803b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5660009ad0 tx=0x7f5660009210 comp rx=0 tx=0).stop 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 msgr2=0x7f56741a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56640092e0 tx=0x7f5664004fa0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 shutdown_connections 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5658039450 0x7f565803b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f567419dfc0 0x7f567419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567410e070 0x7f56741a38c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 --2- 172.21.15.119:0/3557874879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56741a3e00 0x7f567419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:42.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 >> 172.21.15.119:0/3557874879 conn(0x7f5674077760 msgr2=0x7f5674077b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:42.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 shutdown_connections 2024-03-28T18:34:42.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:41.665+0000 7f5678e00700 1 -- 172.21.15.119:0/3557874879 wait complete. 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3557874879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:42 smithi119 ceph-mon[31813]: Reconfiguring daemon mon.b on smithi131 2024-03-28T18:34:42.681 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3557874879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.681 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.682 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:42 smithi131 ceph-mon[36065]: Reconfiguring daemon mon.b on smithi131 2024-03-28T18:34:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3557874879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: Reconfiguring mon.b (monmap changed)... 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:42 smithi196 ceph-mon[36300]: Reconfiguring daemon mon.b on smithi131 2024-03-28T18:34:42.818 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T18:34:42.818 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:34:42.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:34:42.919 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:34:42.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:34:43.058 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:34:43.058 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:34:43.088 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:34:43.088 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:34:43.155 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:34:43.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T18:34:43.190 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:34:43.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:34:43.260 INFO:tasks.cephadm:Adding mgr.x on smithi119 2024-03-28T18:34:43.261 INFO:tasks.cephadm:Adding mgr.y on smithi131 2024-03-28T18:34:43.261 INFO:tasks.cephadm:Adding mgr.z on smithi196 2024-03-28T18:34:43.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi119=x;smithi131=y;smithi196=z' 2024-03-28T18:34:43.533 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:34:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:43 smithi119 ceph-mon[31813]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:43.626 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:43 smithi131 ceph-mon[36065]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:43.655 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:43 smithi196 ceph-mon[36300]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:44.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:44.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:44.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:34:44.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:44.731 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:44.731 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:44.731 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:44 smithi196 ceph-mon[36300]: Reconfiguring daemon mon.c on smithi196 2024-03-28T18:34:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:44.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:34:44.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:44.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:44.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:44.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:44 smithi131 ceph-mon[36065]: Reconfiguring daemon mon.c on smithi196 2024-03-28T18:34:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: Reconfiguring mon.c (monmap changed)... 2024-03-28T18:34:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T18:34:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T18:34:45.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:45.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:44 smithi119 ceph-mon[31813]: Reconfiguring daemon mon.c on smithi196 2024-03-28T18:34:45.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 -- 172.21.15.196:0/2807685874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 msgr2=0x7f9e8c104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 --2- 172.21.15.196:0/2807685874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c104be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009910 tx=0x7f9e7c009c20 comp rx=0 tx=0).stop 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 -- 172.21.15.196:0/2807685874 shutdown_connections 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 --2- 172.21.15.196:0/2807685874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c104be0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 -- 172.21.15.196:0/2807685874 >> 172.21.15.196:0/2807685874 conn(0x7f9e8c0fc1a0 msgr2=0x7f9e8c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 -- 172.21.15.196:0/2807685874 shutdown_connections 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 -- 172.21.15.196:0/2807685874 wait complete. 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.459+0000 7f9e9199f700 1 Processor -- start 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 -- start start 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c102150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e8c1007a0 0x7f9e8c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c107d50 con 0x7f9e8c1027f0 2024-03-28T18:34:45.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c107bd0 con 0x7f9e8c1007a0 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e9199f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8c107ed0 con 0x7f9e8c101160 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35544/0 (socket says 172.21.15.196:35544) 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 -- 172.21.15.196:0/29742929 learned_addr learned my addr 172.21.15.196:0/29742929 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8a7fc700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e8c1007a0 0x7f9e8c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8affd700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c102150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 -- 172.21.15.196:0/29742929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e8c1007a0 msgr2=0x7f9e8c100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e8c1007a0 0x7f9e8c100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 -- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 msgr2=0x7f9e8c102150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c102150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8b7fe700 1 -- 172.21.15.196:0/29742929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8c1a1ae0 con 0x7f9e8c101160 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.460+0000 7f9e8affd700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e8b7fe700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400ba50 tx=0x7f9e8400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84019070 con 0x7f9e8c101160 2024-03-28T18:34:45.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c1a1dd0 con 0x7f9e8c101160 2024-03-28T18:34:45.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e84014440 con 0x7f9e8c101160 2024-03-28T18:34:45.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840075b0 con 0x7f9e8c101160 2024-03-28T18:34:45.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.461+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c1a22b0 con 0x7f9e8c101160 2024-03-28T18:34:45.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.462+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f9e8401c070 con 0x7f9e8c101160 2024-03-28T18:34:45.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.462+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8c0637c0 con 0x7f9e8c101160 2024-03-28T18:34:45.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.462+0000 7f9e9099d700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 0x7f9e7403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:45.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.462+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9e8404b5b0 con 0x7f9e8c101160 2024-03-28T18:34:45.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.462+0000 7f9e8affd700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 0x7f9e7403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:45.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.463+0000 7f9e8affd700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 0x7f9e7403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c005330 tx=0x7f9e7c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:45.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.467+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e84012500 con 0x7f9e8c101160 2024-03-28T18:34:45.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.678+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi119=x;smithi131=y;smithi196=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e8c0625d0 con 0x7f9e74039450 2024-03-28T18:34:45.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.691+0000 7f9e9099d700 1 -- 172.21.15.196:0/29742929 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9e8c0625d0 con 0x7f9e74039450 2024-03-28T18:34:45.692 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 msgr2=0x7f9e7403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 0x7f9e7403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c005330 tx=0x7f9e7c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 msgr2=0x7f9e8c1a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400ba50 tx=0x7f9e8400bd60 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 shutdown_connections 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e74039450 0x7f9e7403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e8c101160 0x7f9e8c1a1520 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e8c1027f0 0x7f9e8c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 --2- 172.21.15.196:0/29742929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9e8c1007a0 0x7f9e8c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:45.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 >> 172.21.15.196:0/29742929 conn(0x7f9e8c0fc1a0 msgr2=0x7f9e8c0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:45.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 shutdown_connections 2024-03-28T18:34:45.695 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:34:45.694+0000 7f9e9199f700 1 -- 172.21.15.196:0/29742929 wait complete. 2024-03-28T18:34:45.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:45 smithi196 ceph-mon[36300]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:45 smithi119 ceph-mon[31813]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:46.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:45 smithi131 ceph-mon[36065]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:46.374 DEBUG:teuthology.orchestra.run.smithi131:mgr.y> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T18:34:46.377 DEBUG:teuthology.orchestra.run.smithi196:mgr.z> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T18:34:46.379 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T18:34:46.379 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:34:46.379 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:34:46.436 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:34:46.436 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-03-28T18:34:46.486 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:34:46.486 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:46.486 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-03-28T18:34:46.486 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:46.487 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:46.487 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-28 18:33:44.375454484 +0000 2024-03-28T18:34:46.487 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-28 18:28:30.161147976 +0000 2024-03-28T18:34:46.487 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-28 18:28:30.161147976 +0000 2024-03-28T18:34:46.487 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-28 18:28:30.161147976 +0000 2024-03-28T18:34:46.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:34:46.569 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-28T18:34:46.570 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-28T18:34:46.570 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000374756 s, 1.4 MB/s 2024-03-28T18:34:46.573 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:34:46.628 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-03-28T18:34:46.678 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-28 18:33:44.443452907 +0000 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-28 18:28:30.713135214 +0000 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-28 18:28:30.713135214 +0000 2024-03-28T18:34:46.679 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-28 18:28:30.713135214 +0000 2024-03-28T18:34:46.679 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:34:46.759 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-28T18:34:46.759 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-28T18:34:46.759 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000350101 s, 1.5 MB/s 2024-03-28T18:34:46.762 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:34:46.815 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi119=x;smithi131=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: Saving service mgr spec with placement smithi119=x;smithi131=y;smithi196=z;count:3 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:46.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:46.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:46.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:46.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:46 smithi119 ceph-mon[31813]: Deploying daemon mgr.z on smithi196 2024-03-28T18:34:46.862 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:34:46.862 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:46.862 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-03-28T18:34:46.862 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:46.862 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:46.863 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-28 18:33:44.501451561 +0000 2024-03-28T18:34:46.863 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-28 18:28:31.295121757 +0000 2024-03-28T18:34:46.863 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-28 18:28:31.295121757 +0000 2024-03-28T18:34:46.863 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-28 18:28:31.295121757 +0000 2024-03-28T18:34:46.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:34:46.947 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-28T18:34:46.947 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-28T18:34:46.948 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000316661 s, 1.6 MB/s 2024-03-28T18:34:46.952 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:34:47.006 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-28 18:33:44.561450169 +0000 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-28 18:28:31.887108070 +0000 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-28 18:28:31.887108070 +0000 2024-03-28T18:34:47.057 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-28 18:28:31.887108070 +0000 2024-03-28T18:34:47.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi119=x;smithi131=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: Saving service mgr spec with placement smithi119=x;smithi131=y;smithi196=z;count:3 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:47.066 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:47.067 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:46 smithi196 ceph-mon[36300]: Deploying daemon mgr.z on smithi196 2024-03-28T18:34:47.146 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-28T18:34:47.146 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-28T18:34:47.147 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000390371 s, 1.3 MB/s 2024-03-28T18:34:47.151 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:34:47.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi119=x;smithi131=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: Saving service mgr spec with placement smithi119=x;smithi131=y;smithi196=z;count:3 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:47.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:46 smithi131 ceph-mon[36065]: Deploying daemon mgr.z on smithi196 2024-03-28T18:34:47.207 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-28T18:34:47.207 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:34:47.224 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:34:47.225 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-28 18:34:30.972244110 +0000 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-28 18:27:19.093694776 +0000 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-28 18:27:19.093694776 +0000 2024-03-28T18:34:47.281 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-28 18:27:19.093694776 +0000 2024-03-28T18:34:47.281 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:34:47.347 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-28T18:34:47.347 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-28T18:34:47.347 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000214302 s, 2.4 MB/s 2024-03-28T18:34:47.348 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:34:47.404 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-03-28T18:34:47.461 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:34:47.461 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:47.461 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T18:34:47.461 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:47.461 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:47.462 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-28 18:34:30.973244089 +0000 2024-03-28T18:34:47.462 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-28 18:27:19.445687233 +0000 2024-03-28T18:34:47.462 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-28 18:27:19.445687233 +0000 2024-03-28T18:34:47.462 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-28 18:27:19.445687233 +0000 2024-03-28T18:34:47.462 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:34:47.528 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-28T18:34:47.528 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-28T18:34:47.528 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.00016421 s, 3.1 MB/s 2024-03-28T18:34:47.529 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:34:47.587 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:47.641 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-28 18:34:30.973244089 +0000 2024-03-28T18:34:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-28 18:27:19.783679989 +0000 2024-03-28T18:34:47.642 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-28 18:27:19.783679989 +0000 2024-03-28T18:34:47.642 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-28 18:27:19.783679989 +0000 2024-03-28T18:34:47.642 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:34:47.708 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-28T18:34:47.708 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-28T18:34:47.709 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000167076 s, 3.1 MB/s 2024-03-28T18:34:47.710 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:34:47.766 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-03-28T18:34:47.820 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:34:47.820 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:47.820 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-28 18:34:30.973244089 +0000 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-28 18:27:20.111672960 +0000 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-28 18:27:20.111672960 +0000 2024-03-28T18:34:47.821 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-28 18:27:20.111672960 +0000 2024-03-28T18:34:47.821 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:34:47.883 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:47 smithi131 ceph-mon[36065]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:47.886 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-28T18:34:47.886 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-28T18:34:47.886 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000177331 s, 2.9 MB/s 2024-03-28T18:34:47.887 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:34:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:47 smithi196 ceph-mon[36300]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:47.935 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:47 smithi196 systemd[1]: Starting Ceph mgr.z for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:34:47.941 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-28T18:34:47.941 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T18:34:47.957 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T18:34:47.957 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-28T18:34:48.011 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-28 18:34:47.720364772 +0000 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-28 18:27:24.802101953 +0000 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-28 18:27:24.802101953 +0000 2024-03-28T18:34:48.012 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-28 18:27:24.802101953 +0000 2024-03-28T18:34:48.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T18:34:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:47 smithi119 ceph-mon[31813]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:48.080 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-28T18:34:48.080 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-28T18:34:48.080 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000272674 s, 1.9 MB/s 2024-03-28T18:34:48.081 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T18:34:48.137 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-28T18:34:48.191 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T18:34:48.191 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:48.191 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T18:34:48.191 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:48.191 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:48.192 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-28 18:34:47.721364750 +0000 2024-03-28T18:34:48.192 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-28 18:27:25.153094361 +0000 2024-03-28T18:34:48.192 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-28 18:27:25.153094361 +0000 2024-03-28T18:34:48.192 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-28 18:27:25.153094361 +0000 2024-03-28T18:34:48.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T18:34:48.254 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:47 smithi196 podman[37920]: 2024-03-28 18:34:47.900967502 +0000 UTC m=+0.017125357 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:34:48.255 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 podman[37920]: 2024-03-28 18:34:48.011090171 +0000 UTC m=+0.127248005 container create 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:34:48.255 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 podman[37920]: 2024-03-28 18:34:48.255411342 +0000 UTC m=+0.371569191 container init 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T18:34:48.262 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-28T18:34:48.262 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-28T18:34:48.262 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000176167 s, 2.9 MB/s 2024-03-28T18:34:48.263 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T18:34:48.317 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-28 18:34:47.721364750 +0000 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-28 18:27:25.502086813 +0000 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-28 18:27:25.502086813 +0000 2024-03-28T18:34:48.373 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-28 18:27:25.502086813 +0000 2024-03-28T18:34:48.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T18:34:48.439 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-28T18:34:48.439 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-28T18:34:48.439 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000194531 s, 2.6 MB/s 2024-03-28T18:34:48.441 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T18:34:48.496 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-28T18:34:48.550 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-28 18:34:47.721364750 +0000 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-28 18:27:25.845079394 +0000 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-28 18:27:25.845079394 +0000 2024-03-28T18:34:48.551 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-28 18:27:25.845079394 +0000 2024-03-28T18:34:48.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T18:34:48.614 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 podman[37920]: 2024-03-28 18:34:48.25894151 +0000 UTC m=+0.375099349 container start 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:34:48.615 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 ceph-mgr[38041]: -- 172.21.15.196:0/2309299727 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592ca3725a0 con 0x5592ca346800 2024-03-28T18:34:48.615 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 bash[37920]: 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 2024-03-28T18:34:48.615 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 systemd[1]: Started Ceph mgr.z for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:34:48.617 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-28T18:34:48.617 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-28T18:34:48.617 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000187389 s, 2.7 MB/s 2024-03-28T18:34:48.618 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T18:34:48.673 INFO:tasks.cephadm:Deploying osd.0 on smithi119 with /dev/vg_nvme/lv_4... 2024-03-28T18:34:48.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:34:48.908 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:48.816+0000 7fa9c393f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:34:48.909 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:48.908+0000 7fa9c393f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:34:49.111 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:49.533 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: Deploying daemon mgr.y on smithi131 2024-03-28T18:34:49.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:49 smithi119 ceph-mon[31813]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:49.691 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:49.542+0000 7fa9c393f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:34:49.691 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:49.634+0000 7fa9c393f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:34:49.691 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: Deploying daemon mgr.y on smithi131 2024-03-28T18:34:49.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-mon[36300]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:49.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: Deploying daemon mgr.y on smithi131 2024-03-28T18:34:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:49 smithi131 ceph-mon[36065]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:49.952 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:49.779+0000 7fa9c393f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:34:49.952 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:49.876+0000 7fa9c393f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:34:50.238 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:49.963+0000 7fa9c393f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:34:50.238 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:50.237+0000 7fa9c393f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:34:50.535 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:50.351+0000 7fa9c393f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:34:50.535 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:50.439+0000 7fa9c393f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:34:50.535 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:50.535+0000 7fa9c393f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:34:50.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 systemd[1]: Starting Ceph mgr.y for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:34:50.919 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:50.715+0000 7fa9c393f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 podman[37439]: 2024-03-28 18:34:50.709480781 +0000 UTC m=+0.108474115 container create 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 podman[37439]: 2024-03-28 18:34:50.618827707 +0000 UTC m=+0.017821052 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 podman[37439]: 2024-03-28 18:34:50.903362509 +0000 UTC m=+0.302355847 container init 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 podman[37439]: 2024-03-28 18:34:50.905918257 +0000 UTC m=+0.304911588 container start 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:50 smithi131 ceph-mgr[37458]: -- 172.21.15.131:0/347055028 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564a8759c5a0 con 0x564a87571000 2024-03-28T18:34:51.003 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:51 smithi131 bash[37439]: 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e 2024-03-28T18:34:51.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:51 smithi131 systemd[1]: Started Ceph mgr.y for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0367304 s, 285 MB/s 2024-03-28T18:34:51.840 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-28T18:34:51.944 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:51 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:51.452+0000 7f1b65050200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:34:51.944 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:51 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:51.550+0000 7f1b65050200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:34:52.075 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:51.705+0000 7fa9c393f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:34:52.075 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:51.846+0000 7fa9c393f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:52.344 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.248+0000 7f1b65050200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:34:52.344 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.344+0000 7f1b65050200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.345 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:52.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:52.435 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:52.203+0000 7fa9c393f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:34:52.435 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:52.298+0000 7fa9c393f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:34:52.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-03-28T18:34:52.665 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.486+0000 7f1b65050200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:34:52.665 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.580+0000 7f1b65050200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:34:52.665 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.665+0000 7f1b65050200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:34:52.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:52.494+0000 7fa9c393f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:34:52.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:52.686+0000 7fa9c393f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:34:53.028 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:34:53.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:52.934+0000 7f1b65050200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:34:53.195 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:53.040+0000 7f1b65050200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:34:53.195 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:53.124+0000 7f1b65050200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:34:53.234 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:52.974+0000 7fa9c393f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:34:53.234 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:53.234+0000 7fa9c393f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:34:53.679 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:53.216+0000 7f1b65050200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:34:53.679 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:53.388+0000 7f1b65050200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:34:53.680 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:53.468+0000 7fa9c393f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:53 smithi119 ceph-mon[31813]: Reconfiguring daemon mgr.x on smithi119 2024-03-28T18:34:53.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:53 smithi196 ceph-mon[36300]: Reconfiguring daemon mgr.x on smithi119 2024-03-28T18:34:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:53 smithi131 ceph-mon[36065]: Reconfiguring daemon mgr.x on smithi119 2024-03-28T18:34:54.685 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:54.267+0000 7fa9c393f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:34:54.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:54.345+0000 7f1b65050200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:34:54.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:54.471+0000 7f1b65050200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:34:55.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:54 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:55.076 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:54.799+0000 7f1b65050200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:34:55.076 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:54.885+0000 7f1b65050200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:34:55.076 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:55.075+0000 7f1b65050200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:34:55.147 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:54 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:55.148 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:54.834+0000 7fa9c393f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:34:55.148 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:54.931+0000 7fa9c393f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:34:55.148 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:34:55.147+0000 7fa9c393f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:34:55.160 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:54 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:55.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:55.256+0000 7f1b65050200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:34:55.799 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:55.537+0000 7f1b65050200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:34:55.799 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:55.799+0000 7f1b65050200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:34:55.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 -- 172.21.15.119:0/3067896241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 msgr2=0x7f8f98105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:55.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 --2- 172.21.15.119:0/3067896241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f98105840 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8f840098f0 tx=0x7f8f84009c00 comp rx=0 tx=0).stop 2024-03-28T18:34:55.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 -- 172.21.15.119:0/3067896241 shutdown_connections 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 --2- 172.21.15.119:0/3067896241 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 0x7f8f98106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 --2- 172.21.15.119:0/3067896241 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f98102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 --2- 172.21.15.119:0/3067896241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f98105840 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 -- 172.21.15.119:0/3067896241 >> 172.21.15.119:0/3067896241 conn(0x7f8f980fc660 msgr2=0x7f8f980feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 -- 172.21.15.119:0/3067896241 shutdown_connections 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.898+0000 7f8f9e200700 1 -- 172.21.15.119:0/3067896241 wait complete. 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.899+0000 7f8f9e200700 1 Processor -- start 2024-03-28T18:34:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.899+0000 7f8f9e200700 1 -- start start 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 0x7f8f981a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f98114350 con 0x7f8f981033b0 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f981141d0 con 0x7f8f98105d80 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f9e200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f981144d0 con 0x7f8f98100a40 2024-03-28T18:34:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f96a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58846/0 (socket says 172.21.15.119:58846) 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f96a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38616/0 (socket says 172.21.15.119:38616) 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97400700 1 -- 172.21.15.119:0/756985962 learned_addr learned my addr 172.21.15.119:0/756985962 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97e00700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 0x7f8f981a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97400700 1 -- 172.21.15.119:0/756985962 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 msgr2=0x7f8f981a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.900+0000 7f8f97400700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 0x7f8f981a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f97400700 1 -- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 msgr2=0x7f8f9819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f97400700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f97400700 1 -- 172.21.15.119:0/756985962 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f981a6650 con 0x7f8f98100a40 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f96a00700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:34:55.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f97400700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e9d0 tx=0x7f8f8800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8800cbd0 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f8800ee70 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88018660 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f981a6940 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.901+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f981a6e80 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.902+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f8f88010430 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.902+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f98068e50 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.903+0000 7f8f8fe00700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 0x7f8f7c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.903+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8f8804c430 con 0x7f8f98100a40 2024-03-28T18:34:55.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.903+0000 7f8f96a00700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 0x7f8f7c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:34:55.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.904+0000 7f8f96a00700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 0x7f8f7c03b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400fb70 tx=0x7f8f84019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:34:55.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.909+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f88018d20 con 0x7f8f98100a40 2024-03-28T18:34:55.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:55.926+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f8f88018800 con 0x7f8f98100a40 2024-03-28T18:34:56.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:56.147+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f980625d0 con 0x7f8f7c039200 2024-03-28T18:34:56.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:56.032+0000 7f1b65050200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:34:56.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: Standby manager daemon z started 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: Standby manager daemon z started 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:34:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: Standby manager daemon z started 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1618186118' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:34:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:34:57.195 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:56.824+0000 7f1b65050200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:34:57.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T18:34:57.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:34:57.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:57.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:57.196 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:57.196 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:57.211 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-28T18:34:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:34:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:34:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:34:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:34:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:34:57.695 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:57.420+0000 7f1b65050200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:34:57.695 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:57.516+0000 7f1b65050200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:34:57.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:34:57.966+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mon.1 v2:172.21.15.196:3300/0 8 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f8f88014230 con 0x7f8f98100a40 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: Standby manager daemon y started 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:34:57.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:57 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:58.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:34:57.726+0000 7f1b65050200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: Standby manager daemon y started 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:34:58.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:57 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:34:58.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: Standby manager daemon y started 2024-03-28T18:34:58.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:34:58.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:34:58.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:34:58.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:57 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/3115780552' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:34:59.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:58 smithi119 ceph-mon[31813]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-28T18:34:59.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:34:59.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:58 smithi196 ceph-mon[36300]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-28T18:34:59.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:34:59.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:58 smithi131 ceph-mon[36065]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-28T18:34:59.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:35:00.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:59 smithi119 ceph-mon[31813]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:00.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]: dispatch 2024-03-28T18:35:00.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]': finished 2024-03-28T18:35:00.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:59 smithi119 ceph-mon[31813]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:35:00.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:34:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:00.405 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:59 smithi196 ceph-mon[36300]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:00.405 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]: dispatch 2024-03-28T18:35:00.406 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]': finished 2024-03-28T18:35:00.406 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:59 smithi196 ceph-mon[36300]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:35:00.406 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:34:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:59 smithi131 ceph-mon[36065]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]: dispatch 2024-03-28T18:35:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3744849954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9791ee-9ecd-4e9e-b59e-90533047d529"}]': finished 2024-03-28T18:35:00.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:59 smithi131 ceph-mon[36065]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T18:35:00.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:34:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:00 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/850484714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:01.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:00 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/850484714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:00 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/850484714' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:01 smithi119 ceph-mon[31813]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:01 smithi196 ceph-mon[36300]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:01 smithi131 ceph-mon[36065]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:03 smithi119 ceph-mon[31813]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:04.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:03 smithi196 ceph-mon[36300]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:03 smithi131 ceph-mon[36065]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:06.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:05 smithi119 ceph-mon[31813]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:05 smithi196 ceph-mon[36300]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:05 smithi131 ceph-mon[36065]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:08.263 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:07 smithi119 ceph-mon[31813]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:08.423 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:07 smithi196 ceph-mon[36300]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:07 smithi131 ceph-mon[36065]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:35:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:35:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T18:35:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:10.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:10 smithi119 ceph-mon[31813]: Deploying daemon osd.0 on smithi119 2024-03-28T18:35:10.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:10 smithi119 ceph-mon[31813]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:10.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:10 smithi196 ceph-mon[36300]: Deploying daemon osd.0 on smithi119 2024-03-28T18:35:10.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:10 smithi196 ceph-mon[36300]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:10 smithi131 ceph-mon[36065]: Deploying daemon osd.0 on smithi119 2024-03-28T18:35:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:10 smithi131 ceph-mon[36065]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:12.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:12 smithi119 ceph-mon[31813]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:12.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:12 smithi196 ceph-mon[36300]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:12 smithi131 ceph-mon[36065]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:14 smithi119 ceph-mon[31813]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:14.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:14 smithi196 ceph-mon[36300]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:14 smithi131 ceph-mon[36065]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:16.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:16 smithi119 ceph-mon[31813]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:16 smithi196 ceph-mon[36300]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:16 smithi131 ceph-mon[36065]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:17.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:17.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:17 smithi119 ceph-mon[31813]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:17.614 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.614 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.614 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:17.614 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:17 smithi131 ceph-mon[36065]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:17 smithi196 ceph-mon[36300]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:20.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:20.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:20.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:20.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:20.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:20.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:20.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:20.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.582+0000 7f8f8fe00700 1 -- 172.21.15.119:0/756985962 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8f980625d0 con 0x7f8f7c039200 2024-03-28T18:35:20.583 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 0 on host 'smithi119' 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 msgr2=0x7f8f7c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 0x7f8f7c03b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400fb70 tx=0x7f8f84019040 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 msgr2=0x7f8f9819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e9d0 tx=0x7f8f8800ece0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 shutdown_connections 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f8f7c039200 0x7f8f7c03b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f98105d80 0x7f8f981a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f98100a40 0x7f8f9819e9f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 --2- 172.21.15.119:0/756985962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f981033b0 0x7f8f9819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 >> 172.21.15.119:0/756985962 conn(0x7f8f980fc660 msgr2=0x7f8f981013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 shutdown_connections 2024-03-28T18:35:20.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:35:20.586+0000 7f8f9e200700 1 -- 172.21.15.119:0/756985962 wait complete. 2024-03-28T18:35:21.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:20 smithi196 ceph-mon[36300]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:35:21.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:21.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:20 smithi131 ceph-mon[36065]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:35:21.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:21.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.300 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:20 smithi119 ceph-mon[31813]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T18:35:21.300 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:21.300 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:21.361 DEBUG:teuthology.orchestra.run.smithi119:osd.0> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T18:35:21.365 INFO:tasks.cephadm:Deploying osd.1 on smithi131 with /dev/vg_nvme/lv_4... 2024-03-28T18:35:21.365 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:35:21.610 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:35:22.056 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:21 smithi131 ceph-mon[36065]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:22.056 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:21 smithi131 ceph-mon[36065]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:35:22.056 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:21 smithi131 ceph-mon[36065]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:35:22.057 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:21 smithi131 ceph-mon[36065]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:35:22.057 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:21 smithi119 ceph-mon[31813]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:21 smithi119 ceph-mon[31813]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:35:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:21 smithi119 ceph-mon[31813]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:35:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:21 smithi119 ceph-mon[31813]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:35:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:21 smithi196 ceph-mon[36300]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:21 smithi196 ceph-mon[36300]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T18:35:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:21 smithi196 ceph-mon[36300]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T18:35:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:21 smithi196 ceph-mon[36300]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:35:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:22.754 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0[46839]: 2024-03-28T18:35:22.492+0000 7efdb1a00700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T18:35:22.755 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0[46839]: 2024-03-28T18:35:22.519+0000 7efda6e00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:35:23.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-mon[31813]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:35:23.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-mon[31813]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:35:23.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:22 smithi119 ceph-mon[31813]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' 2024-03-28T18:35:23.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:22 smithi131 ceph-mon[36065]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:35:23.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:22 smithi131 ceph-mon[36065]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:35:23.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:22 smithi131 ceph-mon[36065]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' 2024-03-28T18:35:23.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:22 smithi196 ceph-mon[36300]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:35:23.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:22 smithi196 ceph-mon[36300]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T18:35:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:22 smithi196 ceph-mon[36300]: from='osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015]' entity='osd.0' 2024-03-28T18:35:23.503 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:35:23.503 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:35:23.503 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-03-28T18:35:23.503 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-03-28T18:35:23.504 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012156 s, 863 MB/s 2024-03-28T18:35:23.504 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] boot 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:35:24.115 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:24.140 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_4 2024-03-28T18:35:24.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] boot 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:35:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:24.373 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:35:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:35:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:35:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T18:35:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: osd.0 [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] boot 2024-03-28T18:35:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T18:35:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:35:25.889 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.887+0000 7f8451c65700 1 -- 172.21.15.131:0/2742478086 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c103d60 msgr2=0x7f844c10a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.887+0000 7f8451c65700 1 --2- 172.21.15.131:0/2742478086 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c103d60 0x7f844c10a610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f844400b0d0 tx=0x7f844400b3e0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 -- 172.21.15.131:0/2742478086 shutdown_connections 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 --2- 172.21.15.131:0/2742478086 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c103d60 0x7f844c10a610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 --2- 172.21.15.131:0/2742478086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1021c0 0x7f844c1025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 --2- 172.21.15.131:0/2742478086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f844c1033c0 0x7f844c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 -- 172.21.15.131:0/2742478086 >> 172.21.15.131:0/2742478086 conn(0x7f844c0fd7a0 msgr2=0x7f844c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 -- 172.21.15.131:0/2742478086 shutdown_connections 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 -- 172.21.15.131:0/2742478086 wait complete. 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 Processor -- start 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.888+0000 7f8451c65700 1 -- start start 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.890 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1033c0 0x7f844c074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f844c103d60 0x7f844c074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10de30 con 0x7f844c103d60 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10dcb0 con 0x7f844c1021c0 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c10dfb0 con 0x7f844c1033c0 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:60068/0 (socket says 172.21.15.131:60068) 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 -- 172.21.15.131:0/1064316073 learned_addr learned my addr 172.21.15.131:0/1064316073 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844affd700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1033c0 0x7f844c074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 -- 172.21.15.131:0/1064316073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1033c0 msgr2=0x7f844c074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1033c0 0x7f844c074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 -- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f844c103d60 msgr2=0x7f844c074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f844c103d60 0x7f844c074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 -- 172.21.15.131:0/1064316073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c075370 con 0x7f844c1021c0 2024-03-28T18:35:25.891 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f844b7fe700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d980 tx=0x7f843c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c017070 con 0x7f844c1021c0 2024-03-28T18:35:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f843c009e10 con 0x7f844c1021c0 2024-03-28T18:35:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c1a2550 con 0x7f844c1021c0 2024-03-28T18:35:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c00f620 con 0x7f844c1021c0 2024-03-28T18:35:25.892 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.889+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1a2ae0 con 0x7f844c1021c0 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.890+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c068e50 con 0x7f844c1021c0 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.891+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f843c009930 con 0x7f844c1021c0 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.892+0000 7f8448ff9700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 0x7f84340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.892+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f843c0ccaf0 con 0x7f844c1021c0 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.892+0000 7f844affd700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 0x7f84340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:25.893 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.893+0000 7f844affd700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 0x7f84340a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8440009a10 tx=0x7f8440009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:25.896 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:25.897+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f843c09b600 con 0x7f844c1021c0 2024-03-28T18:35:26.099 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:26.099+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f844c02de10 con 0x7f84340a2880 2024-03-28T18:35:26.100 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:25 smithi131 ceph-mon[36065]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:26.100 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:25 smithi131 ceph-mon[36065]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:35:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:25 smithi119 ceph-mon[31813]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:25 smithi119 ceph-mon[31813]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:35:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:25 smithi196 ceph-mon[36300]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:25 smithi196 ceph-mon[36300]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi119 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 17091M 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.027 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:27.028 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:26 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi119 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 17091M 2024-03-28T18:35:27.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:27.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:26 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi119 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 17091M 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:26 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:28.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:28 smithi131 ceph-mon[36065]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:28 smithi196 ceph-mon[36300]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:28 smithi119 ceph-mon[31813]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:29.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:29 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/840252645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:29 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:29 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]': finished 2024-03-28T18:35:29.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:29 smithi196 ceph-mon[36300]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:35:29.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:29 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/840252645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:29 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:29 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]': finished 2024-03-28T18:35:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:29 smithi131 ceph-mon[36065]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:35:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:29 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/840252645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:29 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]: dispatch 2024-03-28T18:35:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:29 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56d8077-84c3-494e-8b08-9f5ea7b7cfa4"}]': finished 2024-03-28T18:35:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:29 smithi119 ceph-mon[31813]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T18:35:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:30 smithi196 ceph-mon[36300]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:30 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/3058973462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:30 smithi131 ceph-mon[36065]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:30 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/3058973462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:30 smithi119 ceph-mon[31813]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/3058973462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:31 smithi119 ceph-mon[31813]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:31 smithi196 ceph-mon[36300]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:31.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:31 smithi131 ceph-mon[36065]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:34.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:33 smithi196 ceph-mon[36300]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:34.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:33 smithi131 ceph-mon[36065]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:33 smithi119 ceph-mon[31813]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:35 smithi131 ceph-mon[36065]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:35:35.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:36.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:35 smithi196 ceph-mon[36300]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:36.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:35:36.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:35 smithi119 ceph-mon[31813]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T18:35:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:37.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:36 smithi196 ceph-mon[36300]: Deploying daemon osd.1 on smithi131 2024-03-28T18:35:37.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:36 smithi131 ceph-mon[36065]: Deploying daemon osd.1 on smithi131 2024-03-28T18:35:37.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:36 smithi119 ceph-mon[31813]: Deploying daemon osd.1 on smithi131 2024-03-28T18:35:38.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:37 smithi196 ceph-mon[36300]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:38.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:37 smithi131 ceph-mon[36065]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:37 smithi119 ceph-mon[31813]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:40.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:39 smithi131 ceph-mon[36065]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:40.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:39 smithi119 ceph-mon[31813]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:40.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:39 smithi196 ceph-mon[36300]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:42 smithi196 ceph-mon[36300]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:42 smithi131 ceph-mon[36065]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:42 smithi119 ceph-mon[31813]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:44.134 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.134+0000 7f8448ff9700 1 -- 172.21.15.131:0/1064316073 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f844c02de10 con 0x7f84340a2880 2024-03-28T18:35:44.134 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 1 on host 'smithi131' 2024-03-28T18:35:44.136 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 msgr2=0x7f84340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 0x7f84340a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8440009a10 tx=0x7f8440009190 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 msgr2=0x7f844c075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d980 tx=0x7f843c00dc90 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 shutdown_connections 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84340a2880 0x7f84340a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f844c103d60 0x7f844c074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f844c1021c0 0x7f844c075f30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 --2- 172.21.15.131:0/1064316073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f844c1033c0 0x7f844c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 >> 172.21.15.131:0/1064316073 conn(0x7f844c0fd7a0 msgr2=0x7f844c1066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 shutdown_connections 2024-03-28T18:35:44.137 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:35:44.136+0000 7f8451c65700 1 -- 172.21.15.131:0/1064316073 wait complete. 2024-03-28T18:35:44.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:44 smithi196 ceph-mon[36300]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:44.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:44.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:44 smithi131 ceph-mon[36065]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:44 smithi119 ceph-mon[31813]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:45.109 DEBUG:teuthology.orchestra.run.smithi131:osd.1> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T18:35:45.112 INFO:tasks.cephadm:Deploying osd.2 on smithi196 with /dev/vg_nvme/lv_4... 2024-03-28T18:35:45.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T18:35:45.354 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:35:45.780 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:45.780 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:45.780 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:45 smithi196 ceph-mon[36300]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:45.780 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:45 smithi196 ceph-mon[36300]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:45.780 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:45 smithi196 ceph-mon[36300]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:45.970 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:45.970 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:45.971 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:45 smithi131 ceph-mon[36065]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:45.971 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:45 smithi131 ceph-mon[36065]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:45.971 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:45 smithi131 ceph-mon[36065]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:45 smithi119 ceph-mon[31813]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:45 smithi119 ceph-mon[31813]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:46.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:45 smithi119 ceph-mon[31813]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T18:35:46.785 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:46 smithi196 ceph-mon[36300]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:35:46.785 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:46 smithi196 ceph-mon[36300]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:46.785 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:46 smithi196 ceph-mon[36300]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:35:46.785 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:46 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:46.785 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:46 smithi196 ceph-mon[36300]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:46 smithi131 ceph-mon[36065]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:35:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:46 smithi131 ceph-mon[36065]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:46 smithi131 ceph-mon[36065]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:35:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:46 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:46.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:46 smithi131 ceph-mon[36065]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:47.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:46 smithi119 ceph-mon[31813]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T18:35:47.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:46 smithi119 ceph-mon[31813]: from='osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:47.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:46 smithi119 ceph-mon[31813]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T18:35:47.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:46 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:47.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:46 smithi119 ceph-mon[31813]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:35:47.320 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T18:35:47.320 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T18:35:47.321 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-28T18:35:47.321 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-28T18:35:47.321 INFO:teuthology.orchestra.run.smithi196.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111514 s, 940 MB/s 2024-03-28T18:35:47.321 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-28T18:35:47.467 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1[40686]: 2024-03-28T18:35:47.194+0000 7faec7843700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T18:35:47.467 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1[40686]: 2024-03-28T18:35:47.197+0000 7faebf62e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:35:47.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:35:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:35:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: from='osd.1 ' entity='osd.1' 2024-03-28T18:35:47.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:47.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: from='osd.1 ' entity='osd.1' 2024-03-28T18:35:48.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: from='osd.1 ' entity='osd.1' 2024-03-28T18:35:48.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.227 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi131 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] boot 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:48.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:48.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:48.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:35:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:35:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi131 2024-03-28T18:35:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] boot 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:48.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi131 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: osd.1 [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] boot 2024-03-28T18:35:49.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:35:49.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.761+0000 7ffad6d8a700 1 -- 172.21.15.196:0/989866457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01021e0 msgr2=0x7ffad0102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.761+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/989866457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01021e0 0x7ffad0102600 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffac4009990 tx=0x7ffac4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- 172.21.15.196:0/989866457 shutdown_connections 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/989866457 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 0x7ffad010a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/989866457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01021e0 0x7ffad0102600 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/989866457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01033e0 0x7ffad0103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.763 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- 172.21.15.196:0/989866457 >> 172.21.15.196:0/989866457 conn(0x7ffad00fd7a0 msgr2=0x7ffad00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- 172.21.15.196:0/989866457 shutdown_connections 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- 172.21.15.196:0/989866457 wait complete. 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 Processor -- start 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- start start 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01021e0 0x7ffad0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 0x7ffad019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad010dea0 con 0x7ffad01021e0 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad010dd20 con 0x7ffad0103d80 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.762+0000 7ffad6d8a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad010e020 con 0x7ffad01033e0 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:41588/0 (socket says 172.21.15.196:41588) 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 -- 172.21.15.196:0/845065736 learned_addr learned my addr 172.21.15.196:0/845065736 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffad5327700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 0x7ffad019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 -- 172.21.15.196:0/845065736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 msgr2=0x7ffad019fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 0x7ffad019fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 -- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01021e0 msgr2=0x7ffad0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffad4b26700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01021e0 0x7ffad0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01021e0 0x7ffad0198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 -- 172.21.15.196:0/845065736 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad01a01a0 con 0x7ffad01033e0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacffff700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00b560 tx=0x7ffabc00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc015020 con 0x7ffad01033e0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffabc011d10 con 0x7ffad01033e0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad01a0490 con 0x7ffad01033e0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc010e50 con 0x7ffad01033e0 2024-03-28T18:35:49.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.763+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad01a0aa0 con 0x7ffad01033e0 2024-03-28T18:35:49.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.765+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad0068e50 con 0x7ffad01033e0 2024-03-28T18:35:49.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.765+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7ffabc018020 con 0x7ffad01033e0 2024-03-28T18:35:49.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.766+0000 7ffacdffb700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 0x7ffac00a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:35:49.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.766+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7ffabc098bb0 con 0x7ffad01033e0 2024-03-28T18:35:49.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.766+0000 7ffad4b26700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 0x7ffac00a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:35:49.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.767+0000 7ffad4b26700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 0x7ffac00a4ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffac4006010 tx=0x7ffac4005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:35:49.770 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.771+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffabc01f020 con 0x7ffad01033e0 2024-03-28T18:35:49.975 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:35:49.976+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ffad00625d0 con 0x7ffac00a2a20 2024-03-28T18:35:50.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:49 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 17080M 2024-03-28T18:35:50.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:49 smithi119 ceph-mon[31813]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:50.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:49 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 17080M 2024-03-28T18:35:50.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:49 smithi196 ceph-mon[36300]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:50.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:49 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 17080M 2024-03-28T18:35:50.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:49 smithi131 ceph-mon[36065]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:50.842 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:50 smithi196 ceph-mon[36300]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:35:50.842 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:50 smithi196 ceph-mon[36300]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:50.842 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:50.842 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:50.842 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:50 smithi119 ceph-mon[31813]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:35:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:50 smithi119 ceph-mon[31813]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:51.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:50 smithi131 ceph-mon[36065]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T18:35:51.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:50 smithi131 ceph-mon[36065]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:35:51.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:35:51.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:35:51.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:35:51.856 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:51 smithi196 ceph-mon[36300]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:51 smithi119 ceph-mon[31813]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:52.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:51 smithi131 ceph-mon[36065]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/1883811403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:52 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:52 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]': finished 2024-03-28T18:35:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:52 smithi119 ceph-mon[31813]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:35:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:35:53.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/1883811403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:52 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:52 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]': finished 2024-03-28T18:35:53.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:52 smithi196 ceph-mon[36300]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:35:53.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:35:53.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/1883811403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:52 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]: dispatch 2024-03-28T18:35:53.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:52 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c080cdf-65ee-41c9-be0f-18988515b0f0"}]': finished 2024-03-28T18:35:53.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:52 smithi131 ceph-mon[36065]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T18:35:53.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:35:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:53 smithi119 ceph-mon[31813]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/2072303032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:54.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:53 smithi196 ceph-mon[36300]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:54.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/2072303032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:54.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:53 smithi131 ceph-mon[36065]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:54.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/2072303032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:35:56.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:55 smithi196 ceph-mon[36300]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:56.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:55 smithi131 ceph-mon[36065]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:55 smithi119 ceph-mon[31813]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:58.050 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:57 smithi196 ceph-mon[36300]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:58.184 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:57 smithi131 ceph-mon[36065]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:35:58.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:57 smithi119 ceph-mon[31813]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:00.180 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:59 smithi196 ceph-mon[36300]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:00.181 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:36:00.181 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:35:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:00.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:59 smithi131 ceph-mon[36065]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:00.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:36:00.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:35:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:00.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:59 smithi119 ceph-mon[31813]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:00.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T18:36:00.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:35:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:01.154 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:00 smithi196 ceph-mon[36300]: Deploying daemon osd.2 on smithi196 2024-03-28T18:36:01.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:00 smithi131 ceph-mon[36065]: Deploying daemon osd.2 on smithi196 2024-03-28T18:36:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:00 smithi119 ceph-mon[31813]: Deploying daemon osd.2 on smithi196 2024-03-28T18:36:02.043 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:01 smithi196 ceph-mon[36300]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:02.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:01 smithi131 ceph-mon[36065]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:01 smithi119 ceph-mon[31813]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:04.042 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:03 smithi196 ceph-mon[36300]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:03 smithi119 ceph-mon[31813]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:03 smithi131 ceph-mon[36065]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:05.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:05.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:05.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:05 smithi196 ceph-mon[36300]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:05.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:05.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:05.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:05.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:05 smithi131 ceph-mon[36065]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:06.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:06.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:06.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:06.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:05 smithi119 ceph-mon[31813]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:08.157 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 2 on host 'smithi196' 2024-03-28T18:36:08.157 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.156+0000 7ffacdffb700 1 -- 172.21.15.196:0/845065736 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffad00625d0 con 0x7ffac00a2a20 2024-03-28T18:36:08.157 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:08.158 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.158 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.158 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:08.158 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:08.158 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 msgr2=0x7ffac00a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 0x7ffac00a4ee0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffac4006010 tx=0x7ffac4005e80 comp rx=0 tx=0).stop 2024-03-28T18:36:08.159 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 msgr2=0x7ffad0198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00b560 tx=0x7ffabc00b870 comp rx=0 tx=0).stop 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 shutdown_connections 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffac00a2a20 0x7ffac00a4ee0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffad0103d80 0x7ffad019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffad01021e0 0x7ffad0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 --2- 172.21.15.196:0/845065736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad01033e0 0x7ffad0198a80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 >> 172.21.15.196:0/845065736 conn(0x7ffad00fd7a0 msgr2=0x7ffad0106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 shutdown_connections 2024-03-28T18:36:08.160 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:36:08.159+0000 7ffad6d8a700 1 -- 172.21.15.196:0/845065736 wait complete. 2024-03-28T18:36:08.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:08.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:08.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:08.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:08.774 DEBUG:teuthology.orchestra.run.smithi196:osd.2> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T18:36:08.777 INFO:tasks.cephadm:Deploying osd.3 on smithi119 with /dev/vg_nvme/lv_3... 2024-03-28T18:36:08.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:36:09.234 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:09 smithi119 ceph-mon[31813]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:09 smithi119 ceph-mon[31813]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:09 smithi196 ceph-mon[36300]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:09.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:09 smithi196 ceph-mon[36300]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:09 smithi131 ceph-mon[36065]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:09 smithi131 ceph-mon[36065]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:10.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-mon[36300]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:10.185 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2[41453]: 2024-03-28T18:36:10.140+0000 7f770acf6700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T18:36:10.186 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2[41453]: 2024-03-28T18:36:10.146+0000 7f7706ae9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:36:10.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:36:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:36:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:10 smithi119 ceph-mon[31813]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: from='osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:10.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:10 smithi131 ceph-mon[36065]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:36:11.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:11 smithi196 ceph-mon[36300]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:36:11.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:11 smithi196 ceph-mon[36300]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:36:11.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:11 smithi131 ceph-mon[36065]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:36:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:11 smithi131 ceph-mon[36065]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:36:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:11 smithi119 ceph-mon[31813]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:36:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:11 smithi119 ceph-mon[31813]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T18:36:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:11.985 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:36:11.985 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:36:11.985 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-28T18:36:11.986 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-28T18:36:11.986 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0373163 s, 281 MB/s 2024-03-28T18:36:11.986 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] boot 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:12.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:12.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:36:12.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] boot 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:12.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:36:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: osd.2 [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] boot 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:12.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-03-28T18:36:13.238 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:36:13.645 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:13 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi196 2024-03-28T18:36:13.645 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:13 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-28T18:36:13.645 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:13 smithi119 ceph-mon[31813]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:36:13.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:13.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:13 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi196 2024-03-28T18:36:13.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:13 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-28T18:36:13.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:13 smithi196 ceph-mon[36300]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:36:13.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:13 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:13 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi196 2024-03-28T18:36:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:13 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-28T18:36:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:13 smithi131 ceph-mon[36065]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T18:36:13.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:13 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:14 smithi119 ceph-mon[31813]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:14 smithi119 ceph-mon[31813]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:36:14.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:14 smithi196 ceph-mon[36300]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:14 smithi196 ceph-mon[36300]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:36:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:14 smithi131 ceph-mon[36065]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:14 smithi131 ceph-mon[36065]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T18:36:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:36:15.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 -- 172.21.15.119:0/3837998849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 msgr2=0x7f5b1c10a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 --2- 172.21.15.119:0/3837998849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c10a630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009f90 tx=0x7f5b1400b330 comp rx=0 tx=0).stop 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 -- 172.21.15.119:0/3837998849 shutdown_connections 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 --2- 172.21.15.119:0/3837998849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c10a630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 --2- 172.21.15.119:0/3837998849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 --2- 172.21.15.119:0/3837998849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 0x7f5b1c103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 -- 172.21.15.119:0/3837998849 >> 172.21.15.119:0/3837998849 conn(0x7f5b1c0fd740 msgr2=0x7f5b1c0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 -- 172.21.15.119:0/3837998849 shutdown_connections 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.303+0000 7f5b23800700 1 -- 172.21.15.119:0/3837998849 wait complete. 2024-03-28T18:36:15.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 Processor -- start 2024-03-28T18:36:15.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 -- start start 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 0x7f5b1c1084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c10ddf0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c10dc70 con 0x7f5b1c1033e0 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.305+0000 7f5b23800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c10df70 con 0x7f5b1c103d80 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41678/0 (socket says 172.21.15.119:41678) 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 -- 172.21.15.119:0/1226542027 learned_addr learned my addr 172.21.15.119:0/1226542027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b1be00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 0x7f5b1c1084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:15.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b21800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 -- 172.21.15.119:0/1226542027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 msgr2=0x7f5b1c104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 -- 172.21.15.119:0/1226542027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 msgr2=0x7f5b1c1084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 0x7f5b1c1084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.306+0000 7f5b20e00700 1 -- 172.21.15.119:0/1226542027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c1053c0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b20e00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b1000e9f0 tx=0x7f5b1000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10004220 con 0x7f5b1c1021e0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b100043c0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1a69e0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10003ce0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.307+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c1a6e30 con 0x7f5b1c1021e0 2024-03-28T18:36:15.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.308+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c1a7060 con 0x7f5b1c1021e0 2024-03-28T18:36:15.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.312+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f5b1000c7c0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.313+0000 7f5b19600700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 0x7f5b040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:15.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.313+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f5b100cd880 con 0x7f5b1c1021e0 2024-03-28T18:36:15.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.313+0000 7f5b1be00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 0x7f5b040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:15.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.314+0000 7f5b1be00700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 0x7f5b040a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009c60 tx=0x7f5b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:15.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.316+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b1009bd30 con 0x7f5b1c1021e0 2024-03-28T18:36:15.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.538+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b1c1a7230 con 0x7f5b040a2800 2024-03-28T18:36:15.539 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51487]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:15.539 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51487]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:15.539 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51487]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:15.540 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51487]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:15.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:15.587+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5b10097ca0 con 0x7f5b1c1021e0 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:15 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:15.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:36:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:15 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:36:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T18:36:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:15.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:15.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:15 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:16.318 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51576]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:16.318 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51576]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:16.318 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51576]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:16.318 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:36:15 smithi119 sudo[51576]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:16.427 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44571]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:16.427 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44571]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:16.427 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44571]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:16.427 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44571]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:16.694 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:16.695 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:16.695 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44574]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:16.695 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:36:16 smithi131 sudo[44574]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:16.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:16 smithi131 ceph-mon[36065]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:16.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:16 smithi131 ceph-mon[36065]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:36:16.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:16 smithi131 ceph-mon[36065]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T18:36:16.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:16.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:16.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:16 smithi119 ceph-mon[31813]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:16.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:16 smithi119 ceph-mon[31813]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:36:16.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:16 smithi119 ceph-mon[31813]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T18:36:16.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:16.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:16.934 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:16.935 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:16.935 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45544]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:16.935 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45544]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:16 smithi196 ceph-mon[36300]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:16 smithi196 ceph-mon[36300]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T18:36:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:16 smithi196 ceph-mon[36300]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T18:36:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:17.434 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45547]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:17.434 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45547]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:17.435 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45547]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:17.435 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:36:16 smithi196 sudo[45547]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:17 smithi119 sudo[51654]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:17 smithi119 sudo[51654]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:17 smithi119 sudo[51654]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:17 smithi119 sudo[51654]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:18.414 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:18.415 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:18.415 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45550]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:18.415 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:18.415 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45550]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:18.415 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45550]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:18.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:18.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 sudo[51684]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 sudo[51684]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 sudo[51684]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 sudo[51684]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:18 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45553]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:18.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45553]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:18.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:18 smithi196 sudo[45553]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:19.158 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 sudo[44577]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T18:36:19.159 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 sudo[44577]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:19.159 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 sudo[44577]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:19.159 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:18 smithi131 sudo[44577]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:19.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 sudo[44580]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T18:36:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 sudo[44580]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T18:36:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 sudo[44580]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 sudo[44580]: pam_unix(sudo:session): session closed for user root 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:19.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:36:19.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:36:19.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:36:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3902581249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]': finished 2024-03-28T18:36:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:36:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3902581249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]': finished 2024-03-28T18:36:20.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:36:20.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T18:36:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T18:36:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3902581249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]: dispatch 2024-03-28T18:36:20.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e6e92d-b038-4f17-8cb8-9a0c6db99e93"}]': finished 2024-03-28T18:36:20.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T18:36:20.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:21 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3862515238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:21 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3862515238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:21.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:21 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3862515238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:22 smithi196 ceph-mon[36300]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:22 smithi131 ceph-mon[36065]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:22 smithi119 ceph-mon[31813]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:24.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:24 smithi119 ceph-mon[31813]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:24 smithi196 ceph-mon[36300]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:24 smithi131 ceph-mon[36065]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:26.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:26 smithi119 ceph-mon[31813]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T18:36:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:26 smithi131 ceph-mon[36065]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T18:36:26.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:26 smithi196 ceph-mon[36300]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-28T18:36:27.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:27 smithi119 ceph-mon[31813]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T18:36:27.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:27 smithi196 ceph-mon[36300]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T18:36:27.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:27 smithi131 ceph-mon[36065]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-28T18:36:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:36:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:28 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:36:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:28 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:28 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T18:36:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:28 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:29.666 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:29 smithi119 ceph-mon[31813]: Deploying daemon osd.3 on smithi119 2024-03-28T18:36:29.666 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:29 smithi119 ceph-mon[31813]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T18:36:29.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:29 smithi131 ceph-mon[36065]: Deploying daemon osd.3 on smithi119 2024-03-28T18:36:29.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:29 smithi131 ceph-mon[36065]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T18:36:29.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:29 smithi196 ceph-mon[36300]: Deploying daemon osd.3 on smithi119 2024-03-28T18:36:29.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:29 smithi196 ceph-mon[36300]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-28T18:36:32.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:31 smithi196 ceph-mon[36300]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T18:36:32.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:31 smithi131 ceph-mon[36065]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T18:36:32.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:31 smithi119 ceph-mon[31813]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 895 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-28T18:36:34.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:33 smithi119 ceph-mon[31813]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:34.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:33 smithi196 ceph-mon[36300]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:34.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:33 smithi131 ceph-mon[36065]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:36.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:35 smithi131 ceph-mon[36065]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:36.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:35 smithi119 ceph-mon[31813]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:36.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:35 smithi196 ceph-mon[36300]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-28T18:36:37.758 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.758 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.758 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:37 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:37.758 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:37 smithi119 ceph-mon[31813]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:37.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:37 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:37.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:37 smithi196 ceph-mon[36300]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:37 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:37 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:37 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:37 smithi131 ceph-mon[36065]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:40.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:39 smithi119 ceph-mon[31813]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:40.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:39 smithi196 ceph-mon[36300]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:40.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:39 smithi131 ceph-mon[36065]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:41.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.029+0000 7f5b19600700 1 -- 172.21.15.119:0/1226542027 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5b1c1a7230 con 0x7f5b040a2800 2024-03-28T18:36:41.030 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 3 on host 'smithi119' 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 msgr2=0x7f5b040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 0x7f5b040a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009c60 tx=0x7f5b08009260 comp rx=0 tx=0).stop 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 msgr2=0x7f5b1c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b1000e9f0 tx=0x7f5b1000ed00 comp rx=0 tx=0).stop 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 shutdown_connections 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5b040a2800 0x7f5b040a4cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b1c103d80 0x7f5b1c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:41.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b1c1021e0 0x7f5b1c107f70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:41.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 --2- 172.21.15.119:0/1226542027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b1c1033e0 0x7f5b1c1084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:41.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 >> 172.21.15.119:0/1226542027 conn(0x7f5b1c0fd740 msgr2=0x7f5b1c0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:41.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 shutdown_connections 2024-03-28T18:36:41.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:36:41.033+0000 7f5b23800700 1 -- 172.21.15.119:0/1226542027 wait complete. 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:41.844 DEBUG:teuthology.orchestra.run.smithi119:osd.3> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T18:36:41.850 INFO:tasks.cephadm:Deploying osd.4 on smithi131 with /dev/vg_nvme/lv_3... 2024-03-28T18:36:41.850 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:36:42.095 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:36:42.552 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:42 smithi131 ceph-mon[36065]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:36:42.553 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:42 smithi131 ceph-mon[36065]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:36:42.553 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:42 smithi131 ceph-mon[36065]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:42.553 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:42.553 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:42 smithi131 ceph-mon[36065]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-mon[31813]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:36:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-mon[31813]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:36:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-mon[31813]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-mon[31813]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:42.819 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3[53750]: 2024-03-28T18:36:42.474+0000 7f05ab200700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T18:36:42.819 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:36:42 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3[53750]: 2024-03-28T18:36:42.490+0000 7f05a6000700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:36:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:42 smithi196 ceph-mon[36300]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T18:36:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:42 smithi196 ceph-mon[36300]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T18:36:42.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:42 smithi196 ceph-mon[36300]: from='osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:42.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:42.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:42 smithi196 ceph-mon[36300]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:36:43.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:43 smithi119 ceph-mon[31813]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:36:43.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:43 smithi119 ceph-mon[31813]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T18:36:43.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:43 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:43 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:43 smithi119 ceph-mon[31813]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:43.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:43 smithi196 ceph-mon[36300]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:36:43.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:43 smithi196 ceph-mon[36300]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T18:36:43.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:43 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:43 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:43 smithi196 ceph-mon[36300]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:43.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:43 smithi131 ceph-mon[36065]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:36:43.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:43 smithi131 ceph-mon[36065]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T18:36:43.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:43 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:43 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:43.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:43 smithi131 ceph-mon[36065]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113433 s, 924 MB/s 2024-03-28T18:36:44.043 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] boot 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:44.638 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:44 smithi131 ceph-mon[36065]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:36:44.675 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_3 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] boot 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:44 smithi119 ceph-mon[31813]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:36:44.914 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:36:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:36:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:36:44.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: osd.3 [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] boot 2024-03-28T18:36:44.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T18:36:44.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:36:44.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:44 smithi196 ceph-mon[36300]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T18:36:46.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:46 smithi131 ceph-mon[36065]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:46.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:46 smithi131 ceph-mon[36065]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1506563569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec1022d0 msgr2=0x7fd5ec102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1506563569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec102730 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0099c0 tx=0x7fd5dc009cd0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1506563569 shutdown_connections 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1506563569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5ec102c70 0x7fd5ec109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1506563569 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1506563569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec102730 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.398+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1506563569 >> 172.21.15.131:0/1506563569 conn(0x7fd5ec0fc650 msgr2=0x7fd5ec0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1506563569 shutdown_connections 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1506563569 wait complete. 2024-03-28T18:36:46.405 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 Processor -- start 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- start start 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10ce00 con 0x7fd5ec1022d0 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10cc80 con 0x7fd5ec102c70 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.399+0000 7fd5f35f2700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec10cf80 con 0x7fd5ec1010d0 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:54342/0 (socket says 172.21.15.131:54342) 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 -- 172.21.15.131:0/1395618762 learned_addr learned my addr 172.21.15.131:0/1395618762 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f138e700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 -- 172.21.15.131:0/1395618762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 msgr2=0x7fd5ec06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.406 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec06b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 -- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5ec1022d0 msgr2=0x7fd5ec19d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 -- 172.21.15.131:0/1395618762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec1a0210 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f138e700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f1b8f700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400b4a0 tx=0x7fd5e400b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4015030 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec1a0500 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5e4011630 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40062c0 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.400+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec1a0ac0 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.401+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec068e50 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.402+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd5e4018070 con 0x7fd5ec102c70 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.403+0000 7fd5e27fc700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 0x7fd5d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:36:46.407 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.403+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7fd5e40cbba0 con 0x7fd5ec102c70 2024-03-28T18:36:46.408 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.403+0000 7fd5f138e700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 0x7fd5d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:36:46.408 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.404+0000 7fd5f138e700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 0x7fd5d80a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec102130 tx=0x7fd5e8009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:36:46.408 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.408+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd5e409b030 con 0x7fd5ec102c70 2024-03-28T18:36:46.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:46 smithi196 ceph-mon[36300]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:46.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:46 smithi196 ceph-mon[36300]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:36:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:46 smithi119 ceph-mon[31813]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:46 smithi119 ceph-mon[31813]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T18:36:46.620 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:36:46.620+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5ec0625d0 con 0x7fd5d80a2890 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:47.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:47.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:47.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:47.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:47.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:47.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:36:47.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:48.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:48 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi119 2024-03-28T18:36:48.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:48 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 8545M 2024-03-28T18:36:48.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:48 smithi131 ceph-mon[36065]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:48.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:48 smithi131 ceph-mon[36065]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:48.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:48 smithi131 ceph-mon[36065]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T18:36:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:48 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi119 2024-03-28T18:36:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:48 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 8545M 2024-03-28T18:36:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:48 smithi119 ceph-mon[31813]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:48 smithi119 ceph-mon[31813]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:48 smithi119 ceph-mon[31813]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T18:36:48.635 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:48 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi119 2024-03-28T18:36:48.635 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:48 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 8545M 2024-03-28T18:36:48.635 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:48 smithi196 ceph-mon[36300]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:36:48.635 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:48 smithi196 ceph-mon[36300]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:36:48.635 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:48 smithi196 ceph-mon[36300]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T18:36:49.538 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:49 smithi131 ceph-mon[36065]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T18:36:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:49 smithi119 ceph-mon[31813]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T18:36:49.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:49 smithi196 ceph-mon[36300]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-28T18:36:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/3090925652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]': finished 2024-03-28T18:36:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:36:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:36:50.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:50 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/1776232488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/3090925652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]': finished 2024-03-28T18:36:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:36:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:36:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:50 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/1776232488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/3090925652' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]: dispatch 2024-03-28T18:36:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce922e09-9a15-418f-a01b-b70921dfbe71"}]': finished 2024-03-28T18:36:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T18:36:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:36:50.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:50 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/1776232488' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:36:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:51 smithi119 ceph-mon[31813]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T18:36:51.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:51 smithi196 ceph-mon[36300]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T18:36:51.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:51 smithi131 ceph-mon[36065]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-28T18:36:54.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:53 smithi131 ceph-mon[36065]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-28T18:36:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:53 smithi119 ceph-mon[31813]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-28T18:36:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:53 smithi196 ceph-mon[36300]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-28T18:36:56.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:55 smithi131 ceph-mon[36065]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:36:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:55 smithi119 ceph-mon[31813]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:36:56.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:55 smithi196 ceph-mon[36300]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:36:57.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:36:57.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:57.360 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:36:57.360 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T18:36:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:36:58.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:57 smithi119 ceph-mon[31813]: Deploying daemon osd.4 on smithi131 2024-03-28T18:36:58.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:57 smithi119 ceph-mon[31813]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:36:58.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:57 smithi131 ceph-mon[36065]: Deploying daemon osd.4 on smithi131 2024-03-28T18:36:58.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:57 smithi131 ceph-mon[36065]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:36:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:57 smithi196 ceph-mon[36300]: Deploying daemon osd.4 on smithi131 2024-03-28T18:36:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:57 smithi196 ceph-mon[36300]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:37:00.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:36:59 smithi131 ceph-mon[36065]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:00.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:36:59 smithi119 ceph-mon[31813]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:36:59 smithi196 ceph-mon[36300]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:02.255 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:01 smithi131 ceph-mon[36065]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:01 smithi119 ceph-mon[31813]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:01 smithi196 ceph-mon[36300]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:03.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:03.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:03 smithi131 ceph-mon[36065]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:03 smithi196 ceph-mon[36300]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:03 smithi119 ceph-mon[31813]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:05.857 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.856+0000 7fd5e27fc700 1 -- 172.21.15.131:0/1395618762 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd5ec0625d0 con 0x7fd5d80a2890 2024-03-28T18:37:05.857 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 4 on host 'smithi131' 2024-03-28T18:37:05.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 msgr2=0x7fd5d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 0x7fd5d80a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec102130 tx=0x7fd5e8009570 comp rx=0 tx=0).stop 2024-03-28T18:37:05.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 msgr2=0x7fd5ec19f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:05.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400b4a0 tx=0x7fd5e400b860 comp rx=0 tx=0).stop 2024-03-28T18:37:05.859 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 shutdown_connections 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd5d80a2890 0x7fd5d80a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd5ec102c70 0x7fd5ec19f940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5ec1010d0 0x7fd5ec06b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 --2- 172.21.15.131:0/1395618762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 >> 172.21.15.131:0/1395618762 conn(0x7fd5ec0fc650 msgr2=0x7fd5ec105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 shutdown_connections 2024-03-28T18:37:05.860 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:37:05.858+0000 7fd5f35f2700 1 -- 172.21.15.131:0/1395618762 wait complete. 2024-03-28T18:37:06.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:05 smithi131 ceph-mon[36065]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:06.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:05 smithi119 ceph-mon[31813]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.408 DEBUG:teuthology.orchestra.run.smithi131:osd.4> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T18:37:06.410 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-03-28T18:37:06.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T18:37:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:05 smithi196 ceph-mon[36300]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:06.659 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:37:07.032 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:06 smithi196 ceph-mon[36300]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.032 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:06 smithi196 ceph-mon[36300]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.032 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:06 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:07.032 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:06 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:07.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:06 smithi131 ceph-mon[36065]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:06 smithi131 ceph-mon[36065]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:06 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:07.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:06 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:06 smithi119 ceph-mon[31813]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:06 smithi119 ceph-mon[31813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T18:37:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:06 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:06 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:08.140 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:07 smithi196 ceph-mon[36300]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:08.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:37:08.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:37:08.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:08.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:07 smithi119 ceph-mon[31813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: from='osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:08.317 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-mon[36065]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:37:08.317 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:37:07.966+0000 7f5788eb8700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T18:37:08.318 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:37:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:37:07.971+0000 7f5784cab700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:37:08.691 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T18:37:08.691 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T18:37:08.692 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-28T18:37:08.692 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-28T18:37:08.692 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116209 s, 902 MB/s 2024-03-28T18:37:08.692 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-28T18:37:09.244 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:08 smithi119 ceph-mon[31813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:37:09.244 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:08 smithi119 ceph-mon[31813]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T18:37:09.244 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:08 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.244 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:08 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.312 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:08 smithi196 ceph-mon[36300]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:37:09.312 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:08 smithi196 ceph-mon[36300]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T18:37:09.312 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:08 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.312 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:08 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-03-28T18:37:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:08 smithi131 ceph-mon[36065]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:37:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:08 smithi131 ceph-mon[36065]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T18:37:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:08 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:08 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:09.616 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:37:10.040 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:37:10.040 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] boot 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:10.041 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:09 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:37:10.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:37:10.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] boot 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:10.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:09 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: osd.4 [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] boot 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:10.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:09 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi131 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 8540M 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:11.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:11.100 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- 172.21.15.196:0/2572497955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76981010c0 msgr2=0x7f76981014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- 172.21.15.196:0/2572497955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76981010c0 0x7f76981014e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f768c009990 tx=0x7f768c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- 172.21.15.196:0/2572497955 shutdown_connections 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- 172.21.15.196:0/2572497955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7698102c60 0x7f7698109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- 172.21.15.196:0/2572497955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76981010c0 0x7f76981014e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- 172.21.15.196:0/2572497955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76981022c0 0x7f7698102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- 172.21.15.196:0/2572497955 >> 172.21.15.196:0/2572497955 conn(0x7f76980fc660 msgr2=0x7f76980feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- 172.21.15.196:0/2572497955 shutdown_connections 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- 172.21.15.196:0/2572497955 wait complete. 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 Processor -- start 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- start start 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76981010c0 0x7f7698105050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76981022c0 0x7f76981036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.101 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769810cd00 con 0x7f76981022c0 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769810cb80 con 0x7f76981010c0 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.099+0000 7f769ec5a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769810ce80 con 0x7f7698102c60 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769c9f6700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76981010c0 0x7f7698105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42378/0 (socket says 172.21.15.196:42378) 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 -- 172.21.15.196:0/1813768626 learned_addr learned my addr 172.21.15.196:0/1813768626 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 -- 172.21.15.196:0/1813768626 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76981010c0 msgr2=0x7f7698105050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76981010c0 0x7f7698105050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 -- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76981022c0 msgr2=0x7f76981036a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76981022c0 0x7f76981036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 -- 172.21.15.196:0/1813768626 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698104490 con 0x7f7698102c60 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769d1f7700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7690007ee0 tx=0x7f769000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769000dbd0 con 0x7f7698102c60 2024-03-28T18:37:11.102 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76981a2650 con 0x7f7698102c60 2024-03-28T18:37:11.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f769000dd70 con 0x7f7698102c60 2024-03-28T18:37:11.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690005a60 con 0x7f7698102c60 2024-03-28T18:37:11.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.100+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76981a2b90 con 0x7f7698102c60 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.101+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7698068e50 con 0x7f7698102c60 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.102+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f769001a020 con 0x7f7698102c60 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.103+0000 7f7695ffb700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 0x7f76880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.103+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f76900cd150 con 0x7f7698102c60 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.103+0000 7f769c9f6700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 0x7f76880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:11.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.104+0000 7f769c9f6700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 0x7f76880a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f768c00b5c0 tx=0x7f768c003a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:11.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.107+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7690010070 con 0x7f7698102c60 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi131 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 8540M 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:11.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:11.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:11.311 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:11.311+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f76980fe8a0 con 0x7f76880a2890 2024-03-28T18:37:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi131 2024-03-28T18:37:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 8540M 2024-03-28T18:37:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T18:37:11.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:11.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:11.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:11.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:12.233 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:12 smithi196 ceph-mon[36300]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:12.233 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:12.233 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:12.233 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:12 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:12 smithi131 ceph-mon[36065]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:12.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:12 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:12 smithi119 ceph-mon[31813]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:12 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:13.414 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:13 smithi196 ceph-mon[36300]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:13 smithi131 ceph-mon[36065]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:13 smithi119 ceph-mon[31813]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:14.424 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:14.424 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/3325744335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.424 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.424 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]': finished 2024-03-28T18:37:14.425 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:37:14.425 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/3325744335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]': finished 2024-03-28T18:37:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:37:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/3325744335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]: dispatch 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b57956f-6e7d-4bc7-8b89-5390058b7ba8"}]': finished 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T18:37:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:15.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:15 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/3079683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:15.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:15 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/3079683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:15 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/3079683668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:16 smithi196 ceph-mon[36300]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:16 smithi131 ceph-mon[36065]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:16 smithi119 ceph-mon[31813]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:18.303 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:18 smithi196 ceph-mon[36300]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:18.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:18 smithi131 ceph-mon[36065]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:18 smithi119 ceph-mon[31813]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:20.412 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:20 smithi196 ceph-mon[36300]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:20.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:20 smithi131 ceph-mon[36065]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:20.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:20 smithi119 ceph-mon[31813]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:37:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:37:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T18:37:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:22.329 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:22 smithi196 ceph-mon[36300]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:22.329 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:22 smithi196 ceph-mon[36300]: Deploying daemon osd.5 on smithi196 2024-03-28T18:37:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:22 smithi131 ceph-mon[36065]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:22 smithi131 ceph-mon[36065]: Deploying daemon osd.5 on smithi196 2024-03-28T18:37:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:22 smithi119 ceph-mon[31813]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:22 smithi119 ceph-mon[31813]: Deploying daemon osd.5 on smithi196 2024-03-28T18:37:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:24 smithi196 ceph-mon[36300]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:24 smithi131 ceph-mon[36065]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:24 smithi119 ceph-mon[31813]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:26.280 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:26 smithi196 ceph-mon[36300]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:26 smithi131 ceph-mon[36065]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:26.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:26 smithi119 ceph-mon[31813]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:27.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:27.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:27 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:27 smithi196 ceph-mon[36300]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:27.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:27 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:27 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:27.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:27 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:27 smithi131 ceph-mon[36065]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:27.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:27.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:27 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:27.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:27 smithi119 ceph-mon[31813]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:29 smithi131 ceph-mon[36065]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:29.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:29 smithi196 ceph-mon[36300]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:30.045 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.045+0000 7f7695ffb700 1 -- 172.21.15.196:0/1813768626 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f76980fe8a0 con 0x7f76880a2890 2024-03-28T18:37:30.046 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-03-28T18:37:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 msgr2=0x7f76880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 0x7f76880a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f768c00b5c0 tx=0x7f768c003a20 comp rx=0 tx=0).stop 2024-03-28T18:37:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 msgr2=0x7f7698103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7690007ee0 tx=0x7f769000d3b0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 shutdown_connections 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f76880a2890 0x7f76880a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7698102c60 0x7f7698103be0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f76981010c0 0x7f7698105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 --2- 172.21.15.196:0/1813768626 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76981022c0 0x7f76981036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 >> 172.21.15.196:0/1813768626 conn(0x7f76980fc660 msgr2=0x7f76980fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 shutdown_connections 2024-03-28T18:37:30.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:37:30.047+0000 7f769ec5a700 1 -- 172.21.15.196:0/1813768626 wait complete. 2024-03-28T18:37:30.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:30.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:30.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:30.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:30.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:30.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:29 smithi119 ceph-mon[31813]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:30.703 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T18:37:30.705 INFO:tasks.cephadm:Deploying osd.6 on smithi119 with /dev/vg_nvme/lv_2... 2024-03-28T18:37:30.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:37:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:31 smithi119 ceph-mon[31813]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:31.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:31 smithi119 ceph-mon[31813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:31.161 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:31.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:31 smithi196 ceph-mon[36300]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:31.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:31 smithi196 ceph-mon[36300]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:37:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:31 smithi131 ceph-mon[36065]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:31.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:31 smithi131 ceph-mon[36065]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T18:37:32.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:32.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:37:32.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:37:32.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:32.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:32 smithi119 ceph-mon[31813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-mon[36300]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T18:37:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T18:37:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: from='osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:32 smithi131 ceph-mon[36065]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:37:32.871 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:37:32.473+0000 7f781ae48700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T18:37:32.871 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:37:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:37:32.480+0000 7f7812c33700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:37:33.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:33 smithi196 ceph-mon[36300]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:37:33.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:33 smithi196 ceph-mon[36300]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:37:33.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:33 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:33 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:33 smithi196 ceph-mon[36300]: from='osd.5 ' entity='osd.5' 2024-03-28T18:37:33.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:33 smithi119 ceph-mon[31813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:37:33.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:33 smithi119 ceph-mon[31813]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:37:33.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:33 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:33 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:33 smithi119 ceph-mon[31813]: from='osd.5 ' entity='osd.5' 2024-03-28T18:37:33.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:33 smithi131 ceph-mon[36065]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:37:33.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:33 smithi131 ceph-mon[36065]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T18:37:33.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:33 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:33 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:33.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:33 smithi131 ceph-mon[36065]: from='osd.5 ' entity='osd.5' 2024-03-28T18:37:34.039 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:37:34.039 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:37:34.039 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-28T18:37:34.040 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-28T18:37:34.040 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0321825 s, 326 MB/s 2024-03-28T18:37:34.040 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-28T18:37:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] boot 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:34 smithi119 ceph-mon[31813]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] boot 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:34.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:34.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:34 smithi196 ceph-mon[36300]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:37:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:37:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:37:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-28T18:37:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: osd.5 [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] boot 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:37:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:34 smithi131 ceph-mon[36065]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T18:37:34.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-03-28T18:37:35.282 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:37:35.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:35 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi196 2024-03-28T18:37:35.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:35 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-28T18:37:35.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:35 smithi119 ceph-mon[31813]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:37:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:35 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi196 2024-03-28T18:37:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:35 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-28T18:37:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:35 smithi196 ceph-mon[36300]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:37:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:35 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi196 2024-03-28T18:37:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:35 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-28T18:37:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:35 smithi131 ceph-mon[36065]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T18:37:36.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:36 smithi119 ceph-mon[31813]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:36.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:36 smithi196 ceph-mon[36300]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:36.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:36 smithi131 ceph-mon[36065]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:37.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:37 smithi119 ceph-mon[31813]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:37:37.336 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:37 smithi196 ceph-mon[36300]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:37:37.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:37 smithi131 ceph-mon[36065]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 -- 172.21.15.119:0/2004111940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534076140 msgr2=0x7ff5340745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 --2- 172.21.15.119:0/2004111940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534076140 0x7ff5340745a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff524009f90 tx=0x7ff52400b330 comp rx=0 tx=0).stop 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 -- 172.21.15.119:0/2004111940 shutdown_connections 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 --2- 172.21.15.119:0/2004111940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff53410a0c0 0x7ff53410c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 --2- 172.21.15.119:0/2004111940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534076140 0x7ff5340745a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 --2- 172.21.15.119:0/2004111940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff534074b70 0x7ff534074ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 -- 172.21.15.119:0/2004111940 >> 172.21.15.119:0/2004111940 conn(0x7ff5340fd7a0 msgr2=0x7ff5340ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 -- 172.21.15.119:0/2004111940 shutdown_connections 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.518+0000 7ff539200700 1 -- 172.21.15.119:0/2004111940 wait complete. 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.519+0000 7ff539200700 1 Processor -- start 2024-03-28T18:37:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.519+0000 7ff539200700 1 -- start start 2024-03-28T18:37:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 0x7ff53419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff53410a0c0 0x7ff5341a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53410ffb0 con 0x7ff53410a0c0 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53410fe30 con 0x7ff534074b70 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff539200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534110130 con 0x7ff534076140 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff529400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff529400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59762/0 (socket says 172.21.15.119:59762) 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff529400700 1 -- 172.21.15.119:0/3971504397 learned_addr learned my addr 172.21.15.119:0/3971504397 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff532800700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff53410a0c0 0x7ff5341a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff531e00700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 0x7ff53419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.520+0000 7ff529400700 1 -- 172.21.15.119:0/3971504397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 msgr2=0x7ff53419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff529400700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 0x7ff53419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff529400700 1 -- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff53410a0c0 msgr2=0x7ff5341a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff529400700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff53410a0c0 0x7ff5341a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff529400700 1 -- 172.21.15.119:0/3971504397 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5341a2240 con 0x7ff534076140 2024-03-28T18:37:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff531e00700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 0x7ff53419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:37:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff529400700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00ec40 tx=0x7ff51c00ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c00cce0 con 0x7ff534076140 2024-03-28T18:37:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5341a2530 con 0x7ff534076140 2024-03-28T18:37:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff51c00ce80 con 0x7ff534076140 2024-03-28T18:37:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c010690 con 0x7ff534076140 2024-03-28T18:37:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.521+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5341a2a70 con 0x7ff534076140 2024-03-28T18:37:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.522+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff534068e50 con 0x7ff534076140 2024-03-28T18:37:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.525+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff51c010830 con 0x7ff534076140 2024-03-28T18:37:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.526+0000 7ff52b400700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:37:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.526+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4533+0+0 (secure 0 0 0) 0x7ff51c0ce060 con 0x7ff534076140 2024-03-28T18:37:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.526+0000 7ff531e00700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:37:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.527+0000 7ff531e00700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 0x7ff5180a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff524007f50 tx=0x7ff5240094b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:37:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.528+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff51c09bfb0 con 0x7ff534076140 2024-03-28T18:37:37.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:37:37.756+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5340625d0 con 0x7ff5180a2890 2024-03-28T18:37:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:38 smithi196 ceph-mon[36300]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:38 smithi196 ceph-mon[36300]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:37:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:38.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:38 smithi131 ceph-mon[36065]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:38.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:38 smithi131 ceph-mon[36065]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:37:38.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:38.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:38.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:38 smithi119 ceph-mon[31813]: pgmap v156: 1 pgs: 1 remapped+peering; 577 KiB data, 207 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:38 smithi119 ceph-mon[31813]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T18:37:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:37:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:37:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:39.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:39 smithi119 ceph-mon[31813]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:39.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:39 smithi196 ceph-mon[36300]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:39.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:39 smithi131 ceph-mon[36065]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:37:40.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:40 smithi119 ceph-mon[31813]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-28T18:37:40.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:40 smithi196 ceph-mon[36300]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-28T18:37:40.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:40 smithi131 ceph-mon[36065]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-28T18:37:42.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:37:42.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3058261277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.390 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]': finished 2024-03-28T18:37:42.390 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:37:42.390 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:42 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:37:42.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:37:42.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3058261277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]': finished 2024-03-28T18:37:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:37:42.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:42 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:37:42.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:37:42.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3058261277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]: dispatch 2024-03-28T18:37:42.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b527cff-096d-4e00-bfda-8e3e77bc70e1"}]': finished 2024-03-28T18:37:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T18:37:42.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:42 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:37:43.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:43 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2260106400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:43.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:43 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2260106400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:43 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2260106400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:37:44.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:44 smithi196 ceph-mon[36300]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:37:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:44 smithi131 ceph-mon[36065]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:37:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:44 smithi119 ceph-mon[31813]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:37:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:46 smithi119 ceph-mon[31813]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:37:46.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:46 smithi196 ceph-mon[36300]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:37:46.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:46 smithi131 ceph-mon[36065]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:37:48.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:48 smithi119 ceph-mon[31813]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:37:48.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:48 smithi196 ceph-mon[36300]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:37:48.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:48 smithi131 ceph-mon[36065]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:37:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:50 smithi196 ceph-mon[36300]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:50.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:50 smithi131 ceph-mon[36065]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:50 smithi119 ceph-mon[31813]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:51.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:37:51.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:51.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:37:51.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:51.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T18:37:51.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:37:52.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:52 smithi119 ceph-mon[31813]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:52.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:52 smithi119 ceph-mon[31813]: Deploying daemon osd.6 on smithi119 2024-03-28T18:37:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:52 smithi196 ceph-mon[36300]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:52 smithi196 ceph-mon[36300]: Deploying daemon osd.6 on smithi119 2024-03-28T18:37:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:52 smithi131 ceph-mon[36065]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:52 smithi131 ceph-mon[36065]: Deploying daemon osd.6 on smithi119 2024-03-28T18:37:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:54 smithi119 ceph-mon[31813]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:54 smithi196 ceph-mon[36300]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:54.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:54 smithi131 ceph-mon[36065]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:56.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:56 smithi196 ceph-mon[36300]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:56 smithi131 ceph-mon[36065]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:56.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:56 smithi119 ceph-mon[31813]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:37:58 smithi196 ceph-mon[36300]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:37:58 smithi131 ceph-mon[36065]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:37:58.491 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:37:58 smithi119 ceph-mon[31813]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:00.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:00 smithi119 ceph-mon[31813]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:00.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.426 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:00 smithi196 ceph-mon[36300]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:00 smithi131 ceph-mon[36065]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:02 smithi196 ceph-mon[36300]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:02 smithi131 ceph-mon[36065]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:02.501 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:02 smithi119 ceph-mon[31813]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:04.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.306+0000 7ff52b400700 1 -- 172.21.15.119:0/3971504397 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5340625d0 con 0x7ff5180a2890 2024-03-28T18:38:04.307 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 6 on host 'smithi119' 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.307 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:04 smithi119 ceph-mon[31813]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:38:04.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 msgr2=0x7ff5180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 0x7ff5180a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff524007f50 tx=0x7ff5240094b0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 msgr2=0x7ff53419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00ec40 tx=0x7ff51c00ef50 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 shutdown_connections 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff5180a2890 0x7ff5180a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff53410a0c0 0x7ff5341a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff534074b70 0x7ff53419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 --2- 172.21.15.119:0/3971504397 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff534076140 0x7ff53419ab20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 >> 172.21.15.119:0/3971504397 conn(0x7ff5340fd7a0 msgr2=0x7ff534106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.310+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 shutdown_connections 2024-03-28T18:38:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:38:04.311+0000 7ff539200700 1 -- 172.21.15.119:0/3971504397 wait complete. 2024-03-28T18:38:04.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:04.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:04 smithi196 ceph-mon[36300]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:38:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:04 smithi131 ceph-mon[36065]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T18:38:05.149 DEBUG:teuthology.orchestra.run.smithi119:osd.6> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T18:38:05.155 INFO:tasks.cephadm:Deploying osd.7 on smithi131 with /dev/vg_nvme/lv_2... 2024-03-28T18:38:05.155 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:38:05.403 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:38:05.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:38:05.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:38:05.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:38:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:05 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:38:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:38:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:38:05.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:05.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:05 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:05.938 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:38:05.802+0000 7f77f9200700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T18:38:05.938 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:38:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:38:05.828+0000 7f77ee600700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:38:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:38:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:38:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:06 smithi119 ceph-mon[31813]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' 2024-03-28T18:38:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:06.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:38:06.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:38:06.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:06 smithi196 ceph-mon[36300]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' 2024-03-28T18:38:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:38:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T18:38:06.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:06.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:06 smithi131 ceph-mon[36065]: from='osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795]' entity='osd.6' 2024-03-28T18:38:07.343 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:38:07.344 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:38:07.344 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-03-28T18:38:07.344 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-03-28T18:38:07.344 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113259 s, 926 MB/s 2024-03-28T18:38:07.344 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-03-28T18:38:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:38:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:38:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] boot 2024-03-28T18:38:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:38:07.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] boot 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:38:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: osd.6 [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] boot 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T18:38:07.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:38:07.951 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_2 2024-03-28T18:38:08.187 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:38:08.428 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:08 smithi131 ceph-mon[36065]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:08.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:08 smithi196 ceph-mon[36300]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:08 smithi119 ceph-mon[31813]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-28T18:38:09.256 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:09 smithi131 ceph-mon[36065]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:38:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:09 smithi119 ceph-mon[31813]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:38:09.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:09 smithi196 ceph-mon[36300]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T18:38:09.780 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 -- 172.21.15.131:0/721921797 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d8103d80 msgr2=0x7fd9d810a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 --2- 172.21.15.131:0/721921797 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d8103d80 0x7fd9d810a630 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000b0d0 tx=0x7fd9d000b3e0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 -- 172.21.15.131:0/721921797 shutdown_connections 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 --2- 172.21.15.131:0/721921797 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d8103d80 0x7fd9d810a630 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 --2- 172.21.15.131:0/721921797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81021e0 0x7fd9d8102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 --2- 172.21.15.131:0/721921797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d81033e0 0x7fd9d8103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 -- 172.21.15.131:0/721921797 >> 172.21.15.131:0/721921797 conn(0x7fd9d80fd7a0 msgr2=0x7fd9d80ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 -- 172.21.15.131:0/721921797 shutdown_connections 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.780+0000 7fd9de52f700 1 -- 172.21.15.131:0/721921797 wait complete. 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 Processor -- start 2024-03-28T18:38:09.781 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 -- start start 2024-03-28T18:38:09.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:09.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 0x7fd9d81989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:09.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d8103d80 0x7fd9d819fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:09.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810de70 con 0x7fd9d81033e0 2024-03-28T18:38:09.783 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810dcf0 con 0x7fd9d81021e0 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9de52f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810dff0 con 0x7fd9d8103d80 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:58608/0 (socket says 172.21.15.131:58608) 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 -- 172.21.15.131:0/475355123 learned_addr learned my addr 172.21.15.131:0/475355123 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d77fe700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 0x7fd9d81989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9dcacc700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d8103d80 0x7fd9d819fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 -- 172.21.15.131:0/475355123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d8103d80 msgr2=0x7fd9d819fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d8103d80 0x7fd9d819fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 -- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 msgr2=0x7fd9d81989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.781+0000 7fd9d7fff700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 0x7fd9d81989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d7fff700 1 -- 172.21.15.131:0/475355123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d81a00d0 con 0x7fd9d81021e0 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d77fe700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 0x7fd9d81989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:38:09.784 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d7fff700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800b500 tx=0x7fd9c800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c8015020 con 0x7fd9d81021e0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d81a03c0 con 0x7fd9d81021e0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c8011d10 con 0x7fd9d81021e0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c800f2e0 con 0x7fd9d81021e0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.782+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d81a0900 con 0x7fd9d81021e0 2024-03-28T18:38:09.785 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.783+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d8068e50 con 0x7fd9d81021e0 2024-03-28T18:38:09.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.784+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd9c8018020 con 0x7fd9d81021e0 2024-03-28T18:38:09.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.785+0000 7fd9d57fa700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 0x7fd9c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:09.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.785+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fd9c80cd530 con 0x7fd9d81021e0 2024-03-28T18:38:09.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.785+0000 7fd9d77fe700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 0x7fd9c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:09.786 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.786+0000 7fd9d77fe700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 0x7fd9c00a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc005fd0 tx=0x7fd9cc005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:09.789 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.789+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9c801f020 con 0x7fd9d81021e0 2024-03-28T18:38:09.990 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:09.990+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9d80625d0 con 0x7fd9c00a2890 2024-03-28T18:38:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:10 smithi119 ceph-mon[31813]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:10 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:10.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:10 smithi131 ceph-mon[36065]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:10.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:10.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:10.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:10 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:10 smithi196 ceph-mon[36300]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:10 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:11.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:11.618 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:11.619 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:12 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi119 2024-03-28T18:38:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:12 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 5697M 2024-03-28T18:38:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:12 smithi119 ceph-mon[31813]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:12 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi119 2024-03-28T18:38:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:12 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 5697M 2024-03-28T18:38:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:12 smithi196 ceph-mon[36300]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:12 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi119 2024-03-28T18:38:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:12 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 5697M 2024-03-28T18:38:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:12 smithi131 ceph-mon[36065]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:13 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/200187660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:13 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:13 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]': finished 2024-03-28T18:38:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:13 smithi131 ceph-mon[36065]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:38:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:13 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:13 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/200187660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:13 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:13 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]': finished 2024-03-28T18:38:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:13 smithi119 ceph-mon[31813]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:38:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:13 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:13 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/200187660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:13 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]: dispatch 2024-03-28T18:38:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:13 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "044ba726-17cb-4d14-9040-5efd8e892821"}]': finished 2024-03-28T18:38:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:13 smithi196 ceph-mon[36300]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T18:38:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:13 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:14 smithi119 ceph-mon[31813]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:14 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/1507482829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:14 smithi196 ceph-mon[36300]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:14 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/1507482829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:14 smithi131 ceph-mon[36065]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:14 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/1507482829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:15 smithi119 ceph-mon[31813]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:15.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:15 smithi196 ceph-mon[36300]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:15.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:15 smithi131 ceph-mon[36065]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:18.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:17 smithi131 ceph-mon[36065]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:17 smithi119 ceph-mon[31813]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:17 smithi196 ceph-mon[36300]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:20.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:19 smithi131 ceph-mon[36065]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:19 smithi119 ceph-mon[31813]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:19 smithi196 ceph-mon[36300]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:21.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:38:21.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:21.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:20 smithi131 ceph-mon[36065]: Deploying daemon osd.7 on smithi131 2024-03-28T18:38:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:38:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:20 smithi119 ceph-mon[31813]: Deploying daemon osd.7 on smithi131 2024-03-28T18:38:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T18:38:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:20 smithi196 ceph-mon[36300]: Deploying daemon osd.7 on smithi131 2024-03-28T18:38:21.985 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:21 smithi131 ceph-mon[36065]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:21 smithi119 ceph-mon[31813]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:21 smithi196 ceph-mon[36300]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:24.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:23 smithi119 ceph-mon[31813]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:23 smithi196 ceph-mon[36300]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:24 smithi131 ceph-mon[36065]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:25 smithi119 ceph-mon[31813]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:25 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:25 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:25 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:25 smithi196 ceph-mon[36300]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:25 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:25 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:25 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:26.441 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:25 smithi131 ceph-mon[36065]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:26.441 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:25 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:26.441 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:25 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:26.441 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:25 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:28 smithi119 ceph-mon[31813]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:28 smithi196 ceph-mon[36300]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:28 smithi131 ceph-mon[36065]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:29.325 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.325+0000 7fd9d57fa700 1 -- 172.21.15.131:0/475355123 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9d80625d0 con 0x7fd9c00a2890 2024-03-28T18:38:29.326 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 7 on host 'smithi131' 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.327+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 msgr2=0x7fd9c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 0x7fd9c00a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc005fd0 tx=0x7fd9cc005dc0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 msgr2=0x7fd9d8198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800b500 tx=0x7fd9c800b8c0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 shutdown_connections 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd9c00a2890 0x7fd9c00a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9d8103d80 0x7fd9d819fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9d81021e0 0x7fd9d8198470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 --2- 172.21.15.131:0/475355123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9d81033e0 0x7fd9d81989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:29.328 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 >> 172.21.15.131:0/475355123 conn(0x7fd9d80fd7a0 msgr2=0x7fd9d8106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:29.329 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 shutdown_connections 2024-03-28T18:38:29.329 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:38:29.328+0000 7fd9de52f700 1 -- 172.21.15.131:0/475355123 wait complete. 2024-03-28T18:38:29.899 DEBUG:teuthology.orchestra.run.smithi131:osd.7> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T18:38:29.901 INFO:tasks.cephadm:Deploying osd.8 on smithi196 with /dev/vg_nvme/lv_2... 2024-03-28T18:38:29.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T18:38:30.150 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:38:30.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:30.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:30.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T18:38:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:31 smithi196 ceph-mon[36300]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:31 smithi119 ceph-mon[31813]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: from='osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:38:31.611 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-mon[36065]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-28T18:38:31.945 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:38:31.805+0000 7fef52dae700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T18:38:31.945 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:38:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:38:31.816+0000 7fef4ab99700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:38:32.109 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T18:38:32.110 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T18:38:32.110 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-28T18:38:32.110 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-28T18:38:32.110 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111776 s, 938 MB/s 2024-03-28T18:38:32.110 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-28T18:38:32.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:32 smithi196 ceph-mon[36300]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:38:32.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:32 smithi196 ceph-mon[36300]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:38:32.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:32 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:32.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:32 smithi196 ceph-mon[36300]: from='osd.7 ' entity='osd.7' 2024-03-28T18:38:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:32 smithi131 ceph-mon[36065]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:38:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:32 smithi131 ceph-mon[36065]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:38:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:32 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:32.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:32 smithi131 ceph-mon[36065]: from='osd.7 ' entity='osd.7' 2024-03-28T18:38:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:32 smithi119 ceph-mon[31813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:38:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:32 smithi119 ceph-mon[31813]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T18:38:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:32 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:32 smithi119 ceph-mon[31813]: from='osd.7 ' entity='osd.7' 2024-03-28T18:38:32.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-03-28T18:38:33.084 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] boot 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:33.432 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:33 smithi131 ceph-mon[36065]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:38:33.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:38:33.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:38:33.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] boot 2024-03-28T18:38:33.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:38:33.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:33.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:33.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:33 smithi196 ceph-mon[36300]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:38:33.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: osd.7 [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] boot 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:33 smithi119 ceph-mon[31813]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.745+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/3845164538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 msgr2=0x7ff6e01006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.745+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/3845164538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e01006e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009990 tx=0x7ff6d0009ca0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.745+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/3845164538 shutdown_connections 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/3845164538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e0100c20 0x7ff6e010a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/3845164538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 0x7ff6e0103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/3845164538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e01006e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/3845164538 >> 172.21.15.196:0/3845164538 conn(0x7ff6e00fbde0 msgr2=0x7ff6e00fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:34.748 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/3845164538 shutdown_connections 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/3845164538 wait complete. 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 Processor -- start 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- start start 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e0100c20 0x7ff6e0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 0x7ff6e019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e010dee0 con 0x7ff6e0100c20 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e010dd60 con 0x7ff6e01037e0 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e7fa4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e010e060 con 0x7ff6e0100260 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e5d40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e5d40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:56840/0 (socket says 172.21.15.196:56840) 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.746+0000 7ff6e5d40700 1 -- 172.21.15.196:0/1253193099 learned_addr learned my addr 172.21.15.196:0/1253193099 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e6541700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 0x7ff6e019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:34.749 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 -- 172.21.15.196:0/1253193099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 msgr2=0x7ff6e019fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 0x7ff6e019fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 -- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e0100c20 msgr2=0x7ff6e0198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e0100c20 0x7ff6e0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 -- 172.21.15.196:0/1253193099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e01a01f0 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e5d40700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00b530 tx=0x7ff6dc00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc015020 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc011d10 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e01a04e0 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00f2e0 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.747+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e01a0af0 con 0x7ff6e0100260 2024-03-28T18:38:34.750 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.748+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e0068e50 con 0x7ff6e0100260 2024-03-28T18:38:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.749+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff6dc018020 con 0x7ff6e0100260 2024-03-28T18:38:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.750+0000 7ff6d6ffd700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 0x7ff6cc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:38:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.750+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5426+0+0 (secure 0 0 0) 0x7ff6dc0cda80 con 0x7ff6e0100260 2024-03-28T18:38:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.750+0000 7ff6e553f700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 0x7ff6cc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:38:34.751 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.751+0000 7ff6e553f700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 0x7ff6cc0a4dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098e0 tx=0x7ff6d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:38:34.753 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.754+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff6dc01f020 con 0x7ff6e0100260 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi131 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 5693M 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:34 smithi119 ceph-mon[31813]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi131 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 5693M 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:34.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:34 smithi196 ceph-mon[36300]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:38:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi131 2024-03-28T18:38:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 5693M 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:34.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:34 smithi131 ceph-mon[36065]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T18:38:34.961 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:34.961+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6e00625d0 con 0x7ff6cc0a2910 2024-03-28T18:38:35.812 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:35 smithi196 ceph-mon[36300]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:35.812 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:35 smithi196 ceph-mon[36300]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:35.812 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:35.812 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:35.812 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:35 smithi119 ceph-mon[31813]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:35 smithi119 ceph-mon[31813]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:35 smithi131 ceph-mon[36065]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:35 smithi131 ceph-mon[36065]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:38:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:38:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:38:35.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:36 smithi119 ceph-mon[31813]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:38:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:36 smithi119 ceph-mon[31813]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:38:36.887 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:36 smithi196 ceph-mon[36300]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:38:36.887 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:36 smithi196 ceph-mon[36300]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:38:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:36 smithi131 ceph-mon[36065]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T18:38:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:36 smithi131 ceph-mon[36065]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T18:38:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:37 smithi119 ceph-mon[31813]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:38:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:37 smithi119 ceph-mon[31813]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:37.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:37 smithi196 ceph-mon[36300]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:38:37.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:37 smithi196 ceph-mon[36300]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:37 smithi131 ceph-mon[36065]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T18:38:37.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:37 smithi131 ceph-mon[36065]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/606245200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]': finished 2024-03-28T18:38:38.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:38:38.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:38.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:38 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/1356569226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/606245200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]': finished 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:38 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/1356569226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/606245200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]: dispatch 2024-03-28T18:38:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9fcc00-0b32-4fc0-9ade-3f152779462e"}]': finished 2024-03-28T18:38:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T18:38:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:38.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:38 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/1356569226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:38:39.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:39 smithi119 ceph-mon[31813]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:39.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:39 smithi196 ceph-mon[36300]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:39.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:39 smithi131 ceph-mon[36065]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:40 smithi119 ceph-mon[31813]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:38:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:40 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:38:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:40 smithi196 ceph-mon[36300]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:38:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:40 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:38:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:40 smithi131 ceph-mon[36065]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T18:38:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:40 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:38:41.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:41 smithi196 ceph-mon[36300]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:41.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:41 smithi119 ceph-mon[31813]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:41.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:41 smithi131 ceph-mon[36065]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:44.062 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:43 smithi196 ceph-mon[36300]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:44.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:43 smithi119 ceph-mon[31813]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:43 smithi131 ceph-mon[36065]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:46.033 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:45 smithi196 ceph-mon[36300]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:46.033 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:38:46.033 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:46.033 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:45 smithi196 ceph-mon[36300]: Deploying daemon osd.8 on smithi196 2024-03-28T18:38:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:45 smithi119 ceph-mon[31813]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:38:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:45 smithi119 ceph-mon[31813]: Deploying daemon osd.8 on smithi196 2024-03-28T18:38:46.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:45 smithi131 ceph-mon[36065]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:46.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T18:38:46.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:46.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:45 smithi131 ceph-mon[36065]: Deploying daemon osd.8 on smithi196 2024-03-28T18:38:48.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:47 smithi196 ceph-mon[36300]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:48.246 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:47 smithi131 ceph-mon[36065]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:48.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:47 smithi119 ceph-mon[31813]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:50.136 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:49 smithi196 ceph-mon[36300]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:50.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:49 smithi119 ceph-mon[31813]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:50.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:49 smithi131 ceph-mon[36065]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:52.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:51 smithi196 ceph-mon[36300]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:52.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:52.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:51 smithi131 ceph-mon[36065]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:52.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:51 smithi119 ceph-mon[31813]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:54.072 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:53 smithi196 ceph-mon[36300]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:54.154 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 8 on host 'smithi196' 2024-03-28T18:38:54.154 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.153+0000 7ff6d6ffd700 1 -- 172.21.15.196:0/1253193099 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff6e00625d0 con 0x7ff6cc0a2910 2024-03-28T18:38:54.155 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 msgr2=0x7ff6cc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 0x7ff6cc0a4dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098e0 tx=0x7ff6d0005fb0 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 msgr2=0x7ff6e0198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00b530 tx=0x7ff6dc00b840 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 shutdown_connections 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6cc0a2910 0x7ff6cc0a4dd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6e01037e0 0x7ff6e019fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6e0100260 0x7ff6e0198590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 --2- 172.21.15.196:0/1253193099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e0100c20 0x7ff6e0198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 >> 172.21.15.196:0/1253193099 conn(0x7ff6e00fbde0 msgr2=0x7ff6e01069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 shutdown_connections 2024-03-28T18:38:54.156 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:38:54.155+0000 7ff6e7fa4700 1 -- 172.21.15.196:0/1253193099 wait complete. 2024-03-28T18:38:54.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:53 smithi131 ceph-mon[36065]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:53 smithi119 ceph-mon[31813]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:55.029 DEBUG:teuthology.orchestra.run.smithi196:osd.8> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T18:38:55.031 INFO:tasks.cephadm:Deploying osd.9 on smithi119 with /dev/vg_nvme/lv_1... 2024-03-28T18:38:55.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:38:55.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:55 smithi196 ceph-mon[36300]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:55.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:55 smithi196 ceph-mon[36300]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:55.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:55 smithi131 ceph-mon[36065]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:55.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:55 smithi131 ceph-mon[36065]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:55.491 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:38:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:55 smithi119 ceph-mon[31813]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:55 smithi119 ceph-mon[31813]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T18:38:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:38:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:38:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:56.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:56.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:56.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:38:56.160+0000 7fa00fdb8700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T18:38:56.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:38:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:38:56.169+0000 7fa00bbab700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:38:56.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:56.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] boot 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:38:57.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:38:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:38:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] boot 2024-03-28T18:38:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:38:57.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:57.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:38:57.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T18:38:57.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:57.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: osd.8 [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] boot 2024-03-28T18:38:57.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T18:38:57.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:38:58.269 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:58 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:38:58.269 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:58 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:38:58.270 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:58 smithi196 ceph-mon[36300]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:58.270 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:58 smithi196 ceph-mon[36300]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:38:58.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:58 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:38:58.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:58 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:38:58.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:58 smithi119 ceph-mon[31813]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:58.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:58 smithi119 ceph-mon[31813]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0314521 s, 333 MB/s 2024-03-28T18:38:58.392 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-28T18:38:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:58 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:38:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:58 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:38:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:58 smithi131 ceph-mon[36065]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-28T18:38:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:58 smithi131 ceph-mon[36065]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T18:38:59.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-03-28T18:38:59.801 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi196 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:38:59 smithi196 ceph-mon[36300]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi196 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:38:59 smithi131 ceph-mon[36065]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi196 2024-03-28T18:38:59.946 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:38:59.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:38:59 smithi119 ceph-mon[31813]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T18:39:01.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:01 smithi196 ceph-mon[36300]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:39:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:01 smithi131 ceph-mon[36065]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:39:01.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:01 smithi119 ceph-mon[31813]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T18:39:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 -- 172.21.15.119:0/457313177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105610 msgr2=0x7f875c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 --2- 172.21.15.119:0/457313177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105610 0x7f875c105a70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8748009920 tx=0x7f8748009c30 comp rx=0 tx=0).stop 2024-03-28T18:39:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 -- 172.21.15.119:0/457313177 shutdown_connections 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 --2- 172.21.15.119:0/457313177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c105fb0 0x7f875c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 --2- 172.21.15.119:0/457313177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c104410 0x7f875c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 --2- 172.21.15.119:0/457313177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105610 0x7f875c105a70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 -- 172.21.15.119:0/457313177 >> 172.21.15.119:0/457313177 conn(0x7f875c0ff9b0 msgr2=0x7f875c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 -- 172.21.15.119:0/457313177 shutdown_connections 2024-03-28T18:39:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.061+0000 7f8760a00700 1 -- 172.21.15.119:0/457313177 wait complete. 2024-03-28T18:39:02.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.063+0000 7f8760a00700 1 Processor -- start 2024-03-28T18:39:02.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.063+0000 7f8760a00700 1 -- start start 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c110050 con 0x7f875c105610 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c10fed0 con 0x7f875c105fb0 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8760a00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c1101d0 con 0x7f875c104410 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8759e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8759400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f875a800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8759400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57342/0 (socket says 172.21.15.119:57342) 2024-03-28T18:39:02.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f875a800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53572/0 (socket says 172.21.15.119:53572) 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f875a800700 1 -- 172.21.15.119:0/1349156758 learned_addr learned my addr 172.21.15.119:0/1349156758 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.064+0000 7f8759e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:38072/0 (socket says 172.21.15.119:38072) 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 -- 172.21.15.119:0/1349156758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 msgr2=0x7f875c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 -- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 msgr2=0x7f875c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 -- 172.21.15.119:0/1349156758 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c1a2310 con 0x7f875c105fb0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8759400700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8759e00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f875a800700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f875400ef30 tx=0x7f875400c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540092e0 con 0x7f875c105fb0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8754004bb0 con 0x7f875c105fb0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c1a25a0 con 0x7f875c105fb0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.065+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8754004280 con 0x7f875c105fb0 2024-03-28T18:39:02.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.066+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c1a2b30 con 0x7f875c105fb0 2024-03-28T18:39:02.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.067+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c068e50 con 0x7f875c105fb0 2024-03-28T18:39:02.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.069+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f875401b030 con 0x7f875c105fb0 2024-03-28T18:39:02.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.070+0000 7f8752a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 0x7f87400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:02.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.070+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f87540ce580 con 0x7f875c105fb0 2024-03-28T18:39:02.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.070+0000 7f8759e00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 0x7f87400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:02.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.071+0000 7f8759e00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 0x7f87400a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f875c105470 tx=0x7f874c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:02.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.073+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8754022020 con 0x7f875c105fb0 2024-03-28T18:39:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:02.305+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f875c062690 con 0x7f87400a2910 2024-03-28T18:39:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:02 smithi196 ceph-mon[36300]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:02 smithi196 ceph-mon[36300]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:39:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:02 smithi131 ceph-mon[36065]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:02 smithi131 ceph-mon[36065]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:39:02.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:02 smithi119 ceph-mon[31813]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:02.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:02 smithi119 ceph-mon[31813]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T18:39:03.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:03.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:03.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:03.464 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:03.464 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:03.465 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:03.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:04.421 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:04 smithi119 ceph-mon[31813]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:04.422 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:04 smithi119 ceph-mon[31813]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-28T18:39:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:04 smithi131 ceph-mon[36065]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:04 smithi131 ceph-mon[36065]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-28T18:39:04.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:04 smithi196 ceph-mon[36300]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:04.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:04 smithi196 ceph-mon[36300]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-28T18:39:06.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:06 smithi119 ceph-mon[31813]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:39:06.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:06 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]: dispatch 2024-03-28T18:39:06.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:06 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]': finished 2024-03-28T18:39:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:06 smithi196 ceph-mon[36300]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:39:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:06 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]: dispatch 2024-03-28T18:39:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:06 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]': finished 2024-03-28T18:39:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:06 smithi131 ceph-mon[36065]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T18:39:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:06 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]: dispatch 2024-03-28T18:39:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:06 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1501202323' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db64ebe3-7820-4944-9d3d-9efdad1576bd"}]': finished 2024-03-28T18:39:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:07 smithi119 ceph-mon[31813]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:39:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:07 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1891216277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:07.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:07 smithi196 ceph-mon[36300]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:39:07.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:07.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:07 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1891216277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:07.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:07 smithi131 ceph-mon[36065]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T18:39:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:07 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1891216277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:08.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:08 smithi119 ceph-mon[31813]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:39:08.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:08 smithi196 ceph-mon[36300]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:39:08.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:08 smithi131 ceph-mon[36065]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-28T18:39:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:10 smithi119 ceph-mon[31813]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:39:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:10 smithi196 ceph-mon[36300]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:39:10.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:10 smithi131 ceph-mon[36065]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:39:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:12 smithi119 ceph-mon[31813]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:39:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:12 smithi196 ceph-mon[36300]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:39:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:12 smithi131 ceph-mon[36065]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-28T18:39:14.558 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:14 smithi119 ceph-mon[31813]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:14 smithi196 ceph-mon[36300]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:14 smithi131 ceph-mon[36065]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:15 smithi119 ceph-mon[31813]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:15.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:15 smithi196 ceph-mon[36300]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:15.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:15 smithi131 ceph-mon[36065]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:39:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:16 smithi119 ceph-mon[31813]: Deploying daemon osd.9 on smithi119 2024-03-28T18:39:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:39:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:16 smithi196 ceph-mon[36300]: Deploying daemon osd.9 on smithi119 2024-03-28T18:39:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T18:39:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:16 smithi131 ceph-mon[36065]: Deploying daemon osd.9 on smithi119 2024-03-28T18:39:17.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:17 smithi119 ceph-mon[31813]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:17 smithi196 ceph-mon[36300]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:17 smithi131 ceph-mon[36065]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:20.241 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:19 smithi119 ceph-mon[31813]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:20.397 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:19 smithi131 ceph-mon[36065]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:19 smithi196 ceph-mon[36300]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:21 smithi119 ceph-mon[31813]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:21 smithi196 ceph-mon[36300]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:21 smithi131 ceph-mon[36065]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:23 smithi119 ceph-mon[31813]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:24.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:23 smithi196 ceph-mon[36300]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:23 smithi131 ceph-mon[36065]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:26.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:25 smithi119 ceph-mon[31813]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:25 smithi196 ceph-mon[36300]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:25 smithi131 ceph-mon[36065]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:28 smithi196 ceph-mon[36300]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:28 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:28 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:28 smithi196 ceph-mon[36300]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:39:28.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:28 smithi119 ceph-mon[31813]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:28.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:28 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:28 smithi119 ceph-mon[31813]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:39:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:28 smithi131 ceph-mon[36065]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:28 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:28 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:28 smithi131 ceph-mon[36065]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T18:39:28.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.557+0000 7f8752a00700 1 -- 172.21.15.119:0/1349156758 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f875c062690 con 0x7f87400a2910 2024-03-28T18:39:28.558 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 9 on host 'smithi119' 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 msgr2=0x7f87400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 0x7f87400a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f875c105470 tx=0x7f874c009310 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 msgr2=0x7f875c1a1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f875400ef30 tx=0x7f875400c450 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 shutdown_connections 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f87400a2910 0x7f87400a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f875c105fb0 0x7f875c1a1c70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f875c104410 0x7f875c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 --2- 172.21.15.119:0/1349156758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f875c105610 0x7f875c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.560+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 >> 172.21.15.119:0/1349156758 conn(0x7f875c0ff9b0 msgr2=0x7f875c1089b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.561+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 shutdown_connections 2024-03-28T18:39:28.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:39:28.561+0000 7f8760a00700 1 -- 172.21.15.119:0/1349156758 wait complete. 2024-03-28T18:39:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:39:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:39:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:39:29.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.440 DEBUG:teuthology.orchestra.run.smithi119:osd.9> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T18:39:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:39:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:39:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:39:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.446 INFO:tasks.cephadm:Deploying osd.10 on smithi131 with /dev/vg_nvme/lv_1... 2024-03-28T18:39:29.446 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:39:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T18:39:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T18:39:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-28T18:39:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:29.693 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:39:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:30 smithi119 ceph-mon[31813]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:30 smithi119 ceph-mon[31813]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:39:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:30 smithi119 ceph-mon[31813]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:39:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:30.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:30 smithi131 ceph-mon[36065]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:30 smithi131 ceph-mon[36065]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:39:30.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:30 smithi131 ceph-mon[36065]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:39:30.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:30.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:30 smithi196 ceph-mon[36300]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-28T18:39:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:30 smithi196 ceph-mon[36300]: from='osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-28T18:39:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:30 smithi196 ceph-mon[36300]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T18:39:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:31.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:31 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:39:31.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:31 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:39:31.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:31 smithi119 ceph-mon[31813]: osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] boot 2024-03-28T18:39:31.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:31 smithi119 ceph-mon[31813]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:39:31.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:31 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:39:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:31 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:39:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:31 smithi196 ceph-mon[36300]: osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] boot 2024-03-28T18:39:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:31 smithi196 ceph-mon[36300]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:39:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:31.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:31 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:39:31.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:31 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:39:31.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:31 smithi131 ceph-mon[36065]: osd.9 [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] boot 2024-03-28T18:39:31.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:31 smithi131 ceph-mon[36065]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T18:39:31.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122474 s, 856 MB/s 2024-03-28T18:39:31.782 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-03-28T18:39:32.404 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:32 smithi131 ceph-mon[36065]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:32.404 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:32 smithi131 ceph-mon[36065]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:39:32.431 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_1 2024-03-28T18:39:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:32 smithi119 ceph-mon[31813]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:32 smithi119 ceph-mon[31813]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:39:32.672 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:39:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:32 smithi196 ceph-mon[36300]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:32 smithi196 ceph-mon[36300]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:34.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:34.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 -- 172.21.15.131:0/4165174000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581010c0 msgr2=0x7fca581014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/4165174000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581010c0 0x7fca581014e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fca480099c0 tx=0x7fca48009cd0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 -- 172.21.15.131:0/4165174000 shutdown_connections 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/4165174000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca58102c60 0x7fca58109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/4165174000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581010c0 0x7fca581014e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/4165174000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca581022c0 0x7fca58102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 -- 172.21.15.131:0/4165174000 >> 172.21.15.131:0/4165174000 conn(0x7fca580fc660 msgr2=0x7fca580feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 -- 172.21.15.131:0/4165174000 shutdown_connections 2024-03-28T18:39:34.359 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 -- 172.21.15.131:0/4165174000 wait complete. 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.356+0000 7fca5d6a1700 1 Processor -- start 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 -- start start 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca581010c0 0x7fca58198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca58102c60 0x7fca5819fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5810cd30 con 0x7fca581010c0 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5810cbb0 con 0x7fca581022c0 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5810ceb0 con 0x7fca58102c60 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:59400/0 (socket says 172.21.15.131:59400) 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 -- 172.21.15.131:0/3348711 learned_addr learned my addr 172.21.15.131:0/3348711 (peer_addr_for_me v2:172.21.15.131:0/0) 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca577fe700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca58102c60 0x7fca5819fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:34.360 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 -- 172.21.15.131:0/3348711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca58102c60 msgr2=0x7fca5819fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca58102c60 0x7fca5819fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 -- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca581010c0 msgr2=0x7fca58198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca581010c0 0x7fca58198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 -- 172.21.15.131:0/3348711 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca581a00d0 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca567fc700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9c0 tx=0x7fca4c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00cbd0 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca4c00cd70 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca581a03c0 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.357+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c010430 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.358+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca581a0930 con 0x7fca581022c0 2024-03-28T18:39:34.361 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.359+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca58068e50 con 0x7fca581022c0 2024-03-28T18:39:34.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.360+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fca4c004750 con 0x7fca581022c0 2024-03-28T18:39:34.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.360+0000 7fca3ffff700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 0x7fca400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:39:34.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.360+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fca4c0ce4b0 con 0x7fca581022c0 2024-03-28T18:39:34.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.361+0000 7fca56ffd700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 0x7fca400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:39:34.362 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.361+0000 7fca56ffd700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 0x7fca400a4d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca48003790 tx=0x7fca48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:39:34.364 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.364+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca4c09bd50 con 0x7fca581022c0 2024-03-28T18:39:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:34.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.570 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:34.570+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fca580fd3f0 con 0x7fca400a2890 2024-03-28T18:39:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi119 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:35.512 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:35.513 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:35.513 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:35.513 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:35.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi119 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:35.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi119 2024-03-28T18:39:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 4272M 2024-03-28T18:39:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:35.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:35.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:35.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:39:35.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:39:35.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:36.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:36 smithi131 ceph-mon[36065]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:36.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:36 smithi131 ceph-mon[36065]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:36 smithi119 ceph-mon[31813]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:36 smithi119 ceph-mon[31813]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:36 smithi196 ceph-mon[36300]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:39:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:36 smithi196 ceph-mon[36300]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/267057619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]': finished 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:38.487 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:38 smithi196 ceph-mon[36300]: from='client.? 172.21.15.131:0/3657654638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/267057619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]': finished 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:38 smithi119 ceph-mon[31813]: from='client.? 172.21.15.131:0/3657654638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/267057619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]: dispatch 2024-03-28T18:39:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46b91f6b-8ffe-4d4e-9c36-d3faafc12d12"}]': finished 2024-03-28T18:39:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T18:39:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:38 smithi131 ceph-mon[36065]: from='client.? 172.21.15.131:0/3657654638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:39:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:40 smithi119 ceph-mon[31813]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:40 smithi196 ceph-mon[36300]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:40 smithi131 ceph-mon[36065]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:41.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:41 smithi131 ceph-mon[36065]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:41.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:41 smithi119 ceph-mon[31813]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:41.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:41 smithi196 ceph-mon[36300]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:44.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:43 smithi119 ceph-mon[31813]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:44.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:43 smithi196 ceph-mon[36300]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:43 smithi131 ceph-mon[36065]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:45.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:39:45.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:44 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:45.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:39:45.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:44 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:45.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T18:39:45.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:44 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:46.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:45 smithi131 ceph-mon[36065]: Deploying daemon osd.10 on smithi131 2024-03-28T18:39:46.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:45 smithi131 ceph-mon[36065]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:45 smithi119 ceph-mon[31813]: Deploying daemon osd.10 on smithi131 2024-03-28T18:39:46.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:45 smithi119 ceph-mon[31813]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:46.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:45 smithi196 ceph-mon[36300]: Deploying daemon osd.10 on smithi131 2024-03-28T18:39:46.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:45 smithi196 ceph-mon[36300]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:48.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:48 smithi131 ceph-mon[36065]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:48.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:48 smithi119 ceph-mon[31813]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:48.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:48 smithi196 ceph-mon[36300]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:50.259 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:50 smithi131 ceph-mon[36065]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:50.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:50 smithi119 ceph-mon[31813]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:50 smithi196 ceph-mon[36300]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:51.932 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:51.932 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:51.932 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:51 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:51.932 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:51 smithi131 ceph-mon[36065]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:51 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:51 smithi196 ceph-mon[36300]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:51 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:51 smithi119 ceph-mon[31813]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:54.190 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:54.191 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.191 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.191 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:54.191 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:54.191 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:54.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:54.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:54.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:54.605 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 10 on host 'smithi131' 2024-03-28T18:39:54.605 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.605+0000 7fca3ffff700 1 -- 172.21.15.131:0/3348711 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fca580fd3f0 con 0x7fca400a2890 2024-03-28T18:39:54.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 msgr2=0x7fca400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:54.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 0x7fca400a4d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca48003790 tx=0x7fca48000bc0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 msgr2=0x7fca581989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:39:54.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00e9c0 tx=0x7fca4c00ecd0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.607 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 shutdown_connections 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fca400a2890 0x7fca400a4d50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca58102c60 0x7fca5819fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca581010c0 0x7fca58198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 --2- 172.21.15.131:0/3348711 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fca581022c0 0x7fca581989b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 >> 172.21.15.131:0/3348711 conn(0x7fca580fc660 msgr2=0x7fca580fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 shutdown_connections 2024-03-28T18:39:54.608 INFO:teuthology.orchestra.run.smithi131.stderr:2024-03-28T18:39:54.607+0000 7fca5d6a1700 1 -- 172.21.15.131:0/3348711 wait complete. 2024-03-28T18:39:55.307 DEBUG:teuthology.orchestra.run.smithi131:osd.10> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T18:39:55.311 INFO:tasks.cephadm:Deploying osd.11 on smithi196 with /dev/vg_nvme/lv_1... 2024-03-28T18:39:55.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T18:39:55.560 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.682 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:55 smithi196 ceph-mon[36300]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:55.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:55 smithi119 ceph-mon[31813]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T18:39:55.869 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:55 smithi131 ceph-mon[36065]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:56.840 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-mon[36065]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:39:56.840 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-mon[36065]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:56.840 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-mon[36065]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:39:56.840 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:56.841 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-mon[36065]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:56.841 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:39:56.569+0000 7febf613b700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T18:39:56.841 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:39:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:39:56.585+0000 7febf1f2e700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:39:56.898 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:56 smithi196 ceph-mon[36300]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:39:56.898 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:56 smithi196 ceph-mon[36300]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:56.898 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:56 smithi196 ceph-mon[36300]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:39:56.898 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:56 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:56.898 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:56 smithi196 ceph-mon[36300]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:57.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:56 smithi119 ceph-mon[31813]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T18:39:57.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:56 smithi119 ceph-mon[31813]: from='osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:57.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:56 smithi119 ceph-mon[31813]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T18:39:57.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:56 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:57.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:56 smithi119 ceph-mon[31813]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-28T18:39:57.607 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T18:39:57.607 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T18:39:57.607 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-28T18:39:57.607 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-28T18:39:57.608 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011365 s, 923 MB/s 2024-03-28T18:39:57.608 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-28T18:39:57.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:57 smithi196 ceph-mon[36300]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:39:57.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:57 smithi196 ceph-mon[36300]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:39:57.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:57.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:57 smithi196 ceph-mon[36300]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:57.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:57 smithi131 ceph-mon[36065]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:39:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:57 smithi131 ceph-mon[36065]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:39:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:57 smithi131 ceph-mon[36065]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:57 smithi119 ceph-mon[31813]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-28T18:39:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:57 smithi119 ceph-mon[31813]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T18:39:58.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:57 smithi119 ceph-mon[31813]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-28T18:39:58.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-03-28T18:39:58.473 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] boot 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:58 smithi131 ceph-mon[36065]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:39:58.875 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:39:58.875 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:39:58.875 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] boot 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:58.876 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:58 smithi196 ceph-mon[36300]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:39:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: osd.10 [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] boot 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:39:59.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:39:59.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:39:59.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:58 smithi119 ceph-mon[31813]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T18:39:59.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:59 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi131 2024-03-28T18:39:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:59 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 4270M 2024-03-28T18:39:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:59 smithi196 ceph-mon[36300]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:39:59.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:39:59 smithi196 ceph-mon[36300]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:39:59.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:59 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi131 2024-03-28T18:39:59.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:59 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 4270M 2024-03-28T18:39:59.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:59 smithi131 ceph-mon[36065]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:39:59.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:39:59 smithi131 ceph-mon[36065]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:40:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:59 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi131 2024-03-28T18:40:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:59 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 4270M 2024-03-28T18:40:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:59 smithi119 ceph-mon[31813]: pgmap v259: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:39:59 smithi119 ceph-mon[31813]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 -- 172.21.15.196:0/635420332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 msgr2=0x7f08d8103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 --2- 172.21.15.196:0/635420332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d8103be0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009990 tx=0x7f08cc009ca0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 -- 172.21.15.196:0/635420332 shutdown_connections 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 --2- 172.21.15.196:0/635420332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08d8100be0 0x7f08d810a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 --2- 172.21.15.196:0/635420332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d8103be0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 --2- 172.21.15.196:0/635420332 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d81006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 -- 172.21.15.196:0/635420332 >> 172.21.15.196:0/635420332 conn(0x7f08d80fbde0 msgr2=0x7f08d80fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 -- 172.21.15.196:0/635420332 shutdown_connections 2024-03-28T18:40:00.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.160+0000 7f08dd059700 1 -- 172.21.15.196:0/635420332 wait complete. 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 Processor -- start 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 -- start start 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d8198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08d8100be0 0x7f08d8198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d810de40 con 0x7f08d8100be0 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d810dcc0 con 0x7f08d8100240 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08dd059700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d810dfc0 con 0x7f08d81037c0 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08d759e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08d759e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:44358/0 (socket says 172.21.15.196:44358) 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.161+0000 7f08d759e700 1 -- 172.21.15.196:0/239789050 learned_addr learned my addr 172.21.15.196:0/239789050 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d6d9d700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d8198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 -- 172.21.15.196:0/239789050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 msgr2=0x7f08d8198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:00.167 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d8198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 -- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08d8100be0 msgr2=0x7f08d8198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08d8100be0 0x7f08d8198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 -- 172.21.15.196:0/239789050 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d81a0190 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d6d9d700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08d759e700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f08d000b9a0 tx=0x7f08d000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d0019070 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d81a0480 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08d0014440 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00075d0 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.162+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d81a0a90 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.163+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d8068e50 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.164+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f08d001c070 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.164+0000 7f08bffff700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 0x7f08c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.165+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f08d00cd860 con 0x7f08d81037c0 2024-03-28T18:40:00.168 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.165+0000 7f08d6d9d700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 0x7f08c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:00.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.166+0000 7f08d6d9d700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 0x7f08c00a4d50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:00.169 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.169+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08d009afc0 con 0x7f08d81037c0 2024-03-28T18:40:00.372 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:00.372+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f08d80625d0 con 0x7f08c00a2890 2024-03-28T18:40:00.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:00 smithi196 ceph-mon[36300]: overall HEALTH_OK 2024-03-28T18:40:00.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:40:00.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:40:00.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:00.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:00 smithi196 ceph-mon[36300]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:40:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:00 smithi131 ceph-mon[36065]: overall HEALTH_OK 2024-03-28T18:40:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:40:00.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:40:00.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:00.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:00 smithi131 ceph-mon[36065]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:40:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:00 smithi119 ceph-mon[31813]: overall HEALTH_OK 2024-03-28T18:40:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T18:40:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T18:40:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:00 smithi119 ceph-mon[31813]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T18:40:01.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:01 smithi196 ceph-mon[36300]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:01.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:01 smithi196 ceph-mon[36300]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:01.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:01 smithi196 ceph-mon[36300]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:40:01.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:01 smithi131 ceph-mon[36065]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:01.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:01 smithi131 ceph-mon[36065]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:01.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:01 smithi131 ceph-mon[36065]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:40:02.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:01 smithi119 ceph-mon[31813]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:02.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:01 smithi119 ceph-mon[31813]: pgmap v262: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:02.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:01 smithi119 ceph-mon[31813]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T18:40:04.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/158214346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]': finished 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:04.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:03 smithi119 ceph-mon[31813]: from='client.? 172.21.15.196:0/635821320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/158214346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]': finished 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:03 smithi196 ceph-mon[36300]: from='client.? 172.21.15.196:0/635821320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:40:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-28T18:40:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/158214346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]: dispatch 2024-03-28T18:40:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb4209c-3199-4138-b104-8f07d626633d"}]': finished 2024-03-28T18:40:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T18:40:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:03 smithi131 ceph-mon[36065]: from='client.? 172.21.15.196:0/635821320' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T18:40:06.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:05 smithi119 ceph-mon[31813]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-28T18:40:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:05 smithi196 ceph-mon[36300]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-28T18:40:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:05 smithi131 ceph-mon[36065]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-28T18:40:08.150 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:08 smithi196 ceph-mon[36300]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T18:40:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:08 smithi119 ceph-mon[31813]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T18:40:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:08 smithi131 ceph-mon[36065]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T18:40:10.263 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:10 smithi196 ceph-mon[36300]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:40:10.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:10 smithi119 ceph-mon[31813]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:40:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:10 smithi131 ceph-mon[36065]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T18:40:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:40:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:11.330 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:40:11.330 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T18:40:11.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:12 smithi119 ceph-mon[31813]: Deploying daemon osd.11 on smithi196 2024-03-28T18:40:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:12 smithi119 ceph-mon[31813]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:40:12.363 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:12 smithi196 ceph-mon[36300]: Deploying daemon osd.11 on smithi196 2024-03-28T18:40:12.363 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:12 smithi196 ceph-mon[36300]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:40:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:12 smithi131 ceph-mon[36065]: Deploying daemon osd.11 on smithi196 2024-03-28T18:40:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:12 smithi131 ceph-mon[36065]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T18:40:14.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:14 smithi196 ceph-mon[36300]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:14 smithi119 ceph-mon[31813]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:14 smithi131 ceph-mon[36065]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:16 smithi119 ceph-mon[31813]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:16.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:16 smithi196 ceph-mon[36300]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:16 smithi131 ceph-mon[36065]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:17.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:18 smithi196 ceph-mon[36300]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:18 smithi119 ceph-mon[31813]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:18.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:18 smithi131 ceph-mon[36065]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:19.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.655+0000 7f08bffff700 1 -- 172.21.15.196:0/239789050 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f08d80625d0 con 0x7f08c00a2890 2024-03-28T18:40:19.656 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 11 on host 'smithi196' 2024-03-28T18:40:19.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 msgr2=0x7f08c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:19.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 0x7f08c00a4d50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08cc000c00 tx=0x7f08cc005fb0 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 msgr2=0x7f08d819faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f08d000b9a0 tx=0x7f08d000bd60 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 shutdown_connections 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f08c00a2890 0x7f08c00a4d50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08d81037c0 0x7f08d819faf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f08d8100240 0x7f08d8198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 --2- 172.21.15.196:0/239789050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08d8100be0 0x7f08d8198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.657+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 >> 172.21.15.196:0/239789050 conn(0x7f08d80fbde0 msgr2=0x7f08d8106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.658+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 shutdown_connections 2024-03-28T18:40:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:19.658+0000 7f08dd059700 1 -- 172.21.15.196:0/239789050 wait complete. 2024-03-28T18:40:20.281 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:20 smithi196 ceph-mon[36300]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:20.281 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.282 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.282 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:20 smithi196 ceph-mon[36300]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.282 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:20 smithi196 ceph-mon[36300]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.309 DEBUG:teuthology.orchestra.run.smithi196:osd.11> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T18:40:20.312 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T18:40:20.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T18:40:20.497 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:20 smithi131 ceph-mon[36065]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:20.497 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.497 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.497 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:20 smithi131 ceph-mon[36065]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.497 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:20 smithi131 ceph-mon[36065]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:20 smithi119 ceph-mon[31813]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:20.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:20.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:20 smithi119 ceph-mon[31813]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:20 smithi119 ceph-mon[31813]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T18:40:20.781 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:40:21.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:21 smithi119 ceph-mon[31813]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:40:21.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:21.773 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.773 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:21 smithi131 ceph-mon[36065]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:21.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:21.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T18:40:21.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T18:40:21.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:21.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-28T18:40:21.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-mon[36300]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:21.936 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:40:21.640+0000 7f175661b700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T18:40:21.936 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:40:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:40:21.657+0000 7f175240e700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T18:40:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:22 smithi119 ceph-mon[31813]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:40:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:22 smithi119 ceph-mon[31813]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:40:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:22 smithi196 ceph-mon[36300]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:40:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:22 smithi196 ceph-mon[36300]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:40:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:22 smithi131 ceph-mon[36065]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-28T18:40:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:22 smithi131 ceph-mon[36065]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T18:40:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 -- 172.21.15.119:0/321656866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105fa0 msgr2=0x7f958410c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:23.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 --2- 172.21.15.119:0/321656866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105fa0 0x7f958410c850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f957c009fc0 tx=0x7f957c00b330 comp rx=0 tx=0).stop 2024-03-28T18:40:23.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 -- 172.21.15.119:0/321656866 shutdown_connections 2024-03-28T18:40:23.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 --2- 172.21.15.119:0/321656866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105fa0 0x7f958410c850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 --2- 172.21.15.119:0/321656866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584104400 0x7f9584104820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 --2- 172.21.15.119:0/321656866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584105600 0x7f9584105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 -- 172.21.15.119:0/321656866 >> 172.21.15.119:0/321656866 conn(0x7f95840ff9e0 msgr2=0x7f9584101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 -- 172.21.15.119:0/321656866 shutdown_connections 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.035+0000 7f958c400700 1 -- 172.21.15.119:0/321656866 wait complete. 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.036+0000 7f958c400700 1 Processor -- start 2024-03-28T18:40:23.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 -- start start 2024-03-28T18:40:23.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:23.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 0x7f958419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:23.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 0x7f95841a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:23.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584110020 con 0x7f9584104400 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958410fea0 con 0x7f9584105600 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958c400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95841101a0 con 0x7f9584105fa0 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f9589a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f958a400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 0x7f95841a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f9589a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38504/0 (socket says 172.21.15.119:38504) 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f9589a00700 1 -- 172.21.15.119:0/722719945 learned_addr learned my addr 172.21.15.119:0/722719945 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.037+0000 7f9589000700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 0x7f958419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:23.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 -- 172.21.15.119:0/722719945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 msgr2=0x7f958419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 0x7f958419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 -- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 msgr2=0x7f958419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 -- 172.21.15.119:0/722719945 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95841a2250 con 0x7f9584105fa0 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f9589000700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 0x7f958419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f9589a00700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958a400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 0x7f95841a1bb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f957c007c80 tx=0x7f957c007cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c01f070 con 0x7f9584105fa0 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f957c003e10 con 0x7f9584105fa0 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95841a24e0 con 0x7f9584105fa0 2024-03-28T18:40:23.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c01b6a0 con 0x7f9584105fa0 2024-03-28T18:40:23.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.038+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95841a2930 con 0x7f9584105fa0 2024-03-28T18:40:23.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.039+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9584068e50 con 0x7f9584105fa0 2024-03-28T18:40:23.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.042+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f957c022070 con 0x7f9584105fa0 2024-03-28T18:40:23.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.043+0000 7f9582000700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 0x7f956c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:23.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.043+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f957c0cf5e0 con 0x7f9584105fa0 2024-03-28T18:40:23.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.043+0000 7f9589a00700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 0x7f956c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:23.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.044+0000 7f9589a00700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 0x7f956c0a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9578005950 tx=0x7f9578009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:23.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.045+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f957c09cb10 con 0x7f9584105fa0 2024-03-28T18:40:23.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.267+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f958402db00 con 0x7f9584105fa0 2024-03-28T18:40:23.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.268+0000 7f9582000700 1 -- 172.21.15.119:0/722719945 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f957c09c410 con 0x7f9584105fa0 2024-03-28T18:40:23.268 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.270+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 msgr2=0x7f956c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.270+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 0x7f956c0a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9578005950 tx=0x7f9578009260 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.270+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 msgr2=0x7f95841a1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 0x7f95841a1bb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f957c007c80 tx=0x7f957c007cb0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 shutdown_connections 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2910 0x7f956c0a4dd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584105fa0 0x7f95841a1bb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9584104400 0x7f958419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 --2- 172.21.15.119:0/722719945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9584105600 0x7f958419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 >> 172.21.15.119:0/722719945 conn(0x7f95840ff9e0 msgr2=0x7f9584108830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 shutdown_connections 2024-03-28T18:40:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:23.271+0000 7f958c400700 1 -- 172.21.15.119:0/722719945 wait complete. 2024-03-28T18:40:23.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: purged_snaps scrub starts 2024-03-28T18:40:23.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: purged_snaps scrub ok 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] boot 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:23.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:23 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/722719945' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:40:23.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: purged_snaps scrub starts 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: purged_snaps scrub ok 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] boot 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:23 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/722719945' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: purged_snaps scrub starts 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: purged_snaps scrub ok 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: osd.11 [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] boot 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-28T18:40:23.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:23 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/722719945' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T18:40:24.197 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711651222,"num_in_osds":12,"osd_in_since":1711651203,"num_remapped_pgs":0} 2024-03-28T18:40:24.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:40:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: Detected new or changed devices on smithi196 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:24 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: Detected new or changed devices on smithi196 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.942 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.942 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:24.942 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.942 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.942 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:24 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: Detected new or changed devices on smithi196 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:24.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:24.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:24.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:24 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:25.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:25 smithi196 ceph-mon[36300]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:25.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:25 smithi131 ceph-mon[36065]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:26.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:25 smithi119 ceph-mon[31813]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:26.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 -- 172.21.15.119:0/1404669810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 msgr2=0x7f9c2c102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:26.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 --2- 172.21.15.119:0/1404669810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c102820 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9c200099c0 tx=0x7f9c20009cd0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 -- 172.21.15.119:0/1404669810 shutdown_connections 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 --2- 172.21.15.119:0/1404669810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 0x7f9c2c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 --2- 172.21.15.119:0/1404669810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c102820 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 --2- 172.21.15.119:0/1404669810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 0x7f9c2c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 -- 172.21.15.119:0/1404669810 >> 172.21.15.119:0/1404669810 conn(0x7f9c2c0792f0 msgr2=0x7f9c2c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 -- 172.21.15.119:0/1404669810 shutdown_connections 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.984+0000 7f9c33c00700 1 -- 172.21.15.119:0/1404669810 wait complete. 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.985+0000 7f9c33c00700 1 Processor -- start 2024-03-28T18:40:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.985+0000 7f9c33c00700 1 -- start start 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 0x7f9c2c0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 0x7f9c2c074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c112570 con 0x7f9c2c102440 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c1123f0 con 0x7f9c2c102df0 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c33c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c1126f0 con 0x7f9c2c107510 2024-03-28T18:40:26.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c31200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c31200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38508/0 (socket says 172.21.15.119:38508) 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c31200700 1 -- 172.21.15.119:0/3451179707 learned_addr learned my addr 172.21.15.119:0/3451179707 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.986+0000 7f9c2be00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 0x7f9c2c0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 0x7f9c2c074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 -- 172.21.15.119:0/3451179707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 msgr2=0x7f9c2c074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 0x7f9c2c074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 -- 172.21.15.119:0/3451179707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 msgr2=0x7f9c2c0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 0x7f9c2c0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 -- 172.21.15.119:0/3451179707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c1a6f50 con 0x7f9c2c102440 2024-03-28T18:40:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.987+0000 7f9c31200700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9c20005910 tx=0x7f9c2000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.988+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c20027070 con 0x7f9c2c102440 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.988+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c20003680 con 0x7f9c2c102440 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.988+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c1a71e0 con 0x7f9c2c102440 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.988+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c200173f0 con 0x7f9c2c102440 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.988+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c1a7630 con 0x7f9c2c102440 2024-03-28T18:40:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.989+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c068e50 con 0x7f9c2c102440 2024-03-28T18:40:26.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.991+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9c20022590 con 0x7f9c2c102440 2024-03-28T18:40:26.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.992+0000 7f9c29600700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 0x7f9c140a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:26.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.992+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9c200cf410 con 0x7f9c2c102440 2024-03-28T18:40:26.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.992+0000 7f9c2be00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 0x7f9c140a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:26.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.993+0000 7f9c2be00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 0x7f9c140a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009680 tx=0x7f9c18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:26.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:26.995+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c20013070 con 0x7f9c2c102440 2024-03-28T18:40:27.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.218+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c2c02dbf0 con 0x7f9c2c102440 2024-03-28T18:40:27.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.221+0000 7f9c29600700 1 -- 172.21.15.119:0/3451179707 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18516 (secure 0 0 0) 0x7f9c2009c940 con 0x7f9c2c102440 2024-03-28T18:40:27.221 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:27.222 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:40:23.646844+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.224+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 msgr2=0x7f9c140a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.224+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 0x7f9c140a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9c18009680 tx=0x7f9c18009260 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 msgr2=0x7f9c2c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9c20005910 tx=0x7f9c2000f690 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 shutdown_connections 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9c140a2910 0x7f9c140a4dd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2c107510 0x7f9c2c074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c2c102440 0x7f9c2c074490 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 --2- 172.21.15.119:0/3451179707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c2c102df0 0x7f9c2c0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:27.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 >> 172.21.15.119:0/3451179707 conn(0x7f9c2c0792f0 msgr2=0x7f9c2c1003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:27.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 shutdown_connections 2024-03-28T18:40:27.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:27.225+0000 7f9c33c00700 1 -- 172.21.15.119:0/3451179707 wait complete. 2024-03-28T18:40:28.015 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:40:28.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T18:40:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:28 smithi119 ceph-mon[31813]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:28 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3451179707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:28 smithi196 ceph-mon[36300]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:28 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3451179707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:28 smithi131 ceph-mon[36065]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:28 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3451179707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:28.463 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:40:30.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:30 smithi119 ceph-mon[31813]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:30 smithi196 ceph-mon[36300]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:30 smithi131 ceph-mon[36065]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.566+0000 7fc87f000700 1 -- 172.21.15.119:0/3881914290 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101cd0 msgr2=0x7fc878105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.566+0000 7fc87f000700 1 --2- 172.21.15.119:0/3881914290 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101cd0 0x7fc878105cd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc8640098f0 tx=0x7fc864009c00 comp rx=0 tx=0).stop 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 -- 172.21.15.119:0/3881914290 shutdown_connections 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 --2- 172.21.15.119:0/3881914290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 --2- 172.21.15.119:0/3881914290 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101320 0x7fc878101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 --2- 172.21.15.119:0/3881914290 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101cd0 0x7fc878105cd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 -- 172.21.15.119:0/3881914290 >> 172.21.15.119:0/3881914290 conn(0x7fc8780770b0 msgr2=0x7fc8780774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 -- 172.21.15.119:0/3881914290 shutdown_connections 2024-03-28T18:40:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.567+0000 7fc87f000700 1 -- 172.21.15.119:0/3881914290 wait complete. 2024-03-28T18:40:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.568+0000 7fc87f000700 1 Processor -- start 2024-03-28T18:40:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 -- start start 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101320 0x7fc8781a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 0x7fc8781a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8781113a0 con 0x7fc8781063b0 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878111220 con 0x7fc878101cd0 2024-03-28T18:40:30.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87f000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878111520 con 0x7fc878101320 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc877400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 0x7fc8781a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87d000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87819aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34656/0 (socket says 172.21.15.119:34656) 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc87d000700 1 -- 172.21.15.119:0/4133680530 learned_addr learned my addr 172.21.15.119:0/4133680530 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.569+0000 7fc877e00700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101320 0x7fc8781a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 -- 172.21.15.119:0/4133680530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101320 msgr2=0x7fc8781a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101320 0x7fc8781a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 -- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 msgr2=0x7fc87819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 -- 172.21.15.119:0/4133680530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87819b210 con 0x7fc878101cd0 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc877400700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 0x7fc8781a0de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc864000c00 tx=0x7fc86400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864005040 con 0x7fc878101cd0 2024-03-28T18:40:30.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8640056c0 con 0x7fc878101cd0 2024-03-28T18:40:30.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.570+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87819b4a0 con 0x7fc878101cd0 2024-03-28T18:40:30.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.571+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc864003a00 con 0x7fc878101cd0 2024-03-28T18:40:30.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.571+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8781a7770 con 0x7fc878101cd0 2024-03-28T18:40:30.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.572+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8781027b0 con 0x7fc878101cd0 2024-03-28T18:40:30.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.574+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc86400b560 con 0x7fc878101cd0 2024-03-28T18:40:30.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.575+0000 7fc874c00700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 0x7fc85c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:30.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.575+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8640cf7e0 con 0x7fc878101cd0 2024-03-28T18:40:30.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.575+0000 7fc877e00700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 0x7fc85c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:30.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.576+0000 7fc877e00700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 0x7fc85c0a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc878100170 tx=0x7fc868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:30.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.577+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc864013030 con 0x7fc878101cd0 2024-03-28T18:40:30.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.793+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc878068e50 con 0x7fc878101cd0 2024-03-28T18:40:30.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.793+0000 7fc874c00700 1 -- 172.21.15.119:0/4133680530 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc86409cd10 con 0x7fc878101cd0 2024-03-28T18:40:30.793 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:40:30.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 msgr2=0x7fc85c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 0x7fc85c0a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc878100170 tx=0x7fc868009260 comp rx=0 tx=0).stop 2024-03-28T18:40:30.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 msgr2=0x7fc8781a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:30.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 0x7fc8781a0de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc864000c00 tx=0x7fc86400fa00 comp rx=0 tx=0).stop 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 shutdown_connections 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc85c0a2890 0x7fc85c0a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8781063b0 0x7fc87819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc878101320 0x7fc8781a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 --2- 172.21.15.119:0/4133680530 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc878101cd0 0x7fc8781a0de0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.796+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 >> 172.21.15.119:0/4133680530 conn(0x7fc8780770b0 msgr2=0x7fc878107660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.797+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 shutdown_connections 2024-03-28T18:40:30.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:30.797+0000 7fc87f000700 1 -- 172.21.15.119:0/4133680530 wait complete. 2024-03-28T18:40:31.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:31 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4133680530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:31 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4133680530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:31 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4133680530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:31.587 INFO:tasks.cephadm:Adding mds.a on smithi119 2024-03-28T18:40:31.587 INFO:tasks.cephadm:Adding mds.d on smithi119 2024-03-28T18:40:31.587 INFO:tasks.cephadm:Adding mds.g on smithi119 2024-03-28T18:40:31.587 INFO:tasks.cephadm:Adding mds.j on smithi119 2024-03-28T18:40:31.587 INFO:tasks.cephadm:Adding mds.b on smithi131 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.e on smithi131 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.h on smithi131 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.k on smithi131 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.c on smithi196 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.f on smithi196 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.i on smithi196 2024-03-28T18:40:31.588 INFO:tasks.cephadm:Adding mds.l on smithi196 2024-03-28T18:40:31.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l' 2024-03-28T18:40:31.839 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:40:32.283 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:32 smithi196 ceph-mon[36300]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:32.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:32 smithi119 ceph-mon[31813]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:32 smithi131 ceph-mon[36065]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.428+0000 7f688aca4700 1 -- 172.21.15.196:0/251620732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 msgr2=0x7f688410a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.428+0000 7f688aca4700 1 --2- 172.21.15.196:0/251620732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f688410a6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f687c00b0a0 tx=0x7f687c00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- 172.21.15.196:0/251620732 shutdown_connections 2024-03-28T18:40:33.430 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- 172.21.15.196:0/251620732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f688410a6d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- 172.21.15.196:0/251620732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68841004a0 0x7f68841008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- 172.21.15.196:0/251620732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6884103480 0x7f68841038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- 172.21.15.196:0/251620732 >> 172.21.15.196:0/251620732 conn(0x7f68840fbe20 msgr2=0x7f68840fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- 172.21.15.196:0/251620732 shutdown_connections 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- 172.21.15.196:0/251620732 wait complete. 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 Processor -- start 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- start start 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 0x7f6884102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6884103480 0x7f6884101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688410deb0 con 0x7f6884103480 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688410dd30 con 0x7f68841004a0 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f688aca4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688410e030 con 0x7f6884103e20 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f6889241700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:33.431 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f6889241700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:54822/0 (socket says 172.21.15.196:54822) 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.429+0000 7f6889241700 1 -- 172.21.15.196:0/3212170605 learned_addr learned my addr 172.21.15.196:0/3212170605 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6888a40700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 0x7f6884102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 -- 172.21.15.196:0/3212170605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 msgr2=0x7f6884102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 0x7f6884102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 -- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6884103480 msgr2=0x7f6884101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6884103480 0x7f6884101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 -- 172.21.15.196:0/3212170605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884102220 con 0x7f6884103e20 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6888a40700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 0x7f6884102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6889241700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f687c007f00 tx=0x7f687c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c021030 con 0x7f6884103e20 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f687c007bb0 con 0x7f6884103e20 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68841a2540 con 0x7f6884103e20 2024-03-28T18:40:33.432 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c01d470 con 0x7f6884103e20 2024-03-28T18:40:33.433 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.430+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68841a2a20 con 0x7f6884103e20 2024-03-28T18:40:33.433 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.431+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6884068e50 con 0x7f6884103e20 2024-03-28T18:40:33.433 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.432+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f687c024070 con 0x7f6884103e20 2024-03-28T18:40:33.433 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.433+0000 7f6881ffb700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 0x7f68740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:33.434 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.433+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f687c00f020 con 0x7f6884103e20 2024-03-28T18:40:33.434 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.433+0000 7f6888a40700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 0x7f68740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:33.434 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.434+0000 7f6888a40700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 0x7f68740a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f688406a6f0 tx=0x7f6878008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:33.436 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.436+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f687c09cd90 con 0x7f6884103e20 2024-03-28T18:40:33.641 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.641+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f68840626d0 con 0x7f68740a2890 2024-03-28T18:40:33.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.657+0000 7f6881ffb700 1 -- 172.21.15.196:0/3212170605 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f68840626d0 con 0x7f68740a2890 2024-03-28T18:40:33.657 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mds.all update... 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 msgr2=0x7f68740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 0x7f68740a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f688406a6f0 tx=0x7f6878008040 comp rx=0 tx=0).stop 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 msgr2=0x7f6884101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f687c007f00 tx=0x7f687c00bf10 comp rx=0 tx=0).stop 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 shutdown_connections 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f68740a2890 0x7f68740a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6884103e20 0x7f6884101970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f68841004a0 0x7f6884102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 --2- 172.21.15.196:0/3212170605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6884103480 0x7f6884101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 >> 172.21.15.196:0/3212170605 conn(0x7f68840fbe20 msgr2=0x7f68841066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 shutdown_connections 2024-03-28T18:40:33.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-28T18:40:33.660+0000 7f688aca4700 1 -- 172.21.15.196:0/3212170605 wait complete. 2024-03-28T18:40:34.304 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:34 smithi196 ceph-mon[36300]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:34.304 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:34.304 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:34 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:34 smithi119 ceph-mon[31813]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:34 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:34.329 DEBUG:teuthology.orchestra.run.smithi119:mds.a> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T18:40:34.335 DEBUG:teuthology.orchestra.run.smithi119:mds.d> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T18:40:34.380 DEBUG:teuthology.orchestra.run.smithi119:mds.g> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T18:40:34.427 DEBUG:teuthology.orchestra.run.smithi119:mds.j> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T18:40:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:34 smithi131 ceph-mon[36065]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:34 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:40:34.473 DEBUG:teuthology.orchestra.run.smithi131:mds.b> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T18:40:34.475 DEBUG:teuthology.orchestra.run.smithi131:mds.e> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T18:40:34.521 DEBUG:teuthology.orchestra.run.smithi131:mds.h> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T18:40:34.564 DEBUG:teuthology.orchestra.run.smithi131:mds.k> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T18:40:34.606 DEBUG:teuthology.orchestra.run.smithi196:mds.c> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T18:40:34.609 DEBUG:teuthology.orchestra.run.smithi196:mds.f> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T18:40:34.652 DEBUG:teuthology.orchestra.run.smithi196:mds.i> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T18:40:34.695 DEBUG:teuthology.orchestra.run.smithi196:mds.l> sudo journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T18:40:34.738 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T18:40:34.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:40:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 -- 172.21.15.119:0/1834744885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8076060 msgr2=0x7f92e8076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 --2- 172.21.15.119:0/1834744885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8076060 0x7f92e8076480 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009ca0 tx=0x7f92d802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 -- 172.21.15.119:0/1834744885 shutdown_connections 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 --2- 172.21.15.119:0/1834744885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8077c40 0x7f92e80784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 --2- 172.21.15.119:0/1834744885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 0x7f92e8077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 --2- 172.21.15.119:0/1834744885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8076060 0x7f92e8076480 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.967+0000 7f92ef200640 1 -- 172.21.15.119:0/1834744885 >> 172.21.15.119:0/1834744885 conn(0x7f92e80fee50 msgr2=0x7f92e8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.968+0000 7f92ef200640 1 -- 172.21.15.119:0/1834744885 shutdown_connections 2024-03-28T18:40:34.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.968+0000 7f92ef200640 1 -- 172.21.15.119:0/1834744885 wait complete. 2024-03-28T18:40:34.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.969+0000 7f92ef200640 1 Processor -- start 2024-03-28T18:40:34.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.969+0000 7f92ef200640 1 -- start start 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 0x7f92e819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 0x7f92e81a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8113960 con 0x7f92e8077c40 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e81137e0 con 0x7f92e80772a0 2024-03-28T18:40:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ef200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e8113ae0 con 0x7f92e8076060 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92e7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92ed200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 0x7f92e81a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92e7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59478/0 (socket says 172.21.15.119:59478) 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92e7e00640 1 -- 172.21.15.119:0/998372764 learned_addr learned my addr 172.21.15.119:0/998372764 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.970+0000 7f92df400640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 0x7f92e819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 -- 172.21.15.119:0/998372764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 msgr2=0x7f92e819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 0x7f92e819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 -- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 msgr2=0x7f92e81a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 0x7f92e81a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 -- 172.21.15.119:0/998372764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e81a5be0 con 0x7f92e8076060 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92ed200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 0x7f92e81a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e7e00640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92d80325f0 tx=0x7f92d8032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8041020 con 0x7f92e8076060 2024-03-28T18:40:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92d8034cd0 con 0x7f92e8076060 2024-03-28T18:40:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8039040 con 0x7f92e8076060 2024-03-28T18:40:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e81a5e70 con 0x7f92e8076060 2024-03-28T18:40:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.971+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e81a62d0 con 0x7f92e8076060 2024-03-28T18:40:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.972+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e8110400 con 0x7f92e8076060 2024-03-28T18:40:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.975+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f92d8044020 con 0x7f92e8076060 2024-03-28T18:40:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.976+0000 7f92e5600640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 0x7f92cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.976+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92d80f0810 con 0x7f92e8076060 2024-03-28T18:40:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.976+0000 7f92df400640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 0x7f92cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.977+0000 7f92df400640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 0x7f92cc0a4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92d00095f0 tx=0x7f92d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:34.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:34.978+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f92d804b020 con 0x7f92e8076060 2024-03-28T18:40:35.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.182+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92e802d2c0 con 0x7f92e8076060 2024-03-28T18:40:35.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.184+0000 7f92e5600640 1 -- 172.21.15.119:0/998372764 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18516 (secure 0 0 0) 0x7f92d80b9070 con 0x7f92e8076060 2024-03-28T18:40:35.185 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:35.185 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:40:23.646844+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:40:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.186+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 msgr2=0x7f92cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.186+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 0x7f92cc0a4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92d00095f0 tx=0x7f92d0005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.186+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 msgr2=0x7f92e819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.186+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f92d80325f0 tx=0x7f92d8032ec0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 shutdown_connections 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92cc0a2880 0x7f92cc0a4d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92e8077c40 0x7f92e81a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92e80772a0 0x7f92e819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 --2- 172.21.15.119:0/998372764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92e8076060 0x7f92e819dfc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 >> 172.21.15.119:0/998372764 conn(0x7f92e80fee50 msgr2=0x7f92e8100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 shutdown_connections 2024-03-28T18:40:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.187+0000 7f92ef200640 1 -- 172.21.15.119:0/998372764 wait complete. 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: Saving service mds.all spec with placement smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:35.204 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:35.205 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:35 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:40:35.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:40:35.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 -- 172.21.15.119:0/880003582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 msgr2=0x7fb7ec102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 --2- 172.21.15.119:0/880003582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec102d20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009900 tx=0x7fb7d402bc50 comp rx=0 tx=0).stop 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 -- 172.21.15.119:0/880003582 shutdown_connections 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 --2- 172.21.15.119:0/880003582 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec103260 0x7fb7ec109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 --2- 172.21.15.119:0/880003582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec102d20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 --2- 172.21.15.119:0/880003582 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 -- 172.21.15.119:0/880003582 >> 172.21.15.119:0/880003582 conn(0x7fb7ec0fcc40 msgr2=0x7fb7ec0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 -- 172.21.15.119:0/880003582 shutdown_connections 2024-03-28T18:40:35.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.423+0000 7fb7f2200640 1 -- 172.21.15.119:0/880003582 wait complete. 2024-03-28T18:40:35.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.424+0000 7fb7f2200640 1 Processor -- start 2024-03-28T18:40:35.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.424+0000 7fb7f2200640 1 -- start start 2024-03-28T18:40:35.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 0x7fb7ec1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec10d360 con 0x7fb7ec1028a0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec10d1e0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7f2200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec10d4e0 con 0x7fb7ec103260 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46590/0 (socket says 172.21.15.119:46590) 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 -- 172.21.15.119:0/2077220025 learned_addr learned my addr 172.21.15.119:0/2077220025 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7e2a00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7ebe00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 0x7fb7ec1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 -- 172.21.15.119:0/2077220025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 msgr2=0x7fb7ec1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 0x7fb7ec1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 -- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 msgr2=0x7fb7ec19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7eb400640 1 -- 172.21.15.119:0/2077220025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec1a17b0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.425+0000 7fb7ebe00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 0x7fb7ec1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:40:35.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7eb400640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00e9c0 tx=0x7fb7dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00cd80 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7dc0035f0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec1a1aa0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc010690 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.426+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec1a2010 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.427+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec1a21e0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.429+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb7dc0037a0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.430+0000 7fb7e8c00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 0x7fb7d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.430+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb7dc0cea20 con 0x7fb7ec1016b0 2024-03-28T18:40:35.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.430+0000 7fb7e2a00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 0x7fb7d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.431+0000 7fb7e2a00640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 0x7fb7d00a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7d402c260 tx=0x7fb7d4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: Saving service mds.all spec with placement smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-28T18:40:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:35.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:35.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:35.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:35.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:35 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.434+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7dc09bec0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:40:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: Saving service mds.all spec with placement smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-28T18:40:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:40:35.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:35.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:35.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:35.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:35 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.633+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7ec1a2930 con 0x7fb7ec1016b0 2024-03-28T18:40:35.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.635+0000 7fb7e8c00640 1 -- 172.21.15.119:0/2077220025 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7dc09b7c0 con 0x7fb7ec1016b0 2024-03-28T18:40:35.635 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.636+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 msgr2=0x7fb7d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.636+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 0x7fb7d00a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb7d402c260 tx=0x7fb7d4000bc0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.636+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 msgr2=0x7fb7ec199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.636+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00e9c0 tx=0x7fb7dc00ee80 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 shutdown_connections 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb7d00a2880 0x7fb7d00a4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7ec103260 0x7fb7ec1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7ec1028a0 0x7fb7ec19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 --2- 172.21.15.119:0/2077220025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7ec1016b0 0x7fb7ec199b90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 >> 172.21.15.119:0/2077220025 conn(0x7fb7ec0fcc40 msgr2=0x7fb7ec0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 shutdown_connections 2024-03-28T18:40:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.637+0000 7fb7f2200640 1 -- 172.21.15.119:0/2077220025 wait complete. 2024-03-28T18:40:35.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 -- 172.21.15.119:0/2383122983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 msgr2=0x7f808010bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 --2- 172.21.15.119:0/2383122983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f808010bb30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8078009be0 tx=0x7f807802bef0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 -- 172.21.15.119:0/2383122983 shutdown_connections 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 --2- 172.21.15.119:0/2383122983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f808010bb30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 --2- 172.21.15.119:0/2383122983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801048e0 0x7f8080104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 --2- 172.21.15.119:0/2383122983 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801036f0 0x7f8080103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 -- 172.21.15.119:0/2383122983 >> 172.21.15.119:0/2383122983 conn(0x7f80800fec80 msgr2=0x7f80801010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:35.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 -- 172.21.15.119:0/2383122983 shutdown_connections 2024-03-28T18:40:35.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.874+0000 7f807f200640 1 -- 172.21.15.119:0/2383122983 wait complete. 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 Processor -- start 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 -- start start 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801048e0 0x7f8080199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808010f3c0 con 0x7f80801036f0 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808010f240 con 0x7f80801048e0 2024-03-28T18:40:35.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807f200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808010f540 con 0x7f80801052a0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.875+0000 7f807de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34720/0 (socket says 172.21.15.119:34720) 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f8075400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801048e0 0x7f8080199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59510/0 (socket says 172.21.15.119:59510) 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807de00640 1 -- 172.21.15.119:0/1304846699 learned_addr learned my addr 172.21.15.119:0/1304846699 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 -- 172.21.15.119:0/1304846699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801048e0 msgr2=0x7f8080199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801048e0 0x7f8080199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 -- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 msgr2=0x7f8080199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 -- 172.21.15.119:0/1304846699 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80801a1570 con 0x7f80801052a0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807de00640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f807e800640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f807802c430 tx=0x7f80780337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078043020 con 0x7f80801052a0 2024-03-28T18:40:35.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f807802ca60 con 0x7f80801052a0 2024-03-28T18:40:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.876+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078034690 con 0x7f80801052a0 2024-03-28T18:40:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.877+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80801a1800 con 0x7f80801052a0 2024-03-28T18:40:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.877+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80801a1d00 con 0x7f80801052a0 2024-03-28T18:40:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.878+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808010aa10 con 0x7f80801052a0 2024-03-28T18:40:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.879+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8078046020 con 0x7f80801052a0 2024-03-28T18:40:35.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.880+0000 7f8077400640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 0x7f80640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:35.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.881+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f80780f0b10 con 0x7f80801052a0 2024-03-28T18:40:35.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.881+0000 7f807de00640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 0x7f80640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:35.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.881+0000 7f807de00640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 0x7f80640a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80700058c0 tx=0x7f8070005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:35.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:35.884+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f807804d020 con 0x7f80801052a0 2024-03-28T18:40:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.126+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80800622a0 con 0x7f80801052a0 2024-03-28T18:40:36.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.127+0000 7f8077400640 1 -- 172.21.15.119:0/1304846699 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f80780bdfb0 con 0x7f80801052a0 2024-03-28T18:40:36.128 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:36.129 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T18:40:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 1 2024-03-28T18:40:36.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 msgr2=0x7f80640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:36.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 0x7f80640a4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f80700058c0 tx=0x7f8070005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 msgr2=0x7f80801a0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f807802c430 tx=0x7f80780337b0 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 shutdown_connections 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80640a2880 0x7f80640a4d40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80801052a0 0x7f80801a0f00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80801048e0 0x7f8080199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 --2- 172.21.15.119:0/1304846699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80801036f0 0x7f8080199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.130+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 >> 172.21.15.119:0/1304846699 conn(0x7f80800fec80 msgr2=0x7f8080107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.131+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 shutdown_connections 2024-03-28T18:40:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:36.131+0000 7f807f200640 1 -- 172.21.15.119:0/1304846699 wait complete. 2024-03-28T18:40:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:36 smithi119 ceph-mon[31813]: Deploying daemon mds.l on smithi196 2024-03-28T18:40:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:36 smithi119 ceph-mon[31813]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/998372764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:36.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2077220025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:36.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:36 smithi196 ceph-mon[36300]: Deploying daemon mds.l on smithi196 2024-03-28T18:40:36.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:36 smithi196 ceph-mon[36300]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:36.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:36 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/998372764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:36.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:36 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2077220025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:36.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:36 smithi131 ceph-mon[36065]: Deploying daemon mds.l on smithi196 2024-03-28T18:40:36.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:36 smithi131 ceph-mon[36065]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:36.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:36 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/998372764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:40:36.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:36 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2077220025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:40:37.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:37 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1304846699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:37.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:37 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1304846699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:37.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:37 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1304846699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:37.934 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:40:37 smithi196 systemd[1]: Started Ceph mds.l for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:38.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6832/1244574561,v1:172.21.15.196:6833/1244574561] up:boot 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: fsmap 1 up:standby 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:38 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 -- 172.21.15.119:0/3176368294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 msgr2=0x7f4078101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 --2- 172.21.15.119:0/3176368294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f4078101ae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4068009900 tx=0x7f406802bc30 comp rx=0 tx=0).stop 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 -- 172.21.15.119:0/3176368294 shutdown_connections 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 --2- 172.21.15.119:0/3176368294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4078103270 0x7f4078109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 --2- 172.21.15.119:0/3176368294 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f40781028b0 0x7f4078102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 --2- 172.21.15.119:0/3176368294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f4078101ae0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 -- 172.21.15.119:0/3176368294 >> 172.21.15.119:0/3176368294 conn(0x7f40780fcc50 msgr2=0x7f40780ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 -- 172.21.15.119:0/3176368294 shutdown_connections 2024-03-28T18:40:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.380+0000 7f407dc00640 1 -- 172.21.15.119:0/3176368294 wait complete. 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 Processor -- start 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 -- start start 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40781028b0 0x7f4078197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4078103270 0x7f407819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:38.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407810d310 con 0x7f40781016c0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407810d190 con 0x7f4078103270 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f407dc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407810d490 con 0x7f40781028b0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f4076a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f4076a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34734/0 (socket says 172.21.15.119:34734) 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.381+0000 7f4076a00640 1 -- 172.21.15.119:0/1903313653 learned_addr learned my addr 172.21.15.119:0/1903313653 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076000640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40781028b0 0x7f4078197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4077400640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4078103270 0x7f407819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 -- 172.21.15.119:0/1903313653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40781028b0 msgr2=0x7f4078197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40781028b0 0x7f4078197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 -- 172.21.15.119:0/1903313653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4078103270 msgr2=0x7f407819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4078103270 0x7f407819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 -- 172.21.15.119:0/1903313653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407819f5e0 con 0x7f40781016c0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f4076a00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4068000c00 tx=0x7f4068005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068041070 con 0x7f40781016c0 2024-03-28T18:40:38.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40680044d0 con 0x7f40781016c0 2024-03-28T18:40:38.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407819f870 con 0x7f40781016c0 2024-03-28T18:40:38.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.382+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068034440 con 0x7f40781016c0 2024-03-28T18:40:38.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.383+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407819fd60 con 0x7f40781016c0 2024-03-28T18:40:38.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.384+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4078108a00 con 0x7f40781016c0 2024-03-28T18:40:38.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.386+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4068044070 con 0x7f40781016c0 2024-03-28T18:40:38.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.387+0000 7f406f400640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 0x7f405c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:38.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.387+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f40680f0420 con 0x7f40781016c0 2024-03-28T18:40:38.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.387+0000 7f4076000640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 0x7f405c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:38.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.388+0000 7f4076000640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 0x7f405c0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4078198ed0 tx=0x7f4064005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:38.390 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:38.390 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6832/1244574561,v1:172.21.15.196:6833/1244574561] up:boot 2024-03-28T18:40:38.390 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: fsmap 1 up:standby 2024-03-28T18:40:38.390 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:40:38.390 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.391 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.391 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.391 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:38.391 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:38.391 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:38 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:38.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.395+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f40680bd8c0 con 0x7f40781016c0 2024-03-28T18:40:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6832/1244574561,v1:172.21.15.196:6833/1244574561] up:boot 2024-03-28T18:40:38.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: fsmap 1 up:standby 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:38.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:38 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:38.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.637+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f407802d070 con 0x7f40781016c0 2024-03-28T18:40:38.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.637+0000 7f406f400640 1 -- 172.21.15.119:0/1903313653 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f40680bd1c0 con 0x7f40781016c0 2024-03-28T18:40:38.638 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:38.638 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:38.639 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 2 2024-03-28T18:40:38.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 msgr2=0x7f405c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:38.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 0x7f405c0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4078198ed0 tx=0x7f4064005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 msgr2=0x7f40781979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4068000c00 tx=0x7f4068005d50 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 shutdown_connections 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f405c0a2800 0x7f405c0a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4078103270 0x7f407819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f40781028b0 0x7f4078197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 --2- 172.21.15.119:0/1903313653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40781016c0 0x7f40781979c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 >> 172.21.15.119:0/1903313653 conn(0x7f40780fcc50 msgr2=0x7f4078105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 shutdown_connections 2024-03-28T18:40:38.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:38.640+0000 7f407dc00640 1 -- 172.21.15.119:0/1903313653 wait complete. 2024-03-28T18:40:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:39 smithi119 ceph-mon[31813]: Deploying daemon mds.k on smithi131 2024-03-28T18:40:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:39 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1903313653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:39.407 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:39 smithi131 ceph-mon[36065]: Deploying daemon mds.k on smithi131 2024-03-28T18:40:39.407 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:39 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1903313653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:39.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:39 smithi196 ceph-mon[36300]: Deploying daemon mds.k on smithi131 2024-03-28T18:40:39.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:39 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1903313653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:39.944 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:39 smithi131 systemd[1]: Starting Ceph mds.k for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:40.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:40 smithi119 ceph-mon[31813]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:40.337 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:40 smithi131 ceph-mon[36065]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:40.337 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 podman[65064]: 2024-03-28 18:40:39.983311424 +0000 UTC m=+0.018316662 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:40.337 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 podman[65064]: 2024-03-28 18:40:40.093231486 +0000 UTC m=+0.128236729 container create bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T18:40:40.337 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 podman[65064]: 2024-03-28 18:40:40.338051784 +0000 UTC m=+0.373057022 container init bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T18:40:40.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:40 smithi196 ceph-mon[36300]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:40.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:40.694 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 podman[65064]: 2024-03-28 18:40:40.340854925 +0000 UTC m=+0.375860163 container start bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T18:40:40.695 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 ceph-mds[65083]: -- 172.21.15.131:0/3899768691 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f1dff9a40 con 0x561f1dfccc00 2024-03-28T18:40:40.695 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: starting mds.k at 2024-03-28T18:40:40.695 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 bash[65064]: bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 2024-03-28T18:40:40.695 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:40:40 smithi131 systemd[1]: Started Ceph mds.k for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:40.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 -- 172.21.15.119:0/1789729555 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 msgr2=0x7f2d8010b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1789729555 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d8010b860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009ca0 tx=0x7f2d7802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 -- 172.21.15.119:0/1789729555 shutdown_connections 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1789729555 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d8010b860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1789729555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 0x7f2d80074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1789729555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.853+0000 7f2d7f200640 1 -- 172.21.15.119:0/1789729555 >> 172.21.15.119:0/1789729555 conn(0x7f2d800fca80 msgr2=0x7f2d800feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.854+0000 7f2d7f200640 1 -- 172.21.15.119:0/1789729555 shutdown_connections 2024-03-28T18:40:40.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.854+0000 7f2d7f200640 1 -- 172.21.15.119:0/1789729555 wait complete. 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 Processor -- start 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 -- start start 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 0x7f2d801999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d801a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8010f350 con 0x7f2d800756e0 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8010f1d0 con 0x7f2d801093d0 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d7f200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8010f4d0 con 0x7f2d80074100 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d77e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d77e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34726/0 (socket says 172.21.15.119:34726) 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.855+0000 7f2d77e00640 1 -- 172.21.15.119:0/1055907443 learned_addr learned my addr 172.21.15.119:0/1055907443 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:40.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d7de00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 0x7f2d801999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d7e800640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d801a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 -- 172.21.15.119:0/1055907443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 msgr2=0x7f2d801999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 0x7f2d801999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 -- 172.21.15.119:0/1055907443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 msgr2=0x7f2d801a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d801a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 -- 172.21.15.119:0/1055907443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d801a15c0 con 0x7f2d800756e0 2024-03-28T18:40:40.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d77e00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800e9f0 tx=0x7f2d6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800cd80 con 0x7f2d800756e0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d801a18b0 con 0x7f2d800756e0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d68010910 con 0x7f2d800756e0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.856+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68018990 con 0x7f2d800756e0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.857+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d801a1e00 con 0x7f2d800756e0 2024-03-28T18:40:40.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.858+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d80108100 con 0x7f2d800756e0 2024-03-28T18:40:40.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.861+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2d68010ac0 con 0x7f2d800756e0 2024-03-28T18:40:40.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.862+0000 7f2d76a00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:40.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.862+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2d680ce910 con 0x7f2d800756e0 2024-03-28T18:40:40.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.862+0000 7f2d7de00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:40.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.863+0000 7f2d7de00640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 0x7f2d640a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009bd0 tx=0x7f2d70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:40.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:40.867+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d6809b800 con 0x7f2d800756e0 2024-03-28T18:40:41.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.108+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d8002d2c0 con 0x7f2d800756e0 2024-03-28T18:40:41.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.108+0000 7f2d76a00640 1 -- 172.21.15.119:0/1055907443 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f2d6809f8a0 con 0x7f2d800756e0 2024-03-28T18:40:41.109 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:41.109 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:41.109 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 3 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 msgr2=0x7f2d640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 0x7f2d640a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009bd0 tx=0x7f2d70005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 msgr2=0x7f2d80199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800e9f0 tx=0x7f2d6800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 shutdown_connections 2024-03-28T18:40:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d640a2800 0x7f2d640a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d801093d0 0x7f2d801a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d800756e0 0x7f2d80199ee0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 --2- 172.21.15.119:0/1055907443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d80074100 0x7f2d801999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 >> 172.21.15.119:0/1055907443 conn(0x7f2d800fca80 msgr2=0x7f2d801052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 shutdown_connections 2024-03-28T18:40:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:41.111+0000 7f2d7f200640 1 -- 172.21.15.119:0/1055907443 wait complete. 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:41.767 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: Deploying daemon mds.j on smithi119 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6832/4160748351,v1:172.21.15.131:6833/4160748351] up:boot 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: fsmap 2 up:standby 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:40:41.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:41 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1055907443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:41.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: Deploying daemon mds.j on smithi119 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6832/4160748351,v1:172.21.15.131:6833/4160748351] up:boot 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: fsmap 2 up:standby 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:40:41.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:41 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1055907443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: Deploying daemon mds.j on smithi119 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6832/4160748351,v1:172.21.15.131:6833/4160748351] up:boot 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: fsmap 2 up:standby 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:40:41.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:41 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1055907443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:43.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 -- 172.21.15.119:0/3855469973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc841093d0 msgr2=0x7fdc8410b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 --2- 172.21.15.119:0/3855469973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc841093d0 0x7fdc8410b860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009be0 tx=0x7fdc7c02bef0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 -- 172.21.15.119:0/3855469973 shutdown_connections 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 --2- 172.21.15.119:0/3855469973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc841093d0 0x7fdc8410b860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 --2- 172.21.15.119:0/3855469973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 --2- 172.21.15.119:0/3855469973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc840756e0 0x7fdc84073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 -- 172.21.15.119:0/3855469973 >> 172.21.15.119:0/3855469973 conn(0x7fdc840fca80 msgr2=0x7fdc840feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:43.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 -- 172.21.15.119:0/3855469973 shutdown_connections 2024-03-28T18:40:43.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.360+0000 7fdc8a400640 1 -- 172.21.15.119:0/3855469973 wait complete. 2024-03-28T18:40:43.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 Processor -- start 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 -- start start 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 0x7fdc84199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 0x7fdc841a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8410f3a0 con 0x7fdc84074100 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8410f220 con 0x7fdc840756e0 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.361+0000 7fdc8a400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8410f520 con 0x7fdc841093d0 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34748/0 (socket says 172.21.15.119:34748) 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 -- 172.21.15.119:0/251467075 learned_addr learned my addr 172.21.15.119:0/251467075 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc83e00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 0x7fdc84199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89a00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 0x7fdc841a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 -- 172.21.15.119:0/251467075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 msgr2=0x7fdc841a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:43.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 0x7fdc841a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 -- 172.21.15.119:0/251467075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 msgr2=0x7fdc84199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 0x7fdc84199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 -- 172.21.15.119:0/251467075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc841a1580 con 0x7fdc84074100 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89a00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 0x7fdc841a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc83e00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 0x7fdc84199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.362+0000 7fdc89000640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800e9c0 tx=0x7fdc7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.363+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7800ccc0 con 0x7fdc84074100 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.363+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc78005500 con 0x7fdc84074100 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.363+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc841a1870 con 0x7fdc84074100 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.363+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78010430 con 0x7fdc84074100 2024-03-28T18:40:43.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.363+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc841a1dc0 con 0x7fdc84074100 2024-03-28T18:40:43.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.364+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc84108100 con 0x7fdc84074100 2024-03-28T18:40:43.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.366+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdc780046c0 con 0x7fdc84074100 2024-03-28T18:40:43.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.367+0000 7fdc81600640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 0x7fdc6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:43.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.368+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdc780cea00 con 0x7fdc84074100 2024-03-28T18:40:43.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.368+0000 7fdc83e00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 0x7fdc6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:43.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.368+0000 7fdc83e00640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 0x7fdc6c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc8419ae70 tx=0x7fdc70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:43.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.372+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc7809bea0 con 0x7fdc84074100 2024-03-28T18:40:43.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.618+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc8402d070 con 0x7fdc84074100 2024-03-28T18:40:43.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.619+0000 7fdc81600640 1 -- 172.21.15.119:0/251467075 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7fdc8402d070 con 0x7fdc84074100 2024-03-28T18:40:43.619 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:43.619 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:43.619 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 3 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 msgr2=0x7fdc6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 0x7fdc6c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc8419ae70 tx=0x7fdc70005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 msgr2=0x7fdc84199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc7800e9c0 tx=0x7fdc7800ee80 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 shutdown_connections 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc6c0a2800 0x7fdc6c0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc841093d0 0x7fdc841a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc840756e0 0x7fdc84199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 --2- 172.21.15.119:0/251467075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc84074100 0x7fdc84199960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:43.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 >> 172.21.15.119:0/251467075 conn(0x7fdc840fca80 msgr2=0x7fdc841052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 shutdown_connections 2024-03-28T18:40:43.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:43.621+0000 7fdc8a400640 1 -- 172.21.15.119:0/251467075 wait complete. 2024-03-28T18:40:44.175 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:44 smithi119 ceph-mon[31813]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:44.175 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:44 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/251467075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:44.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:44 smithi196 ceph-mon[36300]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:44.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:44 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/251467075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:44 smithi131 ceph-mon[36065]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:44.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:44 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/251467075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:44.568 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:40:44 smithi119 systemd[1]: Started Ceph mds.j for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:45.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:45.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: Deploying daemon mds.d on smithi119 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6834/2486653544,v1:172.21.15.119:6835/2486653544] up:boot 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: fsmap 3 up:standby 2024-03-28T18:40:45.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:45 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:40:45.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: Deploying daemon mds.d on smithi119 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6834/2486653544,v1:172.21.15.119:6835/2486653544] up:boot 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: fsmap 3 up:standby 2024-03-28T18:40:45.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:45 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: Deploying daemon mds.d on smithi119 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:45.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6834/2486653544,v1:172.21.15.119:6835/2486653544] up:boot 2024-03-28T18:40:45.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: fsmap 3 up:standby 2024-03-28T18:40:45.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:45 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:40:45.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 -- 172.21.15.119:0/2294142606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea81038e0 msgr2=0x7feea8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:45.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 --2- 172.21.15.119:0/2294142606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea81038e0 0x7feea8103d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feea0009be0 tx=0x7feea002bef0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 -- 172.21.15.119:0/2294142606 shutdown_connections 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 --2- 172.21.15.119:0/2294142606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea8105490 0x7feea810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 --2- 172.21.15.119:0/2294142606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8104ad0 0x7feea8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 --2- 172.21.15.119:0/2294142606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea81038e0 0x7feea8103d00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 -- 172.21.15.119:0/2294142606 >> 172.21.15.119:0/2294142606 conn(0x7feea80fee50 msgr2=0x7feea8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 -- 172.21.15.119:0/2294142606 shutdown_connections 2024-03-28T18:40:45.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.840+0000 7feeada00640 1 -- 172.21.15.119:0/2294142606 wait complete. 2024-03-28T18:40:45.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.841+0000 7feeada00640 1 Processor -- start 2024-03-28T18:40:45.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.841+0000 7feeada00640 1 -- start start 2024-03-28T18:40:45.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea8104ad0 0x7feea819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 0x7feea81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea810f5b0 con 0x7feea8105490 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea810f430 con 0x7feea81038e0 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feeada00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea810f730 con 0x7feea8104ad0 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 0x7feea81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea8104ad0 0x7feea819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54406/0 (socket says 172.21.15.119:54406) 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea6a00640 1 -- 172.21.15.119:0/1361297060 learned_addr learned my addr 172.21.15.119:0/1361297060 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea7400640 1 -- 172.21.15.119:0/1361297060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea8104ad0 msgr2=0x7feea819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea7400640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea8104ad0 0x7feea819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.842+0000 7feea7400640 1 -- 172.21.15.119:0/1361297060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 msgr2=0x7feea8199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:45.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feea7400640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feea7400640 1 -- 172.21.15.119:0/1361297060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea81a1830 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feea6a00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feea7400640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 0x7feea81a1190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e970 tx=0x7fee9800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9800ccc0 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee98009e40 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee98004a00 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea81a1b20 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.843+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea81a20c0 con 0x7feea8105490 2024-03-28T18:40:45.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.845+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea810ac00 con 0x7feea8105490 2024-03-28T18:40:45.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.846+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee98004ba0 con 0x7feea8105490 2024-03-28T18:40:45.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.847+0000 7fee9f400640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 0x7fee8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:45.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.848+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fee980ce570 con 0x7feea8105490 2024-03-28T18:40:45.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.848+0000 7feea6a00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 0x7fee8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:45.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.849+0000 7feea6a00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 0x7fee8c0a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feea0006010 tx=0x7feea0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:45.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:45.853+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee9809bab0 con 0x7feea8105490 2024-03-28T18:40:46.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.095+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea802d220 con 0x7feea8105490 2024-03-28T18:40:46.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.096+0000 7fee9f400640 1 -- 172.21.15.119:0/1361297060 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2916 (secure 0 0 0) 0x7fee9809b3b0 con 0x7feea8105490 2024-03-28T18:40:46.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:46.096 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:46.096 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 4 2024-03-28T18:40:46.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.098+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 msgr2=0x7fee8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.098+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 0x7fee8c0a4d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feea0006010 tx=0x7feea0000f40 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.098+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 msgr2=0x7feea81a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.098+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 0x7feea81a1190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e970 tx=0x7fee9800ed70 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 shutdown_connections 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee8c0a2880 0x7fee8c0a4d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea8105490 0x7feea81a1190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea8104ad0 0x7feea819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 --2- 172.21.15.119:0/1361297060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea81038e0 0x7feea8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 >> 172.21.15.119:0/1361297060 conn(0x7feea80fee50 msgr2=0x7feea8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 shutdown_connections 2024-03-28T18:40:46.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:46.099+0000 7feeada00640 1 -- 172.21.15.119:0/1361297060 wait complete. 2024-03-28T18:40:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:46 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1361297060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:46 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1361297060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:46 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1361297060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:47.651 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:47 smithi119 ceph-mon[31813]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:47.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:47 smithi196 ceph-mon[36300]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:47.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:47 smithi131 ceph-mon[36065]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:48.010 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:47 smithi119 systemd[1]: Starting Ceph mds.d for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:48.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:48.315 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 podman[73618]: 2024-03-28 18:40:47.913754316 +0000 UTC m=+0.036212466 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:48.315 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 podman[73618]: 2024-03-28 18:40:48.035385249 +0000 UTC m=+0.157843495 container create 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:40:48.315 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 podman[73618]: 2024-03-28 18:40:48.314378121 +0000 UTC m=+0.436836317 container init 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T18:40:48.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 -- 172.21.15.119:0/3678097814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 msgr2=0x7f759c074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:48.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 --2- 172.21.15.119:0/3678097814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c074580 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7588009930 tx=0x7f758802bc30 comp rx=0 tx=0).stop 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 -- 172.21.15.119:0/3678097814 shutdown_connections 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 --2- 172.21.15.119:0/3678097814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c10dae0 0x7f759c10fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 --2- 172.21.15.119:0/3678097814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c074580 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 --2- 172.21.15.119:0/3678097814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c0756e0 0x7f759c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 -- 172.21.15.119:0/3678097814 >> 172.21.15.119:0/3678097814 conn(0x7f759c0fca80 msgr2=0x7f759c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 -- 172.21.15.119:0/3678097814 shutdown_connections 2024-03-28T18:40:48.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.327+0000 7f75a1800640 1 -- 172.21.15.119:0/3678097814 wait complete. 2024-03-28T18:40:48.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.328+0000 7f75a1800640 1 Processor -- start 2024-03-28T18:40:48.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 -- start start 2024-03-28T18:40:48.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:48.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 0x7f759c1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c1137d0 con 0x7f759c0756e0 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c113650 con 0x7f759c074100 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a1800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c113950 con 0x7f759c10dae0 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f759b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f759be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f759b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34820/0 (socket says 172.21.15.119:34820) 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f759b400640 1 -- 172.21.15.119:0/2802487783 learned_addr learned my addr 172.21.15.119:0/2802487783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.329+0000 7f75a0e00640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 0x7f759c1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 -- 172.21.15.119:0/2802487783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 msgr2=0x7f759c1a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 0x7f759c1a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 -- 172.21.15.119:0/2802487783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 msgr2=0x7f759c19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:48.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 -- 172.21.15.119:0/2802487783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c1a5a10 con 0x7f759c0756e0 2024-03-28T18:40:48.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f75a0e00640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 0x7f759c1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f759b400640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7588000c00 tx=0x7f758802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588043020 con 0x7f759c0756e0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.330+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c1a5ca0 con 0x7f759c0756e0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.331+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75880054e0 con 0x7f759c0756e0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.331+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588003ce0 con 0x7f759c0756e0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.331+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c1a61c0 con 0x7f759c0756e0 2024-03-28T18:40:48.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.332+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c103ba0 con 0x7f759c0756e0 2024-03-28T18:40:48.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.334+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7588046020 con 0x7f759c0756e0 2024-03-28T18:40:48.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.336+0000 7f7598c00640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:48.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.336+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f75880f0ab0 con 0x7f759c0756e0 2024-03-28T18:40:48.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.336+0000 7f759be00640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:48.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.337+0000 7f759be00640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 0x7f75800a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f758c009bd0 tx=0x7f758c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:48.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.339+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f758804d020 con 0x7f759c0756e0 2024-03-28T18:40:48.568 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 podman[73618]: 2024-03-28 18:40:48.327325418 +0000 UTC m=+0.449783621 container start 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T18:40:48.569 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 ceph-mds[73667]: -- 172.21.15.119:0/1236560427 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55828c551a40 con 0x55828c524c00 2024-03-28T18:40:48.569 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d[73650]: starting mds.d at 2024-03-28T18:40:48.569 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 bash[73618]: 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda 2024-03-28T18:40:48.569 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:40:48 smithi119 systemd[1]: Started Ceph mds.d for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:48.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.589+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f759c02d070 con 0x7f759c0756e0 2024-03-28T18:40:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.594+0000 7f7598c00640 1 -- 172.21.15.119:0/2802487783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7f75880bdf50 con 0x7f759c0756e0 2024-03-28T18:40:48.594 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:48.594 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 5 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 msgr2=0x7f75800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 0x7f75800a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f758c009bd0 tx=0x7f758c005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 msgr2=0x7f759c19e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7588000c00 tx=0x7f758802c6c0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 shutdown_connections 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75800a2800 0x7f75800a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f759c10dae0 0x7f759c1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f759c0756e0 0x7f759c19e330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 --2- 172.21.15.119:0/2802487783 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f759c074100 0x7f759c19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:48.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 >> 172.21.15.119:0/2802487783 conn(0x7f759c0fca80 msgr2=0x7f759c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:48.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 shutdown_connections 2024-03-28T18:40:48.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:48.597+0000 7f75a1800640 1 -- 172.21.15.119:0/2802487783 wait complete. 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6836/2841831538,v1:172.21.15.119:6837/2841831538] up:boot 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: fsmap 4 up:standby 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:49.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:49.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2802487783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:49.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:49.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: Deploying daemon mds.e on smithi131 2024-03-28T18:40:49.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:49 smithi119 ceph-mon[31813]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6836/2841831538,v1:172.21.15.119:6837/2841831538] up:boot 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: fsmap 4 up:standby 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2802487783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:49.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:49.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: Deploying daemon mds.e on smithi131 2024-03-28T18:40:49.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:49 smithi196 ceph-mon[36300]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6836/2841831538,v1:172.21.15.119:6837/2841831538] up:boot 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: fsmap 4 up:standby 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2802487783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: Deploying daemon mds.e on smithi131 2024-03-28T18:40:49.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:49 smithi131 ceph-mon[36065]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:50.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 -- 172.21.15.119:0/2673197868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 msgr2=0x7f7fb810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 --2- 172.21.15.119:0/2673197868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb810ff40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009ca0 tx=0x7f7fb002c100 comp rx=0 tx=0).stop 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 -- 172.21.15.119:0/2673197868 shutdown_connections 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 --2- 172.21.15.119:0/2673197868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb810ff40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 --2- 172.21.15.119:0/2673197868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb8109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 --2- 172.21.15.119:0/2673197868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb8107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 -- 172.21.15.119:0/2673197868 >> 172.21.15.119:0/2673197868 conn(0x7f7fb8076ab0 msgr2=0x7f7fb8078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:50.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 -- 172.21.15.119:0/2673197868 shutdown_connections 2024-03-28T18:40:50.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.836+0000 7f7fbd200640 1 -- 172.21.15.119:0/2673197868 wait complete. 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 Processor -- start 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 -- start start 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb81a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113780 con 0x7f7fb8107b00 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113600 con 0x7f7fb81096b0 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.837+0000 7f7fbd200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113900 con 0x7f7fb8108cf0 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42282/0 (socket says 172.21.15.119:42282) 2024-03-28T18:40:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 -- 172.21.15.119:0/1734273371 learned_addr learned my addr 172.21.15.119:0/1734273371 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7e00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb81a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7faea00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 -- 172.21.15.119:0/1734273371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 msgr2=0x7f7fb819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 -- 172.21.15.119:0/1734273371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 msgr2=0x7f7fb81a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb81a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 -- 172.21.15.119:0/1734273371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb81a5980 con 0x7f7fb8107b00 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7faea00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.838+0000 7f7fb7400640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800e9c0 tx=0x7f7fa800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.839+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa800cd80 con 0x7f7fb8107b00 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.839+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa80035f0 con 0x7f7fb8107b00 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.839+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb81a5c70 con 0x7f7fb8107b00 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.839+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8010690 con 0x7f7fb8107b00 2024-03-28T18:40:50.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.839+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81a61c0 con 0x7f7fb8107b00 2024-03-28T18:40:50.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.840+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb810edc0 con 0x7f7fb8107b00 2024-03-28T18:40:50.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.842+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7fa80037a0 con 0x7f7fb8107b00 2024-03-28T18:40:50.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.843+0000 7f7fb4c00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:50.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.844+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7fa80ceac0 con 0x7f7fb8107b00 2024-03-28T18:40:50.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.844+0000 7f7faea00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.844+0000 7f7faea00640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7fb819f270 tx=0x7f7fa0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:50.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:50.848+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7fa809bf60 con 0x7f7fb8107b00 2024-03-28T18:40:50.944 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:50 smithi131 systemd[1]: Starting Ceph mds.e for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:51.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.088+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fb802d070 con 0x7f7fb8107b00 2024-03-28T18:40:51.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.089+0000 7f7fb4c00640 1 -- 172.21.15.119:0/1734273371 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7f7fb802d070 con 0x7f7fb8107b00 2024-03-28T18:40:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:51.089 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T18:40:51.090 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 5 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 msgr2=0x7f7f9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7fb819f270 tx=0x7f7fa0005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 msgr2=0x7f7fb819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800e9c0 tx=0x7f7fa800ee80 comp rx=0 tx=0).stop 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 shutdown_connections 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7f9c0a2800 0x7f7f9c0a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb81096b0 0x7f7fb81a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8108cf0 0x7f7fb819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 --2- 172.21.15.119:0/1734273371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8107b00 0x7f7fb819dd60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:51.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 >> 172.21.15.119:0/1734273371 conn(0x7f7fb8076ab0 msgr2=0x7f7fb810bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:51.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 shutdown_connections 2024-03-28T18:40:51.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:51.092+0000 7f7fbd200640 1 -- 172.21.15.119:0/1734273371 wait complete. 2024-03-28T18:40:51.254 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:50 smithi131 podman[65386]: 2024-03-28 18:40:50.888097973 +0000 UTC m=+0.017810809 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:51.254 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 podman[65386]: 2024-03-28 18:40:51.008393682 +0000 UTC m=+0.138106546 container create 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T18:40:51.254 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 podman[65386]: 2024-03-28 18:40:51.253053522 +0000 UTC m=+0.382766345 container init 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-28T18:40:51.695 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 podman[65386]: 2024-03-28 18:40:51.255710101 +0000 UTC m=+0.385422932 container start 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T18:40:51.695 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 ceph-mds[65405]: -- 172.21.15.131:0/203137817 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558825303a40 con 0x5588252d6800 2024-03-28T18:40:51.695 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: starting mds.e at 2024-03-28T18:40:51.695 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 bash[65386]: 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 2024-03-28T18:40:51.695 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:40:51 smithi131 systemd[1]: Started Ceph mds.e for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1734273371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:52.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1734273371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:52.370 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:52.371 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:52.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:52.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1734273371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:52.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:52.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:53.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:53.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:53 smithi196 ceph-mon[36300]: Deploying daemon mds.f on smithi196 2024-03-28T18:40:53.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:53 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6834/1538863538,v1:172.21.15.131:6835/1538863538] up:boot 2024-03-28T18:40:53.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:53 smithi196 ceph-mon[36300]: fsmap 5 up:standby 2024-03-28T18:40:53.300 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:40:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:53 smithi119 ceph-mon[31813]: Deploying daemon mds.f on smithi196 2024-03-28T18:40:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:53 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6834/1538863538,v1:172.21.15.131:6835/1538863538] up:boot 2024-03-28T18:40:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:53 smithi119 ceph-mon[31813]: fsmap 5 up:standby 2024-03-28T18:40:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:40:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 -- 172.21.15.119:0/3625254686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 msgr2=0x7fbb58109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/3625254686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb58109170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009ca0 tx=0x7fbb4002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 -- 172.21.15.119:0/3625254686 shutdown_connections 2024-03-28T18:40:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/3625254686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb581096b0 0x7fbb5810ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/3625254686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb58109170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/3625254686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb58107b00 0x7fbb58107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 -- 172.21.15.119:0/3625254686 >> 172.21.15.119:0/3625254686 conn(0x7fbb58076ab0 msgr2=0x7fbb58078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 -- 172.21.15.119:0/3625254686 shutdown_connections 2024-03-28T18:40:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.329+0000 7fbb5ca00640 1 -- 172.21.15.119:0/3625254686 wait complete. 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.330+0000 7fbb5ca00640 1 Processor -- start 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.330+0000 7fbb5ca00640 1 -- start start 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb5819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb581096b0 0x7fbb581a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58113780 con 0x7fbb58107b00 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58113600 con 0x7fbb58108cf0 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb5ca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58113900 con 0x7fbb581096b0 2024-03-28T18:40:53.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb55e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb5819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42302/0 (socket says 172.21.15.119:42302) 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 -- 172.21.15.119:0/1594346568 learned_addr learned my addr 172.21.15.119:0/1594346568 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb57200640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb581096b0 0x7fbb581a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 -- 172.21.15.119:0/1594346568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb581096b0 msgr2=0x7fbb581a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb581096b0 0x7fbb581a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 -- 172.21.15.119:0/1594346568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 msgr2=0x7fbb5819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb5819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb56800640 1 -- 172.21.15.119:0/1594346568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb581a5a40 con 0x7fbb58107b00 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.331+0000 7fbb55e00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb5819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb56800640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e9c0 tx=0x7fbb4800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4800cd80 con 0x7fbb58107b00 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb581a5d30 con 0x7fbb58107b00 2024-03-28T18:40:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb480035f0 con 0x7fbb58107b00 2024-03-28T18:40:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48010690 con 0x7fbb58107b00 2024-03-28T18:40:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.332+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb581a61a0 con 0x7fbb58107b00 2024-03-28T18:40:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.333+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5810eea0 con 0x7fbb58107b00 2024-03-28T18:40:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.335+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbb480037a0 con 0x7fbb58107b00 2024-03-28T18:40:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.336+0000 7fbb4f400640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.336+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbb480ce9c0 con 0x7fbb58107b00 2024-03-28T18:40:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.337+0000 7fbb55e00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.337+0000 7fbb55e00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbb40006010 tx=0x7fbb40032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:53.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.341+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbb4809be60 con 0x7fbb58107b00 2024-03-28T18:40:53.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:53 smithi131 ceph-mon[36065]: Deploying daemon mds.f on smithi196 2024-03-28T18:40:53.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:53 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6834/1538863538,v1:172.21.15.131:6835/1538863538] up:boot 2024-03-28T18:40:53.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:53 smithi131 ceph-mon[36065]: fsmap 5 up:standby 2024-03-28T18:40:53.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:40:53.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.580+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb5802d070 con 0x7fbb58107b00 2024-03-28T18:40:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.582+0000 7fbb4f400640 1 -- 172.21.15.119:0/1594346568 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4504 (secure 0 0 0) 0x7fbb4809b760 con 0x7fbb58107b00 2024-03-28T18:40:53.582 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:53.582 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T18:40:53.582 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 6 2024-03-28T18:40:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.584+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 msgr2=0x7fbb3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.584+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbb40006010 tx=0x7fbb40032590 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.584+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 msgr2=0x7fbb5819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.584+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e9c0 tx=0x7fbb4800ee80 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 shutdown_connections 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb581096b0 0x7fbb581a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb58108cf0 0x7fbb5819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 --2- 172.21.15.119:0/1594346568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb58107b00 0x7fbb5819de20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 >> 172.21.15.119:0/1594346568 conn(0x7fbb58076ab0 msgr2=0x7fbb5810c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 shutdown_connections 2024-03-28T18:40:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:53.585+0000 7fbb5ca00640 1 -- 172.21.15.119:0/1594346568 wait complete. 2024-03-28T18:40:53.935 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:53 smithi196 systemd[1]: Starting Ceph mds.f for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:54.296 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:54 smithi196 ceph-mon[36300]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:54.296 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:54 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1594346568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:54.297 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:53 smithi196 podman[66315]: 2024-03-28 18:40:53.898547293 +0000 UTC m=+0.017764662 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:54.297 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 podman[66315]: 2024-03-28 18:40:54.03659207 +0000 UTC m=+0.155809414 container create dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-28T18:40:54.297 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 podman[66315]: 2024-03-28 18:40:54.297319093 +0000 UTC m=+0.416536458 container init dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:40:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:54 smithi119 ceph-mon[31813]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:54 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1594346568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:54.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:54 smithi131 ceph-mon[36065]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:54.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:54 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1594346568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:54.685 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 podman[66315]: 2024-03-28 18:40:54.299912201 +0000 UTC m=+0.419129560 container start dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:40:54.685 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 ceph-mds[66334]: -- 172.21.15.196:0/3098212048 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f0dec75a40 con 0x55f0dec48c00 2024-03-28T18:40:54.685 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: starting mds.f at 2024-03-28T18:40:54.685 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 bash[66315]: dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae 2024-03-28T18:40:54.685 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:40:54 smithi196 systemd[1]: Started Ceph mds.f for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:55.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: Deploying daemon mds.c on smithi196 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:boot 2024-03-28T18:40:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: fsmap 6 up:standby 2024-03-28T18:40:55.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:55 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 -- 172.21.15.119:0/3179103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8574102170 msgr2=0x7f8574108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 --2- 172.21.15.119:0/3179103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8574102170 0x7f8574108a00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f856c009900 tx=0x7f856c02bc40 comp rx=0 tx=0).stop 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 -- 172.21.15.119:0/3179103000 shutdown_connections 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 --2- 172.21.15.119:0/3179103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8574102170 0x7f8574108a00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 --2- 172.21.15.119:0/3179103000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f85741017b0 0x7f8574101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 --2- 172.21.15.119:0/3179103000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741005c0 0x7f85741009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 -- 172.21.15.119:0/3179103000 >> 172.21.15.119:0/3179103000 conn(0x7f85740fbb30 msgr2=0x7f85740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 -- 172.21.15.119:0/3179103000 shutdown_connections 2024-03-28T18:40:55.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.827+0000 7f857b000640 1 -- 172.21.15.119:0/3179103000 wait complete. 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 Processor -- start 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 -- start start 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 0x7f8574197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8574102170 0x7f857419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857410c220 con 0x7f85741005c0 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.828+0000 7f857b000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857410c0a0 con 0x7f8574102170 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f857b000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857410c3a0 con 0x7f85741017b0 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42314/0 (socket says 172.21.15.119:42314) 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 -- 172.21.15.119:0/784975202 learned_addr learned my addr 172.21.15.119:0/784975202 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573400640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 0x7f8574197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8579000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8574102170 0x7f857419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 -- 172.21.15.119:0/784975202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 msgr2=0x7f8574197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 0x7f8574197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 -- 172.21.15.119:0/784975202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8574102170 msgr2=0x7f857419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8574102170 0x7f857419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 -- 172.21.15.119:0/784975202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857419f5d0 con 0x7f85741005c0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573400640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 0x7f8574197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.829+0000 7f8573e00640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f856400e9c0 tx=0x7f856400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.830+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856400cd80 con 0x7f85741005c0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.830+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85640035f0 con 0x7f85741005c0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.830+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857419f8c0 con 0x7f85741005c0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.830+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564010690 con 0x7f85741005c0 2024-03-28T18:40:55.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.830+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857419fe60 con 0x7f85741005c0 2024-03-28T18:40:55.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.831+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8574068490 con 0x7f85741005c0 2024-03-28T18:40:55.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.833+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f85640037a0 con 0x7f85741005c0 2024-03-28T18:40:55.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.834+0000 7f8570c00640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 0x7f85580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:55.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.834+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85640ce990 con 0x7f85741005c0 2024-03-28T18:40:55.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.835+0000 7f8573400640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 0x7f85580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:55.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.835+0000 7f8573400640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 0x7f85580a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8574198ec0 tx=0x7f8560009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:55.837 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.837 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.837 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.837 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:55.837 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: Deploying daemon mds.c on smithi196 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:boot 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: fsmap 6 up:standby 2024-03-28T18:40:55.838 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:55 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:40:55.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:55.839+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f856409be30 con 0x7f85741005c0 2024-03-28T18:40:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: Deploying daemon mds.c on smithi196 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:boot 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: fsmap 6 up:standby 2024-03-28T18:40:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:55 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:40:56.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.075+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85741a00f0 con 0x7f85741005c0 2024-03-28T18:40:56.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.076+0000 7f8570c00640 1 -- 172.21.15.119:0/784975202 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5295 (secure 0 0 0) 0x7f856409b730 con 0x7f85741005c0 2024-03-28T18:40:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:56.076 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T18:40:56.076 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 7 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 msgr2=0x7f85580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 0x7f85580a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8574198ec0 tx=0x7f8560009500 comp rx=0 tx=0).stop 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 msgr2=0x7f85741979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f856400e9c0 tx=0x7f856400ee80 comp rx=0 tx=0).stop 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 shutdown_connections 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85580a2880 0x7f85580a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8574102170 0x7f857419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:56.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85741017b0 0x7f8574197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:56.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 --2- 172.21.15.119:0/784975202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85741005c0 0x7f85741979b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:56.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.078+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 >> 172.21.15.119:0/784975202 conn(0x7f85740fbb30 msgr2=0x7f85740fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:56.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.079+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 shutdown_connections 2024-03-28T18:40:56.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:56.079+0000 7f857b000640 1 -- 172.21.15.119:0/784975202 wait complete. 2024-03-28T18:40:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:56 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/784975202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:56.839 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:56 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/784975202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:56.839 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:56 smithi196 systemd[1]: Starting Ceph mds.c for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:56 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/784975202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:57.135 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:56 smithi196 podman[66625]: 2024-03-28 18:40:56.742064172 +0000 UTC m=+0.017709451 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:57.135 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:56 smithi196 podman[66625]: 2024-03-28 18:40:56.861251401 +0000 UTC m=+0.136896700 container create 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:40:57.135 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 podman[66625]: 2024-03-28 18:40:57.13533979 +0000 UTC m=+0.410985070 container init 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T18:40:57.434 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 podman[66625]: 2024-03-28 18:40:57.137968182 +0000 UTC m=+0.413613463 container start 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:40:57.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mds[66645]: -- 172.21.15.196:0/65671202 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557cf1cafa40 con 0x557cf1c82800 2024-03-28T18:40:57.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: starting mds.c at 2024-03-28T18:40:57.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 bash[66625]: 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 2024-03-28T18:40:57.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:40:57 smithi196 systemd[1]: Started Ceph mds.c for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:40:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:57.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:57 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:57.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:57 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:57.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:57.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:40:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:40:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:40:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:57 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:40:58.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:40:58.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 -- 172.21.15.119:0/3923936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 msgr2=0x7f475410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:58.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 --2- 172.21.15.119:0/3923936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f475410bd20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f474c009ca0 tx=0x7f474c02c100 comp rx=0 tx=0).stop 2024-03-28T18:40:58.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 -- 172.21.15.119:0/3923936 shutdown_connections 2024-03-28T18:40:58.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 --2- 172.21.15.119:0/3923936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f475410bd20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 --2- 172.21.15.119:0/3923936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4754104ad0 0x7f4754104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 --2- 172.21.15.119:0/3923936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47541038e0 0x7f4754103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 -- 172.21.15.119:0/3923936 >> 172.21.15.119:0/3923936 conn(0x7f47540fee50 msgr2=0x7f4754101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 -- 172.21.15.119:0/3923936 shutdown_connections 2024-03-28T18:40:58.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.320+0000 7f4759c00640 1 -- 172.21.15.119:0/3923936 wait complete. 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.321+0000 7f4759c00640 1 Processor -- start 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.321+0000 7f4759c00640 1 -- start start 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.321+0000 7f4759c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47541038e0 0x7f4754199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4759c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754104ad0 0x7f475419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4759c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4759c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475410f560 con 0x7f4754105490 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4759c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475410f3e0 con 0x7f47541038e0 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4759c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475410f6e0 con 0x7f4754104ad0 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42324/0 (socket says 172.21.15.119:42324) 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 -- 172.21.15.119:0/745344727 learned_addr learned my addr 172.21.15.119:0/745344727 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4752a00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47541038e0 0x7f4754199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:58.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f474a000640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754104ad0 0x7f475419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 -- 172.21.15.119:0/745344727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754104ad0 msgr2=0x7f475419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754104ad0 0x7f475419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 -- 172.21.15.119:0/745344727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47541038e0 msgr2=0x7f4754199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47541038e0 0x7f4754199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.322+0000 7f4753400640 1 -- 172.21.15.119:0/745344727 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47541a17a0 con 0x7f4754105490 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f4753400640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f474c02c640 tx=0x7f474c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c039040 con 0x7f4754105490 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f474c044070 con 0x7f4754105490 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47541a1a30 con 0x7f4754105490 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c004460 con 0x7f4754105490 2024-03-28T18:40:58.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.323+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47541a1f20 con 0x7f4754105490 2024-03-28T18:40:58.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.324+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475410ac00 con 0x7f4754105490 2024-03-28T18:40:58.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.326+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f474c004600 con 0x7f4754105490 2024-03-28T18:40:58.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.328+0000 7f474be00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 0x7f47380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:40:58.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.328+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f474c0f0bf0 con 0x7f4754105490 2024-03-28T18:40:58.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.328+0000 7f4752a00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 0x7f47380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:40:58.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.329+0000 7f4752a00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 0x7f47380a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4754104930 tx=0x7f4744005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:40:58.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.332+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f474c0be130 con 0x7f4754105490 2024-03-28T18:40:58.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.565+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f475402d290 con 0x7f4754105490 2024-03-28T18:40:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.566+0000 7f474be00640 1 -- 172.21.15.119:0/745344727 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6089 (secure 0 0 0) 0x7f474c0098b0 con 0x7f4754105490 2024-03-28T18:40:58.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:40:58.566 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:40:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 8 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 msgr2=0x7f47380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 0x7f47380a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4754104930 tx=0x7f4744005f90 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 msgr2=0x7f47541a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f474c02c640 tx=0x7f474c035d10 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 shutdown_connections 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f47380a2800 0x7f47380a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4754105490 0x7f47541a1130 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4754104ad0 0x7f475419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 --2- 172.21.15.119:0/745344727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47541038e0 0x7f4754199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:40:58.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 >> 172.21.15.119:0/745344727 conn(0x7f47540fee50 msgr2=0x7f4754107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:40:58.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 shutdown_connections 2024-03-28T18:40:58.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:40:58.569+0000 7f4759c00640 1 -- 172.21.15.119:0/745344727 wait complete. 2024-03-28T18:40:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:58 smithi119 ceph-mon[31813]: Deploying daemon mds.h on smithi131 2024-03-28T18:40:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:58 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:boot 2024-03-28T18:40:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:58 smithi119 ceph-mon[31813]: fsmap 7 up:standby 2024-03-28T18:40:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:58 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:40:58.852 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:58 smithi131 ceph-mon[36065]: Deploying daemon mds.h on smithi131 2024-03-28T18:40:58.853 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:58 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:boot 2024-03-28T18:40:58.853 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:58 smithi131 ceph-mon[36065]: fsmap 7 up:standby 2024-03-28T18:40:58.853 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:58 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:40:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:58 smithi196 ceph-mon[36300]: Deploying daemon mds.h on smithi131 2024-03-28T18:40:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:58 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:boot 2024-03-28T18:40:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:58 smithi196 ceph-mon[36300]: fsmap 7 up:standby 2024-03-28T18:40:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:58 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:40:59.444 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 systemd[1]: Starting Ceph mds.h for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:40:59.791 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/745344727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:59.792 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:40:59 smithi131 ceph-mon[36065]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:59.792 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 podman[65699]: 2024-03-28 18:40:59.434577548 +0000 UTC m=+0.017362372 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:40:59.792 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 podman[65699]: 2024-03-28 18:40:59.547594383 +0000 UTC m=+0.130379208 container create c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:40:59.792 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 podman[65699]: 2024-03-28 18:40:59.792541229 +0000 UTC m=+0.375326056 container init c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:40:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/745344727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:40:59 smithi119 ceph-mon[31813]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:40:59.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/745344727' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:40:59.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:40:59 smithi196 ceph-mon[36300]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:00.195 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 podman[65699]: 2024-03-28 18:40:59.795221031 +0000 UTC m=+0.378005852 container start c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:41:00.195 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 ceph-mds[65719]: -- 172.21.15.131:0/990357316 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555fed649a40 con 0x555fed61d000 2024-03-28T18:41:00.195 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: starting mds.h at 2024-03-28T18:41:00.195 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 bash[65699]: c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 2024-03-28T18:41:00.195 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:40:59 smithi131 systemd[1]: Started Ceph mds.h for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:41:00.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 -- 172.21.15.119:0/3208307912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4102140 msgr2=0x7f92b4104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 --2- 172.21.15.119:0/3208307912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4102140 0x7f92b4104530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009ca0 tx=0x7f92ac02c100 comp rx=0 tx=0).stop 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 -- 172.21.15.119:0/3208307912 shutdown_connections 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 --2- 172.21.15.119:0/3208307912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 --2- 172.21.15.119:0/3208307912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4104a70 0x7f92b4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 --2- 172.21.15.119:0/3208307912 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4102140 0x7f92b4104530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 -- 172.21.15.119:0/3208307912 >> 172.21.15.119:0/3208307912 conn(0x7f92b40fbb30 msgr2=0x7f92b40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:00.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 -- 172.21.15.119:0/3208307912 shutdown_connections 2024-03-28T18:41:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.808+0000 7f92bb000640 1 -- 172.21.15.119:0/3208307912 wait complete. 2024-03-28T18:41:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 Processor -- start 2024-03-28T18:41:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 -- start start 2024-03-28T18:41:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 0x7f92b4199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 0x7f92b419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b410d350 con 0x7f92b41073a0 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b410d1d0 con 0x7f92b4104a70 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.809+0000 7f92bb000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b410d4d0 con 0x7f92b4102140 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 0x7f92b4199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 0x7f92b419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 0x7f92b4199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:57060/0 (socket says 172.21.15.119:57060) 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3e00640 1 -- 172.21.15.119:0/1537968836 learned_addr learned my addr 172.21.15.119:0/1537968836 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b9000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 -- 172.21.15.119:0/1537968836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 msgr2=0x7f92b4199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 0x7f92b4199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 -- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 msgr2=0x7f92b41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 -- 172.21.15.119:0/1537968836 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b41a1800 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b9000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b3400640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 0x7f92b419a120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f929c00e9c0 tx=0x7f929c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.810+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c00ccc0 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.811+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f929c0035f0 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.811+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b41a1af0 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.811+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c00ce60 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.811+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b41a2040 con 0x7f92b4104a70 2024-03-28T18:41:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.812+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b4108ef0 con 0x7f92b4104a70 2024-03-28T18:41:00.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.814+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f929c0037a0 con 0x7f92b4104a70 2024-03-28T18:41:00.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.816+0000 7f92b0c00640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 0x7f92980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.816+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f929c0ce990 con 0x7f92b4104a70 2024-03-28T18:41:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.816+0000 7f92b3e00640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 0x7f92980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:00.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.817+0000 7f92b3e00640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 0x7f92980a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92ac006010 tx=0x7f92ac004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:00.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:00.817+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f929c09bec0 con 0x7f92b4104a70 2024-03-28T18:41:01.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.058+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92b402d2c0 con 0x7f92b4104a70 2024-03-28T18:41:01.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.059+0000 7f92b0c00640 1 -- 172.21.15.119:0/1537968836 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6883 (secure 0 0 0) 0x7f929c09b7c0 con 0x7f92b4104a70 2024-03-28T18:41:01.059 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:01.059 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T18:41:01.059 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-03-28T18:41:01.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.061+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 msgr2=0x7f92980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.061+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 0x7f92980a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92ac006010 tx=0x7f92ac004a40 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.061+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 msgr2=0x7f92b419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.061+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 0x7f92b419a120 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f929c00e9c0 tx=0x7f929c00edc0 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 shutdown_connections 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f92980a2800 0x7f92980a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92b41073a0 0x7f92b41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f92b4104a70 0x7f92b419a120 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 --2- 172.21.15.119:0/1537968836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b4102140 0x7f92b4199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 >> 172.21.15.119:0/1537968836 conn(0x7f92b40fbb30 msgr2=0x7f92b4101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 shutdown_connections 2024-03-28T18:41:01.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:01.062+0000 7f92bb000640 1 -- 172.21.15.119:0/1537968836 wait complete. 2024-03-28T18:41:01.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: Deploying daemon mds.i on smithi196 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:boot 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: fsmap 8 up:standby 2024-03-28T18:41:01.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:00 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:41:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: Deploying daemon mds.i on smithi196 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:boot 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: fsmap 8 up:standby 2024-03-28T18:41:01.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:00 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:41:01.358 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: Deploying daemon mds.i on smithi196 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:boot 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: fsmap 8 up:standby 2024-03-28T18:41:01.359 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:00 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:41:02.008 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:01 smithi196 ceph-mon[36300]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:02.008 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:01 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1537968836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:01 smithi119 ceph-mon[31813]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:01 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1537968836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:02.412 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 systemd[1]: Starting Ceph mds.i for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:41:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:01 smithi131 ceph-mon[36065]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:01 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1537968836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:02.685 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 podman[66939]: 2024-03-28 18:41:02.314809847 +0000 UTC m=+0.017792509 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:41:02.685 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 podman[66939]: 2024-03-28 18:41:02.44643883 +0000 UTC m=+0.149421495 container create 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:41:03.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 podman[66939]: 2024-03-28 18:41:02.697324431 +0000 UTC m=+0.400307090 container init 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 podman[66939]: 2024-03-28 18:41:02.700097034 +0000 UTC m=+0.403079681 container start 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 ceph-mds[66958]: -- 172.21.15.196:0/3064481209 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c320a7a40 con 0x558c3207ac00 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: starting mds.i at 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 bash[66939]: 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e 2024-03-28T18:41:03.185 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:41:02 smithi196 systemd[1]: Started Ceph mds.i for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:41:03.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 -- 172.21.15.119:0/2180826321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1081016c0 msgr2=0x7fd108101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 --2- 172.21.15.119:0/2180826321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1081016c0 0x7fd108101ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009900 tx=0x7fd0f802bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 -- 172.21.15.119:0/2180826321 shutdown_connections 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 --2- 172.21.15.119:0/2180826321 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd108103270 0x7fd108109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 --2- 172.21.15.119:0/2180826321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 --2- 172.21.15.119:0/2180826321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1081016c0 0x7fd108101ae0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.280+0000 7fd10e200640 1 -- 172.21.15.119:0/2180826321 >> 172.21.15.119:0/2180826321 conn(0x7fd1080fcc50 msgr2=0x7fd1080ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.281+0000 7fd10e200640 1 -- 172.21.15.119:0/2180826321 shutdown_connections 2024-03-28T18:41:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.281+0000 7fd10e200640 1 -- 172.21.15.119:0/2180826321 wait complete. 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.281+0000 7fd10e200640 1 Processor -- start 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 -- start start 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1081016c0 0x7fd108075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 0x7fd108073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810d360 con 0x7fd1081028b0 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810d1e0 con 0x7fd1081016c0 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd10e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810d4e0 con 0x7fd108103270 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd106a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd106a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54100/0 (socket says 172.21.15.119:54100) 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd106a00640 1 -- 172.21.15.119:0/804363063 learned_addr learned my addr 172.21.15.119:0/804363063 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd107400640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1081016c0 0x7fd108075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.282+0000 7fd107e00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 0x7fd108073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 -- 172.21.15.119:0/804363063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1081016c0 msgr2=0x7fd108075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1081016c0 0x7fd108075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 -- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 msgr2=0x7fd108073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 -- 172.21.15.119:0/804363063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd108074860 con 0x7fd108103270 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd106a00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:03.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd107e00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 0x7fd108073fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd10000da10 tx=0x7fd10000ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10000bd70 con 0x7fd108103270 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd100003e50 con 0x7fd108103270 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000106a0 con 0x7fd108103270 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.283+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1081a1b20 con 0x7fd108103270 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.284+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1081a2070 con 0x7fd108103270 2024-03-28T18:41:03.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.285+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1081089e0 con 0x7fd108103270 2024-03-28T18:41:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.286+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd100045be0 con 0x7fd108103270 2024-03-28T18:41:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.288+0000 7fd0ffe00640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 0x7fd0ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.288+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd1000ce9d0 con 0x7fd108103270 2024-03-28T18:41:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.288+0000 7fd107400640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 0x7fd0ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.289+0000 7fd107400640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 0x7fd0ec0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd0f802c170 tx=0x7fd0f802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.290+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd10009bef0 con 0x7fd108103270 2024-03-28T18:41:03.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.532+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd10802d2c0 con 0x7fd108103270 2024-03-28T18:41:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.533+0000 7fd0ffe00640 1 -- 172.21.15.119:0/804363063 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7679 (secure 0 0 0) 0x7fd10009b7f0 con 0x7fd108103270 2024-03-28T18:41:03.533 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:03.533 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:41:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 10 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 msgr2=0x7fd0ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 0x7fd0ec0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd0f802c170 tx=0x7fd0f802df90 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 msgr2=0x7fd108073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 0x7fd108073fd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd10000da10 tx=0x7fd10000ded0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 shutdown_connections 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ec0a2880 0x7fd0ec0a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd108103270 0x7fd108073fd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1081028b0 0x7fd108073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 --2- 172.21.15.119:0/804363063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd1081016c0 0x7fd108075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 >> 172.21.15.119:0/804363063 conn(0x7fd1080fcc50 msgr2=0x7fd108105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 shutdown_connections 2024-03-28T18:41:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:03.536+0000 7fd10e200640 1 -- 172.21.15.119:0/804363063 wait complete. 2024-03-28T18:41:04.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: Deploying daemon mds.g on smithi119 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:boot 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: fsmap 9 up:standby 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:41:04.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:03 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/804363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: Deploying daemon mds.g on smithi119 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:boot 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: fsmap 9 up:standby 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:41:04.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:03 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/804363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:04.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: Deploying daemon mds.g on smithi119 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:boot 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: fsmap 9 up:standby 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:41:04.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:03 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/804363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:05.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:05.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 -- 172.21.15.119:0/1260270715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 msgr2=0x7f5494104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 --2- 172.21.15.119:0/1260270715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494104550 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5484009ca0 tx=0x7f548402c100 comp rx=0 tx=0).stop 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 -- 172.21.15.119:0/1260270715 shutdown_connections 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 --2- 172.21.15.119:0/1260270715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54941073c0 0x7f5494109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 --2- 172.21.15.119:0/1260270715 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5494104a90 0x7f5494106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 --2- 172.21.15.119:0/1260270715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494104550 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 -- 172.21.15.119:0/1260270715 >> 172.21.15.119:0/1260270715 conn(0x7f54940fbb50 msgr2=0x7f54940fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 -- 172.21.15.119:0/1260270715 shutdown_connections 2024-03-28T18:41:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:05.999+0000 7f5499c00640 1 -- 172.21.15.119:0/1260270715 wait complete. 2024-03-28T18:41:06.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 Processor -- start 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 -- start start 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5494104a90 0x7f549419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54941073c0 0x7f54941a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549410d390 con 0x7f5494102160 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549410d210 con 0x7f54941073c0 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5499c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549410d510 con 0x7f5494104a90 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5492a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5492a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54122/0 (socket says 172.21.15.119:54122) 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5492a00640 1 -- 172.21.15.119:0/224383430 learned_addr learned my addr 172.21.15.119:0/224383430 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:06.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.001+0000 7f5492000640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5494104a90 0x7f549419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5493400640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54941073c0 0x7f54941a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 -- 172.21.15.119:0/224383430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5494104a90 msgr2=0x7f549419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5494104a90 0x7f549419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 -- 172.21.15.119:0/224383430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54941073c0 msgr2=0x7f54941a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54941073c0 0x7f54941a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 -- 172.21.15.119:0/224383430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54941a1870 con 0x7f5494102160 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.002+0000 7f5492a00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f54840325f0 tx=0x7f5484035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.003+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484039040 con 0x7f5494102160 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.003+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5484044070 con 0x7f5494102160 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.003+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54941a1b00 con 0x7f5494102160 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.003+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54840044d0 con 0x7f5494102160 2024-03-28T18:41:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.003+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54941a1fc0 con 0x7f5494102160 2024-03-28T18:41:06.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.004+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5494108f10 con 0x7f5494102160 2024-03-28T18:41:06.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.006+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5484004670 con 0x7f5494102160 2024-03-28T18:41:06.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.007+0000 7f548b400640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 0x7f54780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:06.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.007+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f54840f0c40 con 0x7f5494102160 2024-03-28T18:41:06.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.007+0000 7f5492000640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 0x7f54780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:06.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.008+0000 7f5492000640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 0x7f54780a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f547c0095f0 tx=0x7f547c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:06.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.013+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f54840be0e0 con 0x7f5494102160 2024-03-28T18:41:06.280 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:06 smithi119 ceph-mon[31813]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:06.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.291+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54940622a0 con 0x7f5494102160 2024-03-28T18:41:06.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.293+0000 7f548b400640 1 -- 172.21.15.119:0/224383430 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7679 (secure 0 0 0) 0x7f54840bd9e0 con 0x7f5494102160 2024-03-28T18:41:06.294 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:06.294 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:41:06.294 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 10 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.296+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 msgr2=0x7f54780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.296+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 0x7f54780a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f547c0095f0 tx=0x7f547c005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.296+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 msgr2=0x7f5494199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.296+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f54840325f0 tx=0x7f5484035d10 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 shutdown_connections 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f54780a2800 0x7f54780a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54941073c0 0x7f54941a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5494104a90 0x7f549419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 --2- 172.21.15.119:0/224383430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5494102160 0x7f5494199c60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 >> 172.21.15.119:0/224383430 conn(0x7f54940fbb50 msgr2=0x7f5494101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 shutdown_connections 2024-03-28T18:41:06.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:06.297+0000 7f5499c00640 1 -- 172.21.15.119:0/224383430 wait complete. 2024-03-28T18:41:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:06 smithi196 ceph-mon[36300]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:06 smithi131 ceph-mon[36065]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/224383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:07.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:07.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:07 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/224383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:07.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:07.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:07 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/224383430' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:07.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:07.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:07.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:07.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:07 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:08 smithi119 ceph-mon[31813]: Deploying daemon mds.a on smithi119 2024-03-28T18:41:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:08 smithi119 ceph-mon[31813]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:08 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6838/1834295684,v1:172.21.15.119:6839/1834295684] up:boot 2024-03-28T18:41:08.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:08 smithi119 ceph-mon[31813]: fsmap 10 up:standby 2024-03-28T18:41:08.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:08 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:41:08.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:08.335 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:08 smithi196 ceph-mon[36300]: Deploying daemon mds.a on smithi119 2024-03-28T18:41:08.335 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:08 smithi196 ceph-mon[36300]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:08.335 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:08 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6838/1834295684,v1:172.21.15.119:6839/1834295684] up:boot 2024-03-28T18:41:08.335 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:08 smithi196 ceph-mon[36300]: fsmap 10 up:standby 2024-03-28T18:41:08.335 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:08 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:41:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:08 smithi131 ceph-mon[36065]: Deploying daemon mds.a on smithi119 2024-03-28T18:41:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:08 smithi131 ceph-mon[36065]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:08.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:08 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6838/1834295684,v1:172.21.15.119:6839/1834295684] up:boot 2024-03-28T18:41:08.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:08 smithi131 ceph-mon[36065]: fsmap 10 up:standby 2024-03-28T18:41:08.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:08 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:41:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 -- 172.21.15.119:0/4151333594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb0101f70 msgr2=0x7f5eb0104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/4151333594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb0101f70 0x7f5eb0104360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009ca0 tx=0x7f5ea002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 -- 172.21.15.119:0/4151333594 shutdown_connections 2024-03-28T18:41:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/4151333594 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 0x7f5eb0109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/4151333594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb01048a0 0x7f5eb0106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/4151333594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb0101f70 0x7f5eb0104360 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 -- 172.21.15.119:0/4151333594 >> 172.21.15.119:0/4151333594 conn(0x7f5eb00fb980 msgr2=0x7f5eb00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 -- 172.21.15.119:0/4151333594 shutdown_connections 2024-03-28T18:41:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.546+0000 7f5eb5e00640 1 -- 172.21.15.119:0/4151333594 wait complete. 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.547+0000 7f5eb5e00640 1 Processor -- start 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.547+0000 7f5eb5e00640 1 -- start start 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb01048a0 0x7f5eb00fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 0x7f5eb0100460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb010d180 con 0x7f5eb01048a0 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb010d000 con 0x7f5eb01071d0 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb010d300 con 0x7f5eb0101f70 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:57116/0 (socket says 172.21.15.119:57116) 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eafe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb01048a0 0x7f5eb00fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 -- 172.21.15.119:0/1120506958 learned_addr learned my addr 172.21.15.119:0/1120506958 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb5400640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 0x7f5eb0100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 -- 172.21.15.119:0/1120506958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 msgr2=0x7f5eb0100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 0x7f5eb0100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 -- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb01048a0 msgr2=0x7f5eb00fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb01048a0 0x7f5eb00fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.548+0000 7f5eb4a00640 1 -- 172.21.15.119:0/1120506958 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0100cf0 con 0x7f5eb0101f70 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5eb4a00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00325f0 tx=0x7f5ea0032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0041020 con 0x7f5eb0101f70 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea0034cd0 con 0x7f5eb0101f70 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb01a1870 con 0x7f5eb0101f70 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0039040 con 0x7f5eb0101f70 2024-03-28T18:41:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.549+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb01a1d90 con 0x7f5eb0101f70 2024-03-28T18:41:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.550+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb00fd470 con 0x7f5eb0101f70 2024-03-28T18:41:08.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.552+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5ea0044020 con 0x7f5eb0101f70 2024-03-28T18:41:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.553+0000 7f5ead600640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 0x7f5e940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.553+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5ea00f0890 con 0x7f5eb0101f70 2024-03-28T18:41:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.554+0000 7f5eafe00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 0x7f5e940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:08.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.554+0000 7f5eafe00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 0x7f5e940a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0101650 tx=0x7f5e98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:08.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.556+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ea004b020 con 0x7f5eb0101f70 2024-03-28T18:41:08.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.816+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb002d070 con 0x7f5eb0101f70 2024-03-28T18:41:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.817+0000 7f5ead600640 1 -- 172.21.15.119:0/1120506958 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8474 (secure 0 0 0) 0x7f5ea00bddc0 con 0x7f5eb0101f70 2024-03-28T18:41:08.817 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:08.817 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:41:08.818 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 11 2024-03-28T18:41:08.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 msgr2=0x7f5e940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:08.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 0x7f5e940a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0101650 tx=0x7f5e98005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:08.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 msgr2=0x7f5eb01018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00325f0 tx=0x7f5ea0032ec0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 shutdown_connections 2024-03-28T18:41:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e940a2800 0x7f5e940a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb01071d0 0x7f5eb0100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb01048a0 0x7f5eb00fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 --2- 172.21.15.119:0/1120506958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb0101f70 0x7f5eb01018e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 >> 172.21.15.119:0/1120506958 conn(0x7f5eb00fb980 msgr2=0x7f5eb0102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 shutdown_connections 2024-03-28T18:41:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:08.820+0000 7f5eb5e00640 1 -- 172.21.15.119:0/1120506958 wait complete. 2024-03-28T18:41:09.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:09 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1120506958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:09 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1120506958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:09 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1120506958' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:10 smithi119 ceph-mon[31813]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:10.418 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 podman[74601]: 2024-03-28 18:41:10.172295442 +0000 UTC m=+0.143198264 container create 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-28T18:41:10.419 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 podman[74601]: 2024-03-28 18:41:10.418017644 +0000 UTC m=+0.388920520 container init 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T18:41:10.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:10 smithi196 ceph-mon[36300]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:10 smithi131 ceph-mon[36065]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:10.818 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 podman[74601]: 2024-03-28 18:41:10.428121951 +0000 UTC m=+0.399024754 container start 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:41:10.819 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 ceph-mds[74620]: -- 172.21.15.119:0/1434739936 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a3a46c9a40 con 0x55a3a469cc00 2024-03-28T18:41:10.819 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: starting mds.a at 2024-03-28T18:41:10.819 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 bash[74601]: 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 2024-03-28T18:41:10.819 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:41:10 smithi119 systemd[1]: Started Ceph mds.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:41:10.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.065+0000 7f3a0e200640 1 -- 172.21.15.119:0/3109971697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a080ffd60 msgr2=0x7f3a08102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.065+0000 7f3a0e200640 1 --2- 172.21.15.119:0/3109971697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a080ffd60 0x7f3a08102190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009900 tx=0x7f39f802bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 -- 172.21.15.119:0/3109971697 shutdown_connections 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 --2- 172.21.15.119:0/3109971697 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a08105080 0x7f3a08105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 --2- 172.21.15.119:0/3109971697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a081026d0 0x7f3a08104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 --2- 172.21.15.119:0/3109971697 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a080ffd60 0x7f3a08102190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 -- 172.21.15.119:0/3109971697 >> 172.21.15.119:0/3109971697 conn(0x7f3a080fb980 msgr2=0x7f3a080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 -- 172.21.15.119:0/3109971697 shutdown_connections 2024-03-28T18:41:11.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.066+0000 7f3a0e200640 1 -- 172.21.15.119:0/3109971697 wait complete. 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 Processor -- start 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 -- start start 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a08105080 0x7f3a081a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08113740 con 0x7f3a080ffd60 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081135c0 con 0x7f3a081026d0 2024-03-28T18:41:11.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081138c0 con 0x7f3a08105080 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f39ffe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f39ffe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:50394/0 (socket says 172.21.15.119:50394) 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f39ffe00640 1 -- 172.21.15.119:0/1803632857 learned_addr learned my addr 172.21.15.119:0/1803632857 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37282/0 (socket says 172.21.15.119:37282) 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.067+0000 7f3a0d800640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a08105080 0x7f3a081a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 -- 172.21.15.119:0/1803632857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a08105080 msgr2=0x7f3a081a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a08105080 0x7f3a081a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 -- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 msgr2=0x7f3a0819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 -- 172.21.15.119:0/1803632857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a081a59d0 con 0x7f3a081026d0 2024-03-28T18:41:11.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a0ce00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f39ffe00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39f000e9f0 tx=0x7f39f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f000cd80 con 0x7f3a081026d0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39f0010910 con 0x7f3a081026d0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0018990 con 0x7f3a081026d0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a081a5cc0 con 0x7f3a081026d0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.068+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a081a6200 con 0x7f3a081026d0 2024-03-28T18:41:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.070+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a080fd470 con 0x7f3a081026d0 2024-03-28T18:41:11.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.071+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f39f0010ac0 con 0x7f3a081026d0 2024-03-28T18:41:11.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.073+0000 7f3a06000640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 0x7f39ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:11.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.073+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f39f00ce8f0 con 0x7f3a081026d0 2024-03-28T18:41:11.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.073+0000 7f3a0ce00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 0x7f39ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:11.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.074+0000 7f3a0ce00640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 0x7f39ec0a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f39f802c7b0 tx=0x7f39f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:11.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.077+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39f009b740 con 0x7f3a081026d0 2024-03-28T18:41:11.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.314+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a0802d070 con 0x7f3a081026d0 2024-03-28T18:41:11.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.315+0000 7f3a06000640 1 -- 172.21.15.119:0/1803632857 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9266 (secure 0 0 0) 0x7f39f009f7e0 con 0x7f3a081026d0 2024-03-28T18:41:11.316 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:11.316 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:41:11.316 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 12 2024-03-28T18:41:11.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 msgr2=0x7f39ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:11.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 0x7f39ec0a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f39f802c7b0 tx=0x7f39f8005fd0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 msgr2=0x7f3a0819e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:11.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39f000e9f0 tx=0x7f39f000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 shutdown_connections 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39ec0a2880 0x7f39ec0a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a08105080 0x7f3a081a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a081026d0 0x7f3a0819e2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 --2- 172.21.15.119:0/1803632857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a080ffd60 0x7f3a0819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 >> 172.21.15.119:0/1803632857 conn(0x7f3a080fb980 msgr2=0x7f3a08103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 shutdown_connections 2024-03-28T18:41:11.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:11.318+0000 7f3a0e200640 1 -- 172.21.15.119:0/1803632857 wait complete. 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: Deploying daemon mds.b on smithi131 2024-03-28T18:41:11.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:11.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6840/683166305,v1:172.21.15.119:6841/683166305] up:boot 2024-03-28T18:41:11.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: fsmap 11 up:standby 2024-03-28T18:41:11.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:41:11.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:11 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1803632857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: Deploying daemon mds.b on smithi131 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6840/683166305,v1:172.21.15.119:6841/683166305] up:boot 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: fsmap 11 up:standby 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:41:12.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:11 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1803632857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:12.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: Deploying daemon mds.b on smithi131 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6840/683166305,v1:172.21.15.119:6841/683166305] up:boot 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: fsmap 11 up:standby 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:41:12.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:11 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1803632857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:12.945 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:12 smithi131 systemd[1]: Starting Ceph mds.b for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:41:13.281 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 podman[66004]: 2024-03-28 18:41:12.923837051 +0000 UTC m=+0.017869777 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:41:13.281 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 podman[66004]: 2024-03-28 18:41:13.062175962 +0000 UTC m=+0.156208670 container create db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:41:13.281 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 podman[66004]: 2024-03-28 18:41:13.281546153 +0000 UTC m=+0.375578867 container init db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T18:41:13.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.561+0000 7ff93b200640 1 -- 172.21.15.119:0/3529343308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff934104ad0 msgr2=0x7ff934104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.561+0000 7ff93b200640 1 --2- 172.21.15.119:0/3529343308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff934104ad0 0x7ff934104f50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff920009900 tx=0x7ff92002bc40 comp rx=0 tx=0).stop 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 -- 172.21.15.119:0/3529343308 shutdown_connections 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 --2- 172.21.15.119:0/3529343308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 0x7ff93410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 --2- 172.21.15.119:0/3529343308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff934104ad0 0x7ff934104f50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 --2- 172.21.15.119:0/3529343308 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff9341038e0 0x7ff934103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 -- 172.21.15.119:0/3529343308 >> 172.21.15.119:0/3529343308 conn(0x7ff9340fee90 msgr2=0x7ff9341012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 -- 172.21.15.119:0/3529343308 shutdown_connections 2024-03-28T18:41:13.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.562+0000 7ff93b200640 1 -- 172.21.15.119:0/3529343308 wait complete. 2024-03-28T18:41:13.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 Processor -- start 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 -- start start 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 0x7ff93419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 0x7ff9341a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410f590 con 0x7ff9341038e0 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410f410 con 0x7ff934104ad0 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff93b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410f710 con 0x7ff934105490 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff933e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff933e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37294/0 (socket says 172.21.15.119:37294) 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff933e00640 1 -- 172.21.15.119:0/2386731428 learned_addr learned my addr 172.21.15.119:0/2386731428 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.563+0000 7ff933400640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 0x7ff93419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff939200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 0x7ff9341a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 -- 172.21.15.119:0/2386731428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 msgr2=0x7ff9341a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 0x7ff9341a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 -- 172.21.15.119:0/2386731428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 msgr2=0x7ff93419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 0x7ff93419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 -- 172.21.15.119:0/2386731428 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9341a17c0 con 0x7ff9341038e0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933400640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 0x7ff93419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff933e00640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff92400e9c0 tx=0x7ff92400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.564+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92400cd80 con 0x7ff9341038e0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.565+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9240035f0 con 0x7ff9341038e0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.565+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924010690 con 0x7ff9341038e0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.565+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9341a1ab0 con 0x7ff9341038e0 2024-03-28T18:41:13.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.565+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9341a1f80 con 0x7ff9341038e0 2024-03-28T18:41:13.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.566+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9340688e0 con 0x7ff9341038e0 2024-03-28T18:41:13.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.568+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff9240037a0 con 0x7ff9341038e0 2024-03-28T18:41:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.569+0000 7ff930c00640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 0x7ff9180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.569+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9240ceac0 con 0x7ff9341038e0 2024-03-28T18:41:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.570+0000 7ff933400640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 0x7ff9180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:13.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.571+0000 7ff933400640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 0x7ff9180a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff92002c250 tx=0x7ff92002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:13.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.574+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff92409bf60 con 0x7ff9341038e0 2024-03-28T18:41:13.694 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 podman[66004]: 2024-03-28 18:41:13.284406031 +0000 UTC m=+0.378438739 container start db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T18:41:13.695 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 ceph-mds[66023]: -- 172.21.15.131:0/3195049677 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e3bb17da40 con 0x55e3bb150c00 2024-03-28T18:41:13.695 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: starting mds.b at 2024-03-28T18:41:13.695 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 bash[66004]: db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d 2024-03-28T18:41:13.695 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:13 smithi131 systemd[1]: Started Ceph mds.b for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:41:13.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.820+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff93410ac00 con 0x7ff9341038e0 2024-03-28T18:41:13.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.821+0000 7ff930c00640 1 -- 172.21.15.119:0/2386731428 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9266 (secure 0 0 0) 0x7ff92409b860 con 0x7ff9341038e0 2024-03-28T18:41:13.821 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:13.821 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T18:41:13.821 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 12 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 msgr2=0x7ff9180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 0x7ff9180a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff92002c250 tx=0x7ff92002cf90 comp rx=0 tx=0).stop 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 msgr2=0x7ff934199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff92400e9c0 tx=0x7ff92400ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 shutdown_connections 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff9180a2800 0x7ff9180a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff934105490 0x7ff9341a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff934104ad0 0x7ff93419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 --2- 172.21.15.119:0/2386731428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9341038e0 0x7ff934199ba0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.823+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 >> 172.21.15.119:0/2386731428 conn(0x7ff9340fee90 msgr2=0x7ff934107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.824+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 shutdown_connections 2024-03-28T18:41:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:13.824+0000 7ff93b200640 1 -- 172.21.15.119:0/2386731428 wait complete. 2024-03-28T18:41:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2386731428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:14 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2386731428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:14 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2386731428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:14.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:14 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:15.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:15 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:boot 2024-03-28T18:41:15.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:15 smithi119 ceph-mon[31813]: fsmap 12 up:standby 2024-03-28T18:41:15.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:15 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:41:15.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:15 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:boot 2024-03-28T18:41:15.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:15 smithi196 ceph-mon[36300]: fsmap 12 up:standby 2024-03-28T18:41:15.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:15 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:41:15.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:15 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:boot 2024-03-28T18:41:15.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:15 smithi131 ceph-mon[36065]: fsmap 12 up:standby 2024-03-28T18:41:15.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:15 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:41:15.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 -- 172.21.15.119:0/3038970607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa80103270 msgr2=0x7faa80109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 --2- 172.21.15.119:0/3038970607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa80103270 0x7faa80109b00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa78009be0 tx=0x7faa7802bef0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 -- 172.21.15.119:0/3038970607 shutdown_connections 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 --2- 172.21.15.119:0/3038970607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa80103270 0x7faa80109b00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 --2- 172.21.15.119:0/3038970607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 0x7faa80102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 --2- 172.21.15.119:0/3038970607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa801016c0 0x7faa80101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.045+0000 7faa86200640 1 -- 172.21.15.119:0/3038970607 >> 172.21.15.119:0/3038970607 conn(0x7faa800fcc50 msgr2=0x7faa800ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.046+0000 7faa86200640 1 -- 172.21.15.119:0/3038970607 shutdown_connections 2024-03-28T18:41:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.046+0000 7faa86200640 1 -- 172.21.15.119:0/3038970607 wait complete. 2024-03-28T18:41:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.046+0000 7faa86200640 1 Processor -- start 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 -- start start 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 0x7faa8019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa80103270 0x7faa801a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8010d370 con 0x7faa80103270 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8010d1f0 con 0x7faa801016c0 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa86200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8010d4f0 con 0x7faa801028b0 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa80103270 0x7faa801a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa7f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa7f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:50430/0 (socket says 172.21.15.119:50430) 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa7f400640 1 -- 172.21.15.119:0/738513643 learned_addr learned my addr 172.21.15.119:0/738513643 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.047+0000 7faa76a00640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 0x7faa8019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 -- 172.21.15.119:0/738513643 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 msgr2=0x7faa8019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 0x7faa8019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 -- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa80103270 msgr2=0x7faa801a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa80103270 0x7faa801a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 -- 172.21.15.119:0/738513643 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa801a1760 con 0x7faa801016c0 2024-03-28T18:41:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7f400640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa7000e9f0 tx=0x7faa7000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000ccc0 con 0x7faa801016c0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa700035f0 con 0x7faa801016c0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa801a1a50 con 0x7faa801016c0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.048+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7000ce60 con 0x7faa801016c0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.049+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa801a1ff0 con 0x7faa801016c0 2024-03-28T18:41:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.050+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa801a2280 con 0x7faa801016c0 2024-03-28T18:41:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.051+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faa70004020 con 0x7faa801016c0 2024-03-28T18:41:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.052+0000 7faa7cc00640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 0x7faa640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.053+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faa700ce810 con 0x7faa801016c0 2024-03-28T18:41:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.053+0000 7faa76a00640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 0x7faa640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.053+0000 7faa76a00640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 0x7faa640a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa68005fd0 tx=0x7faa68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.056+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faa7009bd50 con 0x7faa801016c0 2024-03-28T18:41:16.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.315+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa801a2450 con 0x7faa801016c0 2024-03-28T18:41:16.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.316+0000 7faa7cc00640 1 -- 172.21.15.119:0/738513643 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10061 (secure 0 0 0) 0x7faa7009b650 con 0x7faa801016c0 2024-03-28T18:41:16.317 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:16.317 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14460,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24563,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T18:41:16.317 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 13 2024-03-28T18:41:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:16 smithi119 ceph-mon[31813]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:16 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 msgr2=0x7faa640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 0x7faa640a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa68005fd0 tx=0x7faa68005850 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 msgr2=0x7faa80199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa7000e9f0 tx=0x7faa7000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 shutdown_connections 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7faa640a2880 0x7faa640a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa80103270 0x7faa801a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa801028b0 0x7faa8019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 --2- 172.21.15.119:0/738513643 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faa801016c0 0x7faa80199b40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 >> 172.21.15.119:0/738513643 conn(0x7faa800fcc50 msgr2=0x7faa800738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 shutdown_connections 2024-03-28T18:41:16.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.319+0000 7faa86200640 1 -- 172.21.15.119:0/738513643 wait complete. 2024-03-28T18:41:16.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:41:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:16 smithi196 ceph-mon[36300]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:16 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:16 smithi131 ceph-mon[36065]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:16 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.533+0000 7f398e600640 1 -- 172.21.15.119:0/3893699261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 msgr2=0x7f3988104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.533+0000 7f398e600640 1 --2- 172.21.15.119:0/3893699261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988104d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3974009900 tx=0x7f397402bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 -- 172.21.15.119:0/3893699261 shutdown_connections 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 --2- 172.21.15.119:0/3893699261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f398810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 --2- 172.21.15.119:0/3893699261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988104d70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 --2- 172.21.15.119:0/3893699261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 0x7f3988103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 -- 172.21.15.119:0/3893699261 >> 172.21.15.119:0/3893699261 conn(0x7f39880fec70 msgr2=0x7f39881010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 -- 172.21.15.119:0/3893699261 shutdown_connections 2024-03-28T18:41:16.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.534+0000 7f398e600640 1 -- 172.21.15.119:0/3893699261 wait complete. 2024-03-28T18:41:16.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 Processor -- start 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 -- start start 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 0x7f3988199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f39881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398810f3b0 con 0x7f39881052b0 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398810f230 con 0x7f39881048f0 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.535+0000 7f398e600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398810f530 con 0x7f3988103700 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f39881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f3987e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f3987e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:50446/0 (socket says 172.21.15.119:50446) 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f3987e00640 1 -- 172.21.15.119:0/3325985364 learned_addr learned my addr 172.21.15.119:0/3325985364 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398d200640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 0x7f3988199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 -- 172.21.15.119:0/3325985364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 msgr2=0x7f3988199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 0x7f3988199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 -- 172.21.15.119:0/3325985364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 msgr2=0x7f3988199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 -- 172.21.15.119:0/3325985364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39881a1640 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f3987e00640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.536+0000 7f398dc00640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f39881a0fd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3980009fd0 tx=0x7f398000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.537+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000ce20 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.537+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39881a1930 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.537+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3980005020 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.537+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980010720 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.537+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39881a1e50 con 0x7f39881052b0 2024-03-28T18:41:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.538+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398810aa20 con 0x7f39881052b0 2024-03-28T18:41:16.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.540+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3980005540 con 0x7f39881052b0 2024-03-28T18:41:16.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.541+0000 7f3985600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 0x7f39700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.541+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f39800ced40 con 0x7f39881052b0 2024-03-28T18:41:16.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.541+0000 7f398d200640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 0x7f39700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.542+0000 7f398d200640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 0x7f39700a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3988104750 tx=0x7f397c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.546+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f398009c1e0 con 0x7f39881052b0 2024-03-28T18:41:16.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.755+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39880622a0 con 0x7f39881052b0 2024-03-28T18:41:16.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.757+0000 7f3985600640 1 -- 172.21.15.119:0/3325985364 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18516 (secure 0 0 0) 0x7f398009bae0 con 0x7f39881052b0 2024-03-28T18:41:16.758 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:16.758 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:40:23.646844+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:16.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 msgr2=0x7f39700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 0x7f39700a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3988104750 tx=0x7f397c005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 msgr2=0x7f39881a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f39881a0fd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3980009fd0 tx=0x7f398000ef10 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 shutdown_connections 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f39700a2800 0x7f39700a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39881052b0 0x7f39881a0fd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f39881048f0 0x7f3988199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.759+0000 7f398e600640 1 --2- 172.21.15.119:0/3325985364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3988103700 0x7f3988199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.760+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 >> 172.21.15.119:0/3325985364 conn(0x7f39880fec70 msgr2=0x7f3988107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.760+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 shutdown_connections 2024-03-28T18:41:16.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.760+0000 7f398e600640 1 -- 172.21.15.119:0/3325985364 wait complete. 2024-03-28T18:41:16.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:41:16.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 -- 172.21.15.119:0/83934189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 msgr2=0x7fe4d41099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 --2- 172.21.15.119:0/83934189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d41099e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009be0 tx=0x7fe4cc02bef0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 -- 172.21.15.119:0/83934189 shutdown_connections 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 --2- 172.21.15.119:0/83934189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d41099e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 --2- 172.21.15.119:0/83934189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 0x7fe4d4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 --2- 172.21.15.119:0/83934189 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 0x7fe4d41019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 -- 172.21.15.119:0/83934189 >> 172.21.15.119:0/83934189 conn(0x7fe4d40fcb10 msgr2=0x7fe4d40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 -- 172.21.15.119:0/83934189 shutdown_connections 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.968+0000 7fe4d9400640 1 -- 172.21.15.119:0/83934189 wait complete. 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.969+0000 7fe4d9400640 1 Processor -- start 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.969+0000 7fe4d9400640 1 -- start start 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.969+0000 7fe4d9400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 0x7fe4d4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d9400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 0x7fe4d4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d9400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d410d250 con 0x7fe4d4102790 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d9400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d410d0d0 con 0x7fe4d41015a0 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d9400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d410d3d0 con 0x7fe4d4103150 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 0x7fe4d4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 0x7fe4d4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:55992/0 (socket says 172.21.15.119:55992) 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 -- 172.21.15.119:0/835138967 learned_addr learned my addr 172.21.15.119:0/835138967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 -- 172.21.15.119:0/835138967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 msgr2=0x7fe4d4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 0x7fe4d4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 -- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 msgr2=0x7fe4d4073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 0x7fe4d4073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 -- 172.21.15.119:0/835138967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d4074860 con 0x7fe4d4103150 2024-03-28T18:41:16.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.970+0000 7fe4d8a00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036040 tx=0x7fe4cc0337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.971+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc043020 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.971+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc02ca60 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.971+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc034690 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.971+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d41a1950 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.971+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d41a1ef0 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.972+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d4108860 con 0x7fe4d4103150 2024-03-28T18:41:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.974+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe4cc046020 con 0x7fe4d4103150 2024-03-28T18:41:16.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.975+0000 7fe4d0c00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 0x7fe4b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:16.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.975+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe4cc0f1360 con 0x7fe4d4103150 2024-03-28T18:41:16.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.975+0000 7fe4d3e00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 0x7fe4b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.976+0000 7fe4d3e00640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 0x7fe4b80a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe4d41025f0 tx=0x7fe4c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:16.977+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe4cc04d020 con 0x7fe4d4103150 2024-03-28T18:41:17.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.179+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4d402d200 con 0x7fe4d4103150 2024-03-28T18:41:17.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.180+0000 7fe4d0c00640 1 -- 172.21.15.119:0/835138967 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe4cc0be920 con 0x7fe4d4103150 2024-03-28T18:41:17.181 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.182+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 msgr2=0x7fe4b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.182+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 0x7fe4b80a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe4d41025f0 tx=0x7fe4c4005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.182+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 msgr2=0x7fe4d4073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.182+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036040 tx=0x7fe4cc0337b0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 shutdown_connections 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe4b80a2880 0x7fe4b80a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4d4103150 0x7fe4d4073fd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d4102790 0x7fe4d4073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 --2- 172.21.15.119:0/835138967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe4d41015a0 0x7fe4d4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 >> 172.21.15.119:0/835138967 conn(0x7fe4d40fcb10 msgr2=0x7fe4d4105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 shutdown_connections 2024-03-28T18:41:17.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.183+0000 7fe4d9400640 1 -- 172.21.15.119:0/835138967 wait complete. 2024-03-28T18:41:17.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T18:41:17.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/738513643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:17 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3325985364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 -- 172.21.15.119:0/2874416367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 msgr2=0x7f143410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 --2- 172.21.15.119:0/2874416367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f143410bd20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f142c009ca0 tx=0x7f142c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 -- 172.21.15.119:0/2874416367 shutdown_connections 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 --2- 172.21.15.119:0/2874416367 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f143410bd20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 --2- 172.21.15.119:0/2874416367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1434104ad0 0x7f1434104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 --2- 172.21.15.119:0/2874416367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14341038e0 0x7f1434103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 -- 172.21.15.119:0/2874416367 >> 172.21.15.119:0/2874416367 conn(0x7f14340fee90 msgr2=0x7f14341012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 -- 172.21.15.119:0/2874416367 shutdown_connections 2024-03-28T18:41:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.400+0000 7f1438a00640 1 -- 172.21.15.119:0/2874416367 wait complete. 2024-03-28T18:41:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.401+0000 7f1438a00640 1 Processor -- start 2024-03-28T18:41:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.401+0000 7f1438a00640 1 -- start start 2024-03-28T18:41:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1434104ad0 0x7f143419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f14341a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143410f560 con 0x7f1434104ad0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143410f3e0 con 0x7f1434105490 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1438a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143410f6e0 con 0x7f14341038e0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1434104ad0 0x7f143419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:56020/0 (socket says 172.21.15.119:56020) 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 -- 172.21.15.119:0/1761955829 learned_addr learned my addr 172.21.15.119:0/1761955829 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1432800640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f14341a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 -- 172.21.15.119:0/1761955829 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 msgr2=0x7f14341a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f14341a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 -- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1434104ad0 msgr2=0x7f143419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1434104ad0 0x7f143419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.402+0000 7f1431e00640 1 -- 172.21.15.119:0/1761955829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14341a1790 con 0x7f14341038e0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f1431e00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f142400e9c0 tx=0x7f142400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400cd80 con 0x7f14341038e0 2024-03-28T18:41:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14240035f0 con 0x7f14341038e0 2024-03-28T18:41:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424010690 con 0x7f14341038e0 2024-03-28T18:41:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14341a1a80 con 0x7f14341038e0 2024-03-28T18:41:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.403+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341a1f80 con 0x7f14341038e0 2024-03-28T18:41:17.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.404+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143410ac00 con 0x7f14341038e0 2024-03-28T18:41:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.407+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f14240037a0 con 0x7f14341038e0 2024-03-28T18:41:17.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.409+0000 7f142aa00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 0x7f14180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.409+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f14240cea10 con 0x7f14341038e0 2024-03-28T18:41:17.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.409+0000 7f1431400640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 0x7f14180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.410+0000 7f1431400640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 0x7f14180a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f143419b080 tx=0x7f1420005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.413+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f142409beb0 con 0x7f14341038e0 2024-03-28T18:41:17.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/738513643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:17.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:17.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:17 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3325985364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/738513643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:17 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3325985364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:17.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.618+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f14340622d0 con 0x7f14341038e0 2024-03-28T18:41:17.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.618+0000 7f142aa00640 1 -- 172.21.15.119:0/1761955829 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f142409b7b0 con 0x7f14341038e0 2024-03-28T18:41:17.619 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:17.619 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-03-28T18:41:17.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:41:17.620 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-03-28T18:41:17.620 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-03-28T18:41:17.620 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:41:17.620 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 msgr2=0x7f14180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 0x7f14180a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f143419b080 tx=0x7f1420005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 msgr2=0x7f1434199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f142400e9c0 tx=0x7f142400ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 shutdown_connections 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f14180a2800 0x7f14180a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1434105490 0x7f14341a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1434104ad0 0x7f143419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 --2- 172.21.15.119:0/1761955829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14341038e0 0x7f1434199b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 >> 172.21.15.119:0/1761955829 conn(0x7f14340fee90 msgr2=0x7f1434107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:17.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 shutdown_connections 2024-03-28T18:41:17.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.621+0000 7f1438a00640 1 -- 172.21.15.119:0/1761955829 wait complete. 2024-03-28T18:41:17.643 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T18:41:17.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.837+0000 7f56d7200640 1 -- 172.21.15.119:0/3128939467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8109770 msgr2=0x7f56d8110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.837+0000 7f56d7200640 1 --2- 172.21.15.119:0/3128939467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8109770 0x7f56d8110000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56d0009ca0 tx=0x7f56d002c100 comp rx=0 tx=0).stop 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 -- 172.21.15.119:0/3128939467 shutdown_connections 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 --2- 172.21.15.119:0/3128939467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8109770 0x7f56d8110000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 --2- 172.21.15.119:0/3128939467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 0x7f56d8109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 --2- 172.21.15.119:0/3128939467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8107bc0 0x7f56d8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 -- 172.21.15.119:0/3128939467 >> 172.21.15.119:0/3128939467 conn(0x7f56d8076b70 msgr2=0x7f56d8078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 -- 172.21.15.119:0/3128939467 shutdown_connections 2024-03-28T18:41:17.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.838+0000 7f56d7200640 1 -- 172.21.15.119:0/3128939467 wait complete. 2024-03-28T18:41:17.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 Processor -- start 2024-03-28T18:41:17.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 -- start start 2024-03-28T18:41:17.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8107bc0 0x7f56d819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 0x7f56d819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d8113840 con 0x7f56d8107bc0 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d81136c0 con 0x7f56d8108db0 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d81139c0 con 0x7f56d8109770 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8107bc0 0x7f56d819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:56038/0 (socket says 172.21.15.119:56038) 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d6800640 1 -- 172.21.15.119:0/1197562741 learned_addr learned my addr 172.21.15.119:0/1197562741 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.839+0000 7f56d5400640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 0x7f56d819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 -- 172.21.15.119:0/1197562741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 msgr2=0x7f56d819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 0x7f56d819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 -- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8107bc0 msgr2=0x7f56d819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8107bc0 0x7f56d819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 -- 172.21.15.119:0/1197562741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d81a5ac0 con 0x7f56d8109770 2024-03-28T18:41:17.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d6800640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f56d0037040 tx=0x7f56d0035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0039040 con 0x7f56d8109770 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d81a5d50 con 0x7f56d8109770 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56d0044070 con 0x7f56d8109770 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0004360 con 0x7f56d8109770 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.840+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d81a61e0 con 0x7f56d8109770 2024-03-28T18:41:17.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.841+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d810ee80 con 0x7f56d8109770 2024-03-28T18:41:17.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.843+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f56d0004500 con 0x7f56d8109770 2024-03-28T18:41:17.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.844+0000 7f56cea00640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:17.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.845+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56d00f0b30 con 0x7f56d8109770 2024-03-28T18:41:17.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.845+0000 7f56d5e00640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:17.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.845+0000 7f56d5e00640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56d8108c10 tx=0x7f56c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:17.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:17.849+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56d00bdfd0 con 0x7f56d8109770 2024-03-28T18:41:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:18.043+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f56d80622a0 con 0x7f56d8109770 2024-03-28T18:41:18.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:18 smithi119 ceph-mon[31813]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:18.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:18 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/835138967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:41:18.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:18 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1761955829' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:41:18.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:18 smithi131 ceph-mon[36065]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:18.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:18 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/835138967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:41:18.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:18 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1761955829' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:41:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:18 smithi196 ceph-mon[36300]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:18 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/835138967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:41:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:18 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1761955829' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T18:41:19.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.047+0000 7f56cea00640 1 -- 172.21.15.119:0/1197562741 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56d00bd8d0 con 0x7f56d8109770 2024-03-28T18:41:19.049 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-03-28T18:41:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.053+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 msgr2=0x7f56bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.053+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f56d8108c10 tx=0x7f56c8005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.053+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 msgr2=0x7f56d81a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.053+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f56d0037040 tx=0x7f56d0035d10 comp rx=0 tx=0).stop 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 shutdown_connections 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56d8109770 0x7f56d81a5450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56d8108db0 0x7f56d819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 --2- 172.21.15.119:0/1197562741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f56d8107bc0 0x7f56d819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 >> 172.21.15.119:0/1197562741 conn(0x7f56d8076b70 msgr2=0x7f56d810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 shutdown_connections 2024-03-28T18:41:19.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.054+0000 7f56d7200640 1 -- 172.21.15.119:0/1197562741 wait complete. 2024-03-28T18:41:19.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T18:41:19.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 -- 172.21.15.119:0/2179205831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54103260 msgr2=0x7fba54109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:19.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 --2- 172.21.15.119:0/2179205831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54103260 0x7fba54109af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009be0 tx=0x7fba4c02bef0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 -- 172.21.15.119:0/2179205831 shutdown_connections 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 --2- 172.21.15.119:0/2179205831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba54103260 0x7fba54109af0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 --2- 172.21.15.119:0/2179205831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541028a0 0x7fba54102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 --2- 172.21.15.119:0/2179205831 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba541016b0 0x7fba54101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 -- 172.21.15.119:0/2179205831 >> 172.21.15.119:0/2179205831 conn(0x7fba540fcc20 msgr2=0x7fba540ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.266+0000 7fba5c200640 1 -- 172.21.15.119:0/2179205831 shutdown_connections 2024-03-28T18:41:19.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.267+0000 7fba5c200640 1 -- 172.21.15.119:0/2179205831 wait complete. 2024-03-28T18:41:19.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.267+0000 7fba5c200640 1 Processor -- start 2024-03-28T18:41:19.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 -- start start 2024-03-28T18:41:19.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba541028a0 0x7fba54073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 0x7fba54073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5410d380 con 0x7fba541028a0 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5410d200 con 0x7fba54103260 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5c200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5410d500 con 0x7fba541016b0 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba59800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba59800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:55524/0 (socket says 172.21.15.119:55524) 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba59800640 1 -- 172.21.15.119:0/3330402442 learned_addr learned my addr 172.21.15.119:0/3330402442 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba58e00640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba541028a0 0x7fba54073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.268+0000 7fba5a200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 0x7fba54073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 -- 172.21.15.119:0/3330402442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 msgr2=0x7fba54073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 0x7fba54073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 -- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba541028a0 msgr2=0x7fba54073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba541028a0 0x7fba54073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 -- 172.21.15.119:0/3330402442 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54074860 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba5a200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 0x7fba54073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba59800640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800ccc0 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba541a1b00 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba480035f0 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4800ce60 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.269+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba541a2050 con 0x7fba541016b0 2024-03-28T18:41:19.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.271+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba54108970 con 0x7fba541016b0 2024-03-28T18:41:19.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.272+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fba48004020 con 0x7fba541016b0 2024-03-28T18:41:19.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.273+0000 7fba52000640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:19.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.274+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fba480cea60 con 0x7fba541016b0 2024-03-28T18:41:19.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.274+0000 7fba58e00640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:19.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.274+0000 7fba58e00640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba540751c0 tx=0x7fba40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:19.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.278+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba4809be10 con 0x7fba541016b0 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1197562741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:19 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1197562741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:19 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1197562741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T18:41:19.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:19 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:19.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:19.483+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fba5402d070 con 0x7fba541016b0 2024-03-28T18:41:20.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.051+0000 7fba52000640 1 -- 172.21.15.119:0/3330402442 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7fba48086120 con 0x7fba541016b0 2024-03-28T18:41:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-03-28T18:41:20.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.059+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 msgr2=0x7fba3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.059+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba540751c0 tx=0x7fba40005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.059+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 msgr2=0x7fba54075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.059+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 shutdown_connections 2024-03-28T18:41:20.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fba3c0a2800 0x7fba3c0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fba54103260 0x7fba54073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba541028a0 0x7fba54073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 --2- 172.21.15.119:0/3330402442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba541016b0 0x7fba54075450 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 >> 172.21.15.119:0/3330402442 conn(0x7fba540fcc20 msgr2=0x7fba54105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 shutdown_connections 2024-03-28T18:41:20.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.061+0000 7fba5c200640 1 -- 172.21.15.119:0/3330402442 wait complete. 2024-03-28T18:41:20.091 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 -- 172.21.15.119:0/1472668662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 msgr2=0x7f77f4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 --2- 172.21.15.119:0/1472668662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4103d00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009900 tx=0x7f77e402bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 -- 172.21.15.119:0/1472668662 shutdown_connections 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 --2- 172.21.15.119:0/1472668662 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4105490 0x7f77f410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 --2- 172.21.15.119:0/1472668662 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4104ad0 0x7f77f4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 --2- 172.21.15.119:0/1472668662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4103d00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 -- 172.21.15.119:0/1472668662 >> 172.21.15.119:0/1472668662 conn(0x7f77f40fee50 msgr2=0x7f77f4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 -- 172.21.15.119:0/1472668662 shutdown_connections 2024-03-28T18:41:20.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.266+0000 7f77fb800640 1 -- 172.21.15.119:0/1472668662 wait complete. 2024-03-28T18:41:20.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 Processor -- start 2024-03-28T18:41:20.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 -- start start 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 0x7f77f419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4105490 0x7f77f41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410f5b0 con 0x7f77f41038e0 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410f430 con 0x7f77f4105490 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.267+0000 7f77fb800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410f730 con 0x7f77f4104ad0 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f8e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f8e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53510/0 (socket says 172.21.15.119:53510) 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f8e00640 1 -- 172.21.15.119:0/3372027192 learned_addr learned my addr 172.21.15.119:0/3372027192 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 0x7f77f419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f9800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4105490 0x7f77f41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 -- 172.21.15.119:0/3372027192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4105490 msgr2=0x7f77f41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4105490 0x7f77f41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 -- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 msgr2=0x7f77f4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 -- 172.21.15.119:0/3372027192 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f41a1830 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f8e00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77ebe00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 0x7f77f419a160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e9f0 tx=0x7f77dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.268+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00ccc0 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.269+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77dc010910 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.269+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc018960 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.269+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f41a1b20 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.269+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f41a20c0 con 0x7f77f4104ad0 2024-03-28T18:41:20.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.270+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f410ac00 con 0x7f77f4104ad0 2024-03-28T18:41:20.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.271+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f77dc010ac0 con 0x7f77f4104ad0 2024-03-28T18:41:20.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.273+0000 7f77f2000640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 0x7f77d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:20.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.273+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f77dc0cecf0 con 0x7f77f4104ad0 2024-03-28T18:41:20.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.273+0000 7f77f8e00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 0x7f77d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:20.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.274+0000 7f77f8e00640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 0x7f77d40a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77f419b0f0 tx=0x7f77e402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:20.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.277+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f77dc09b850 con 0x7f77f4104ad0 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3330402442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:20 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.346 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3330402442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.347 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:20 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:20.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T18:41:20.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T18:41:20.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:41:20.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:41:20.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:41:20.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3330402442' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:20 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T18:41:20.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:20.506+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f77f4062310 con 0x7f77f4104ad0 2024-03-28T18:41:21.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.103+0000 7f77f2000640 1 -- 172.21.15.119:0/3372027192 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f77dc09f8f0 con 0x7f77f4104ad0 2024-03-28T18:41:21.104 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T18:41:21.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.108+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 msgr2=0x7f77d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.108+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 0x7f77d40a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77f419b0f0 tx=0x7f77e402df90 comp rx=0 tx=0).stop 2024-03-28T18:41:21.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.108+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 msgr2=0x7f77f419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.108+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 0x7f77f419a160 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e9f0 tx=0x7f77dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 shutdown_connections 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f77d40a2880 0x7f77d40a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f77f4105490 0x7f77f41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77f4104ad0 0x7f77f419a160 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 --2- 172.21.15.119:0/3372027192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f41038e0 0x7f77f4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 >> 172.21.15.119:0/3372027192 conn(0x7f77f40fee50 msgr2=0x7f77f4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:21.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.109+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 shutdown_connections 2024-03-28T18:41:21.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.110+0000 7f77fb800640 1 -- 172.21.15.119:0/3372027192 wait complete. 2024-03-28T18:41:21.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:41:21.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 -- 172.21.15.119:0/3458876566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 msgr2=0x7f791c109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 --2- 172.21.15.119:0/3458876566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c109350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7908009860 tx=0x7f790802bb70 comp rx=0 tx=0).stop 2024-03-28T18:41:21.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 -- 172.21.15.119:0/3458876566 shutdown_connections 2024-03-28T18:41:21.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 --2- 172.21.15.119:0/3458876566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 --2- 172.21.15.119:0/3458876566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c109350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 --2- 172.21.15.119:0/3458876566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 -- 172.21.15.119:0/3458876566 >> 172.21.15.119:0/3458876566 conn(0x7f791c076cb0 msgr2=0x7f791c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:21.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 -- 172.21.15.119:0/3458876566 shutdown_connections 2024-03-28T18:41:21.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.316+0000 7f7923a00640 1 -- 172.21.15.119:0/3458876566 wait complete. 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.317+0000 7f7923a00640 1 Processor -- start 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.317+0000 7f7923a00640 1 -- start start 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.317+0000 7f7923a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7923a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7923a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7923a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c113940 con 0x7f791c107ce0 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7923a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c1137c0 con 0x7f791c108ed0 2024-03-28T18:41:21.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7923a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c113ac0 con 0x7f791c109890 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7921000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7921000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53524/0 (socket says 172.21.15.119:53524) 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7921000640 1 -- 172.21.15.119:0/3808823318 learned_addr learned my addr 172.21.15.119:0/3808823318 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7921a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 -- 172.21.15.119:0/3808823318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 msgr2=0x7f791c1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 -- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 msgr2=0x7f791c19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f791be00640 1 -- 172.21.15.119:0/3808823318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c1a5c00 con 0x7f791c108ed0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.318+0000 7f7921a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f791be00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c19e520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f790802c0b0 tx=0x7f7908032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908041020 con 0x7f791c108ed0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f790802c9f0 con 0x7f791c108ed0 2024-03-28T18:41:21.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c1a5e90 con 0x7f791c108ed0 2024-03-28T18:41:21.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908039040 con 0x7f791c108ed0 2024-03-28T18:41:21.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.319+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c1a62f0 con 0x7f791c108ed0 2024-03-28T18:41:21.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.320+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c10f000 con 0x7f791c108ed0 2024-03-28T18:41:21.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.322+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7908044020 con 0x7f791c108ed0 2024-03-28T18:41:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.324+0000 7f7919600640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 0x7f79040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.324+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f79080f0210 con 0x7f791c108ed0 2024-03-28T18:41:21.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.324+0000 7f7921000640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 0x7f79040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.325+0000 7f7921000640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 0x7f79040a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7910009c60 tx=0x7f7910009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:21.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.327+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f790804b020 con 0x7f791c108ed0 2024-03-28T18:41:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:21 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:41:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:21 smithi196 ceph-mon[36300]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:41:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:21 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3372027192' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:21 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:21 smithi196 ceph-mon[36300]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T18:41:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:21 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:41:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:21 smithi131 ceph-mon[36065]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:41:21.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:21 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3372027192' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:21 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:21 smithi131 ceph-mon[36065]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T18:41:21.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.524+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f791c062340 con 0x7f791c108ed0 2024-03-28T18:41:21.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.527+0000 7f7919600640 1 -- 172.21.15.119:0/3808823318 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22292 (secure 0 0 0) 0x7f79080bd3a0 con 0x7f791c108ed0 2024-03-28T18:41:21.528 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:21.528 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":85,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:41:20.509879+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 msgr2=0x7f79040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 0x7f79040a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7910009c60 tx=0x7f7910009260 comp rx=0 tx=0).stop 2024-03-28T18:41:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 msgr2=0x7f791c19e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c19e520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f790802c0b0 tx=0x7f7908032430 comp rx=0 tx=0).stop 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 shutdown_connections 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f79040a2800 0x7f79040a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f791c109890 0x7f791c1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f791c108ed0 0x7f791c19e520 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 --2- 172.21.15.119:0/3808823318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f791c107ce0 0x7f791c19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.529+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 >> 172.21.15.119:0/3808823318 conn(0x7f791c076cb0 msgr2=0x7f791c10c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.530+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 shutdown_connections 2024-03-28T18:41:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.530+0000 7f7923a00640 1 -- 172.21.15.119:0/3808823318 wait complete. 2024-03-28T18:41:21.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:21 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T18:41:21.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:21 smithi119 ceph-mon[31813]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T18:41:21.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:21 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3372027192' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:21 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T18:41:21.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:21 smithi119 ceph-mon[31813]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T18:41:21.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:41:21.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 -- 172.21.15.119:0/382329777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394108ed0 msgr2=0x7f3394109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 --2- 172.21.15.119:0/382329777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394108ed0 0x7f3394109350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f337c009900 tx=0x7f337c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 -- 172.21.15.119:0/382329777 shutdown_connections 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 --2- 172.21.15.119:0/382329777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394109890 0x7f3394110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 --2- 172.21.15.119:0/382329777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394108ed0 0x7f3394109350 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 --2- 172.21.15.119:0/382329777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394107ce0 0x7f3394108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.752+0000 7f339b000640 1 -- 172.21.15.119:0/382329777 >> 172.21.15.119:0/382329777 conn(0x7f3394076cb0 msgr2=0x7f33940790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.753+0000 7f339b000640 1 -- 172.21.15.119:0/382329777 shutdown_connections 2024-03-28T18:41:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.753+0000 7f339b000640 1 -- 172.21.15.119:0/382329777 wait complete. 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.753+0000 7f339b000640 1 Processor -- start 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 -- start start 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394108ed0 0x7f339419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394109890 0x7f33941a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394113940 con 0x7f3394107ce0 2024-03-28T18:41:21.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941137c0 con 0x7f3394109890 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f339b000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3394113ac0 con 0x7f3394108ed0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f3393e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f3393e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53550/0 (socket says 172.21.15.119:53550) 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f3393e00640 1 -- 172.21.15.119:0/3771766616 learned_addr learned my addr 172.21.15.119:0/3771766616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f338b400640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394108ed0 0x7f339419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.754+0000 7f3399000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394109890 0x7f33941a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 -- 172.21.15.119:0/3771766616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394108ed0 msgr2=0x7f339419e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394108ed0 0x7f339419e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 -- 172.21.15.119:0/3771766616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394109890 msgr2=0x7f33941a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394109890 0x7f33941a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 -- 172.21.15.119:0/3771766616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33941a5c00 con 0x7f3394107ce0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3393e00640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f338400e9c0 tx=0x7f338400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338400cd80 con 0x7f3394107ce0 2024-03-28T18:41:21.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33840035f0 con 0x7f3394107ce0 2024-03-28T18:41:21.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384010690 con 0x7f3394107ce0 2024-03-28T18:41:21.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33941a5ef0 con 0x7f3394107ce0 2024-03-28T18:41:21.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.755+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33941a6300 con 0x7f3394107ce0 2024-03-28T18:41:21.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.757+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339410f000 con 0x7f3394107ce0 2024-03-28T18:41:21.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.760+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f33840037a0 con 0x7f3394107ce0 2024-03-28T18:41:21.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.761+0000 7f3391600640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:21.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.762+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f33840ced70 con 0x7f3394107ce0 2024-03-28T18:41:21.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.762+0000 7f338b400640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:21.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.762+0000 7f338b400640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 0x7f33780a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f337c02c7b0 tx=0x7f337c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.764+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f338409bf50 con 0x7f3394107ce0 2024-03-28T18:41:21.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.950+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f339402d320 con 0x7f3394107ce0 2024-03-28T18:41:21.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.953+0000 7f3391600640 1 -- 172.21.15.119:0/3771766616 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22292 (secure 0 0 0) 0x7f33840aa730 con 0x7f3394107ce0 2024-03-28T18:41:21.954 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:21.954 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":85,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:41:20.509879+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:21.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 msgr2=0x7f33780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 0x7f33780a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f337c02c7b0 tx=0x7f337c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 msgr2=0x7f339419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f338400e9c0 tx=0x7f338400ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 shutdown_connections 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3394109890 0x7f33941a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3394108ed0 0x7f339419e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 --2- 172.21.15.119:0/3771766616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3394107ce0 0x7f339419dfe0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.955+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 >> 172.21.15.119:0/3771766616 conn(0x7f3394076cb0 msgr2=0x7f339410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.956+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 shutdown_connections 2024-03-28T18:41:21.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:21.956+0000 7f339b000640 1 -- 172.21.15.119:0/3771766616 wait complete. 2024-03-28T18:41:21.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 -- 172.21.15.119:0/2563932752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 msgr2=0x7f2f20077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 --2- 172.21.15.119:0/2563932752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f20077700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009900 tx=0x7f2f0c02bc40 comp rx=0 tx=0).stop 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 -- 172.21.15.119:0/2563932752 shutdown_connections 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 --2- 172.21.15.119:0/2563932752 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20077c40 0x7f2f200784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 --2- 172.21.15.119:0/2563932752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f20077700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 --2- 172.21.15.119:0/2563932752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20076060 0x7f2f20076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 -- 172.21.15.119:0/2563932752 >> 172.21.15.119:0/2563932752 conn(0x7f2f200fee50 msgr2=0x7f2f20101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 -- 172.21.15.119:0/2563932752 shutdown_connections 2024-03-28T18:41:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.179+0000 7f2f26e00640 1 -- 172.21.15.119:0/2563932752 wait complete. 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 Processor -- start 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 -- start start 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f2019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20077c40 0x7f2f201a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201139b0 con 0x7f2f200772a0 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20113830 con 0x7f2f20076060 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.180+0000 7f2f26e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20113b30 con 0x7f2f20077c40 2024-03-28T18:41:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f2019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34280/0 (socket says 172.21.15.119:34280) 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 -- 172.21.15.119:0/3425920771 learned_addr learned my addr 172.21.15.119:0/3425920771 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f24e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20077c40 0x7f2f201a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 -- 172.21.15.119:0/3425920771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20077c40 msgr2=0x7f2f201a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20077c40 0x7f2f201a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 -- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 msgr2=0x7f2f2019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f2019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 -- 172.21.15.119:0/3425920771 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f201a5c50 con 0x7f2f20076060 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1fe00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d770 tx=0x7f2f1000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.181+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10017070 con 0x7f2f20076060 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.182+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f10005500 con 0x7f2f20076060 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.182+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f201a5f40 con 0x7f2f20076060 2024-03-28T18:41:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.182+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10004e00 con 0x7f2f20076060 2024-03-28T18:41:22.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.182+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f201a63a0 con 0x7f2f20076060 2024-03-28T18:41:22.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.183+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f20110250 con 0x7f2f20076060 2024-03-28T18:41:22.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.184+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2f10004fa0 con 0x7f2f20076060 2024-03-28T18:41:22.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.186+0000 7f2f1cc00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 0x7f2f040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:22.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.186+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2f100ce370 con 0x7f2f20076060 2024-03-28T18:41:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.186+0000 7f2f1f400640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 0x7f2f040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.187+0000 7f2f1f400640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 0x7f2f040a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c02efd0 tx=0x7f2f0c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:22.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.190+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f1009b4c0 con 0x7f2f20076060 2024-03-28T18:41:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: pgmap v310: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:22.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3808823318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:22 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3771766616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:22.431+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2f200622a0 con 0x7f2f20076060 2024-03-28T18:41:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: pgmap v310: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:41:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:41:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:41:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T18:41:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T18:41:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3808823318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:22 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3771766616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: pgmap v310: 97 pgs: 5 creating+peering, 4 active+clean, 88 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T18:41:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3808823318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:22.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:22 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3771766616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:23.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.120+0000 7f2f1cc00640 1 -- 172.21.15.119:0/3425920771 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2f1009adc0 con 0x7f2f20076060 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 msgr2=0x7f2f040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 0x7f2f040a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c02efd0 tx=0x7f2f0c02cf90 comp rx=0 tx=0).stop 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 msgr2=0x7f2f2019e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000d770 tx=0x7f2f1000dc30 comp rx=0 tx=0).stop 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 shutdown_connections 2024-03-28T18:41:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2f040a2880 0x7f2f040a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f20077c40 0x7f2f201a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f200772a0 0x7f2f2019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 --2- 172.21.15.119:0/3425920771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2f20076060 0x7f2f2019e030 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 >> 172.21.15.119:0/3425920771 conn(0x7f2f200fee50 msgr2=0x7f2f20100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 shutdown_connections 2024-03-28T18:41:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.124+0000 7f2f26e00640 1 -- 172.21.15.119:0/3425920771 wait complete. 2024-03-28T18:41:23.147 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-28T18:41:23.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-28T18:41:23.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 -- 172.21.15.119:0/456687621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a8103260 msgr2=0x7f58a8109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:23.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 --2- 172.21.15.119:0/456687621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a8103260 0x7f58a8109af0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f58a0009be0 tx=0x7f58a002bef0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 -- 172.21.15.119:0/456687621 shutdown_connections 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 --2- 172.21.15.119:0/456687621 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a8103260 0x7f58a8109af0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 --2- 172.21.15.119:0/456687621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a81028a0 0x7f58a8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 --2- 172.21.15.119:0/456687621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 0x7f58a8101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 -- 172.21.15.119:0/456687621 >> 172.21.15.119:0/456687621 conn(0x7f58a80fcc40 msgr2=0x7f58a80ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 -- 172.21.15.119:0/456687621 shutdown_connections 2024-03-28T18:41:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.356+0000 7f58b0200640 1 -- 172.21.15.119:0/456687621 wait complete. 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.357+0000 7f58b0200640 1 Processor -- start 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.357+0000 7f58b0200640 1 -- start start 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 0x7f58a8075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a8103260 0x7f58a8073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a810d380 con 0x7f58a8103260 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a810d200 con 0x7f58a81028a0 2024-03-28T18:41:23.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58b0200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a810d500 con 0x7f58a81016b0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34310/0 (socket says 172.21.15.119:34310) 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 -- 172.21.15.119:0/3643269749 learned_addr learned my addr 172.21.15.119:0/3643269749 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ae200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a8103260 0x7f58a8073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ad800640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 0x7f58a8075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 -- 172.21.15.119:0/3643269749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 msgr2=0x7f58a8075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 0x7f58a8075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 -- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a8103260 msgr2=0x7f58a8073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a8103260 0x7f58a8073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.358+0000 7f58ace00640 1 -- 172.21.15.119:0/3643269749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a8074860 con 0x7f58a81028a0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58ace00640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f589400e9f0 tx=0x7f589400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589400cd80 con 0x7f58a81028a0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5894010910 con 0x7f58a81028a0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5894018990 con 0x7f58a81028a0 2024-03-28T18:41:23.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a81a1a80 con 0x7f58a81028a0 2024-03-28T18:41:23.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.359+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a81a1f80 con 0x7f58a81028a0 2024-03-28T18:41:23.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.360+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a81089d0 con 0x7f58a81028a0 2024-03-28T18:41:23.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.363+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5894010ac0 con 0x7f58a81028a0 2024-03-28T18:41:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.364+0000 7f58a6000640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 0x7f58900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.364+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f58940ced20 con 0x7f58a81028a0 2024-03-28T18:41:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.364+0000 7f58ad800640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 0x7f58900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.365+0000 7f58ad800640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 0x7f58900a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f58a8102700 tx=0x7f589c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:23.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.368+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f589409be70 con 0x7f58a81028a0 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:active 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3425920771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:23 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:active 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3425920771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:23 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T18:41:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] up:active 2024-03-28T18:41:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:23.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3425920771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:23 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T18:41:23.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:23.610+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f58a8062340 con 0x7f58a81028a0 2024-03-28T18:41:24.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.125+0000 7f58a6000640 1 -- 172.21.15.119:0/3643269749 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f589409b770 con 0x7f58a81028a0 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.129+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 msgr2=0x7f58900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.129+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 0x7f58900a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f58a8102700 tx=0x7f589c009260 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.129+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 msgr2=0x7f58a8073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.129+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f589400e9f0 tx=0x7f589400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.129+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 shutdown_connections 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f58900a2880 0x7f58900a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58a8103260 0x7f58a8073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f58a81028a0 0x7f58a8073a90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 --2- 172.21.15.119:0/3643269749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58a81016b0 0x7f58a8075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 >> 172.21.15.119:0/3643269749 conn(0x7f58a80fcc40 msgr2=0x7f58a8105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 shutdown_connections 2024-03-28T18:41:24.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.130+0000 7f58b0200640 1 -- 172.21.15.119:0/3643269749 wait complete. 2024-03-28T18:41:24.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T18:41:24.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 -- 172.21.15.119:0/302874842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1020c0 msgr2=0x7f804c1044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:24.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 --2- 172.21.15.119:0/302874842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1020c0 0x7f804c1044b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f803c009900 tx=0x7f803c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:24.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 -- 172.21.15.119:0/302874842 shutdown_connections 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 --2- 172.21.15.119:0/302874842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c107320 0x7f804c109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 --2- 172.21.15.119:0/302874842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c1049f0 0x7f804c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 --2- 172.21.15.119:0/302874842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1020c0 0x7f804c1044b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 -- 172.21.15.119:0/302874842 >> 172.21.15.119:0/302874842 conn(0x7f804c0fba10 msgr2=0x7f804c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 -- 172.21.15.119:0/302874842 shutdown_connections 2024-03-28T18:41:24.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.349+0000 7f8051200640 1 -- 172.21.15.119:0/302874842 wait complete. 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.350+0000 7f8051200640 1 Processor -- start 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 -- start start 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 0x7f804c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1049f0 0x7f804c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c10d250 con 0x7f804c1049f0 2024-03-28T18:41:24.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c10d0d0 con 0x7f804c107320 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f8051200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c10d3d0 con 0x7f804c1020c0 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f804aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1049f0 0x7f804c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f804be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f804b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 0x7f804c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f804be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34328/0 (socket says 172.21.15.119:34328) 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.351+0000 7f804be00640 1 -- 172.21.15.119:0/3760910871 learned_addr learned my addr 172.21.15.119:0/3760910871 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 -- 172.21.15.119:0/3760910871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 msgr2=0x7f804c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 0x7f804c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 -- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1049f0 msgr2=0x7f804c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1049f0 0x7f804c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 -- 172.21.15.119:0/3760910871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c1a16c0 con 0x7f804c107320 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804b400640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 0x7f804c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f804be00640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f804400eb10 tx=0x7f804400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:24.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044004b20 con 0x7f804c107320 2024-03-28T18:41:24.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8044004cc0 con 0x7f804c107320 2024-03-28T18:41:24.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804c1a19b0 con 0x7f804c107320 2024-03-28T18:41:24.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.352+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044010760 con 0x7f804c107320 2024-03-28T18:41:24.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.353+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804c1a1eb0 con 0x7f804c107320 2024-03-28T18:41:24.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.354+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804c108e10 con 0x7f804c107320 2024-03-28T18:41:24.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.356+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8044044910 con 0x7f804c107320 2024-03-28T18:41:24.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.357+0000 7f8043e00640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 0x7f80300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:24.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.357+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f80440ceec0 con 0x7f804c107320 2024-03-28T18:41:24.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.357+0000 7f804b400640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 0x7f80300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:24.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.358+0000 7f804b400640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 0x7f80300a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f803c02c240 tx=0x7f803c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:24.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.359+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f804409c090 con 0x7f804c107320 2024-03-28T18:41:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: pgmap v313: 97 pgs: 5 creating+peering, 48 active+clean, 44 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3643269749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:24 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: pgmap v313: 97 pgs: 5 creating+peering, 48 active+clean, 44 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3643269749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:24 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: pgmap v313: 97 pgs: 5 creating+peering, 48 active+clean, 44 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-28T18:41:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T18:41:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:41:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T18:41:24.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:24.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3643269749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:24 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T18:41:24.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:24.602+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f804c02d2c0 con 0x7f804c107320 2024-03-28T18:41:25.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.139+0000 7f8043e00640 1 -- 172.21.15.119:0/3760910871 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f804409b990 con 0x7f804c107320 2024-03-28T18:41:25.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.143+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 msgr2=0x7f80300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:25.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 0x7f80300a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f803c02c240 tx=0x7f803c02df90 comp rx=0 tx=0).stop 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 msgr2=0x7f804c1a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f804400eb10 tx=0x7f804400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 shutdown_connections 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80300a2800 0x7f80300a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f804c107320 0x7f804c1a1050 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f804c1049f0 0x7f804c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 --2- 172.21.15.119:0/3760910871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f804c1020c0 0x7f804c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 >> 172.21.15.119:0/3760910871 conn(0x7f804c0fba10 msgr2=0x7f804c1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:25.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 shutdown_connections 2024-03-28T18:41:25.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.144+0000 7f8051200640 1 -- 172.21.15.119:0/3760910871 wait complete. 2024-03-28T18:41:25.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T18:41:25.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 -- 172.21.15.119:0/827697220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c0772a0 msgr2=0x7f953c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 --2- 172.21.15.119:0/827697220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c0772a0 0x7f953c077700 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9528009900 tx=0x7f952802bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 -- 172.21.15.119:0/827697220 shutdown_connections 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 --2- 172.21.15.119:0/827697220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c077c40 0x7f953c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 --2- 172.21.15.119:0/827697220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c0772a0 0x7f953c077700 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 --2- 172.21.15.119:0/827697220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c076060 0x7f953c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 -- 172.21.15.119:0/827697220 >> 172.21.15.119:0/827697220 conn(0x7f953c0fed10 msgr2=0x7f953c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 -- 172.21.15.119:0/827697220 shutdown_connections 2024-03-28T18:41:25.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.370+0000 7f9542400640 1 -- 172.21.15.119:0/827697220 wait complete. 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.371+0000 7f9542400640 1 Processor -- start 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.371+0000 7f9542400640 1 -- start start 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c0772a0 0x7f953c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 0x7f953c1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c113870 con 0x7f953c077c40 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1136f0 con 0x7f953c076060 2024-03-28T18:41:25.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9542400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c1139f0 con 0x7f953c0772a0 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34342/0 (socket says 172.21.15.119:34342) 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 -- 172.21.15.119:0/3860285267 learned_addr learned my addr 172.21.15.119:0/3860285267 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f953be00640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c0772a0 0x7f953c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541a00640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 0x7f953c1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 -- 172.21.15.119:0/3860285267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c0772a0 msgr2=0x7f953c19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c0772a0 0x7f953c19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 -- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 msgr2=0x7f953c1a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 0x7f953c1a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541000640 1 -- 172.21.15.119:0/3860285267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c1a5ab0 con 0x7f953c076060 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.372+0000 7f9541a00640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 0x7f953c1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9541000640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f953000d950 tx=0x7f953000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530017070 con 0x7f953c076060 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9530009c30 con 0x7f953c076060 2024-03-28T18:41:25.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953000f6b0 con 0x7f953c076060 2024-03-28T18:41:25.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953c1a5da0 con 0x7f953c076060 2024-03-28T18:41:25.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.373+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953c1a61b0 con 0x7f953c076060 2024-03-28T18:41:25.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.374+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953c110110 con 0x7f953c076060 2024-03-28T18:41:25.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.376+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f95300109d0 con 0x7f953c076060 2024-03-28T18:41:25.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.377+0000 7f9539600640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 0x7f95240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:25.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.377+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f95300ce580 con 0x7f953c076060 2024-03-28T18:41:25.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.377+0000 7f953be00640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 0x7f95240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:25.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.378+0000 7f953be00640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 0x7f95240a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f952802c240 tx=0x7f952802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:25.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.381+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9530009de0 con 0x7f953c076060 2024-03-28T18:41:25.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3760910871' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:25 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.444 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: 2024-03-28T18:41:25.020+0000 7f0c83c48700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T18:41:25.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:41:25.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:41:25.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:41:25.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:25.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T18:41:25.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T18:41:25.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3760910871' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:25 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T18:41:25.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T18:41:25.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T18:41:25.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T18:41:25.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T18:41:25.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T18:41:25.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3760910871' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:25 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T18:41:25.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:25.628+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f953c02d070 con 0x7f953c076060 2024-03-28T18:41:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.153+0000 7f9539600640 1 -- 172.21.15.119:0/3860285267 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f953009b6d0 con 0x7f953c076060 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.157+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 msgr2=0x7f95240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.157+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 0x7f95240a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f952802c240 tx=0x7f952802df90 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.157+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 msgr2=0x7f953c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.157+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f953000d950 tx=0x7f953000de10 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 shutdown_connections 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f95240a2880 0x7f95240a4d40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f953c077c40 0x7f953c1a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f953c0772a0 0x7f953c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 --2- 172.21.15.119:0/3860285267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f953c076060 0x7f953c19de90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 >> 172.21.15.119:0/3860285267 conn(0x7f953c0fed10 msgr2=0x7f953c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 shutdown_connections 2024-03-28T18:41:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.158+0000 7f9542400640 1 -- 172.21.15.119:0/3860285267 wait complete. 2024-03-28T18:41:26.184 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-28T18:41:26.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.378+0000 7f4995e00640 1 -- 172.21.15.119:0/170042368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 msgr2=0x7f4990104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.378+0000 7f4995e00640 1 --2- 172.21.15.119:0/170042368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f4990104f40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4978009900 tx=0x7f497802bc40 comp rx=0 tx=0).stop 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 -- 172.21.15.119:0/170042368 shutdown_connections 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 --2- 172.21.15.119:0/170042368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4990105480 0x7f499010bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 --2- 172.21.15.119:0/170042368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f4990104f40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 --2- 172.21.15.119:0/170042368 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f49901038d0 0x7f4990103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 -- 172.21.15.119:0/170042368 >> 172.21.15.119:0/170042368 conn(0x7f49900fee60 msgr2=0x7f4990101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 -- 172.21.15.119:0/170042368 shutdown_connections 2024-03-28T18:41:26.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.379+0000 7f4995e00640 1 -- 172.21.15.119:0/170042368 wait complete. 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 Processor -- start 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 -- start start 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 0x7f4990199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f499019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 0x7f49901a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010f550 con 0x7f49901038d0 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010f3d0 con 0x7f4990105480 2024-03-28T18:41:26.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4995e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010f6d0 con 0x7f4990104ac0 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f498ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 0x7f4990199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f498f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 0x7f49901a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f4987e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f499019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f498ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 0x7f4990199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53634/0 (socket says 172.21.15.119:53634) 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.380+0000 7f498ea00640 1 -- 172.21.15.119:0/3609859670 learned_addr learned my addr 172.21.15.119:0/3609859670 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 -- 172.21.15.119:0/3609859670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 msgr2=0x7f499019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f499019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 -- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 msgr2=0x7f4990199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 0x7f4990199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 -- 172.21.15.119:0/3609859670 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49901a17e0 con 0x7f4990105480 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f498f400640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 0x7f49901a1170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f498800ec50 tx=0x7f498800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988003b70 con 0x7f4990105480 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4988003d10 con 0x7f4990105480 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988010890 con 0x7f4990105480 2024-03-28T18:41:26.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49901a1ad0 con 0x7f4990105480 2024-03-28T18:41:26.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.381+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49901a1fa0 con 0x7f4990105480 2024-03-28T18:41:26.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.383+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499010abf0 con 0x7f4990105480 2024-03-28T18:41:26.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.384+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4988003fc0 con 0x7f4990105480 2024-03-28T18:41:26.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.385+0000 7f4994a00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 0x7f49740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:26.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.386+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f49880ceeb0 con 0x7f4990105480 2024-03-28T18:41:26.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.386+0000 7f498ea00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 0x7f49740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:26.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.386+0000 7f498ea00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 0x7f49740a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f49800058c0 tx=0x7f4980005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:26.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.390+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f498809c000 con 0x7f4990105480 2024-03-28T18:41:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: pgmap v314: 97 pgs: 5 creating+peering, 90 active+clean, 2 unknown; 578 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-28T18:41:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:41:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:active 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3860285267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:26 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: pgmap v314: 97 pgs: 5 creating+peering, 90 active+clean, 2 unknown; 578 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:active 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T18:41:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:41:26.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:41:26.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3860285267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:26 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: pgmap v314: 97 pgs: 5 creating+peering, 90 active+clean, 2 unknown; 578 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-28T18:41:26.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T18:41:26.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] up:active 2024-03-28T18:41:26.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T18:41:26.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T18:41:26.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-28T18:41:26.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-28T18:41:26.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3860285267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T18:41:26.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:26.618+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f4990061ae0 con 0x7f49740a2880 2024-03-28T18:41:27.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:41:26.627+0000 7f3c42400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:41:27.069 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:41:26.627+0000 7f3c42400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:41:27.069 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:41:26.627+0000 7f3c42400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:41:27.069 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:41:26.627+0000 7f3c42400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:41:27.069 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:41:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:41:26.627+0000 7f3c42400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:41:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.203+0000 7f4994a00640 1 -- 172.21.15.119:0/3609859670 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4990061ae0 con 0x7f49740a2880 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 msgr2=0x7f49740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 0x7f49740a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f49800058c0 tx=0x7f4980005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 msgr2=0x7f49901a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 0x7f49901a1170 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f498800ec50 tx=0x7f498800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 shutdown_connections 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f49740a2880 0x7f49740a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4990105480 0x7f49901a1170 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4990104ac0 0x7f499019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 --2- 172.21.15.119:0/3609859670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49901038d0 0x7f4990199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 >> 172.21.15.119:0/3609859670 conn(0x7f49900fee60 msgr2=0x7f4990107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 shutdown_connections 2024-03-28T18:41:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.206+0000 7f4995e00640 1 -- 172.21.15.119:0/3609859670 wait complete. 2024-03-28T18:41:27.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T18:41:27.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 -- 172.21.15.119:0/3784150148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd0104ad0 msgr2=0x7f6cd0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/3784150148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd0104ad0 0x7f6cd0104f50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009900 tx=0x7f6cbc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 -- 172.21.15.119:0/3784150148 shutdown_connections 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/3784150148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 0x7f6cd010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/3784150148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd0104ad0 0x7f6cd0104f50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/3784150148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd01038e0 0x7f6cd0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 -- 172.21.15.119:0/3784150148 >> 172.21.15.119:0/3784150148 conn(0x7f6cd00fee50 msgr2=0x7f6cd0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 -- 172.21.15.119:0/3784150148 shutdown_connections 2024-03-28T18:41:27.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.433+0000 7f6cd4a00640 1 -- 172.21.15.119:0/3784150148 wait complete. 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:active 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:41:27.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:27 smithi196 ceph-mon[36300]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.434+0000 7f6cd4a00640 1 Processor -- start 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.434+0000 7f6cd4a00640 1 -- start start 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 0x7f6cd019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 0x7f6cd01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd010f590 con 0x7f6cd0104ad0 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd010f410 con 0x7f6cd01038e0 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cd4a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd010f710 con 0x7f6cd0105490 2024-03-28T18:41:27.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34406/0 (socket says 172.21.15.119:34406) 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 -- 172.21.15.119:0/1174902713 learned_addr learned my addr 172.21.15.119:0/1174902713 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccd400640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 0x7f6cd019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6cce800640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 0x7f6cd01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 -- 172.21.15.119:0/1174902713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 msgr2=0x7f6cd01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 0x7f6cd01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 -- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 msgr2=0x7f6cd019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 0x7f6cd019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccde00640 1 -- 172.21.15.119:0/1174902713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd01a1810 con 0x7f6cd01038e0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.435+0000 7f6ccd400640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 0x7f6cd019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6ccde00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ebc0 tx=0x7f6cc800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8003970 con 0x7f6cd01038e0 2024-03-28T18:41:27.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cc8004030 con 0x7f6cd01038e0 2024-03-28T18:41:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80197b0 con 0x7f6cd01038e0 2024-03-28T18:41:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd01a1b00 con 0x7f6cd01038e0 2024-03-28T18:41:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.436+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd01a2070 con 0x7f6cd01038e0 2024-03-28T18:41:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.437+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd010ac80 con 0x7f6cd01038e0 2024-03-28T18:41:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.439+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6cc80041e0 con 0x7f6cd01038e0 2024-03-28T18:41:27.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.440+0000 7f6cc6a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 0x7f6cb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.440+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6cc80cf280 con 0x7f6cd01038e0 2024-03-28T18:41:27.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.440+0000 7f6ccd400640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 0x7f6cb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.441+0000 7f6ccd400640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 0x7f6cb40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc02c240 tx=0x7f6cbc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:27.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.444+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6cc809c3d0 con 0x7f6cd01038e0 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:active 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:41:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:27 smithi131 ceph-mon[36065]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:41:27.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T18:41:27.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] up:active 2024-03-28T18:41:27.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T18:41:27.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-28T18:41:27.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-28T18:41:27.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:27 smithi119 ceph-mon[31813]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-28T18:41:27.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.673+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cd0061ae0 con 0x7f6cb40a2880 2024-03-28T18:41:27.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.707+0000 7f6cc6a00640 1 -- 172.21.15.119:0/1174902713 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6cd0061ae0 con 0x7f6cb40a2880 2024-03-28T18:41:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 msgr2=0x7f6cb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 0x7f6cb40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc02c240 tx=0x7f6cbc02df90 comp rx=0 tx=0).stop 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 msgr2=0x7f6cd0199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ebc0 tx=0x7f6cc800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 shutdown_connections 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6cb40a2880 0x7f6cb40a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cd0105490 0x7f6cd01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cd0104ad0 0x7f6cd019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 --2- 172.21.15.119:0/1174902713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6cd01038e0 0x7f6cd0199c40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 >> 172.21.15.119:0/1174902713 conn(0x7f6cd00fee50 msgr2=0x7f6cd0107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 shutdown_connections 2024-03-28T18:41:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.711+0000 7f6cd4a00640 1 -- 172.21.15.119:0/1174902713 wait complete. 2024-03-28T18:41:27.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T18:41:27.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 -- 172.21.15.119:0/1670221154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 msgr2=0x7f25741080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 --2- 172.21.15.119:0/1670221154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f25741080e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2568009900 tx=0x7f256802bc40 comp rx=0 tx=0).stop 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 -- 172.21.15.119:0/1670221154 shutdown_connections 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 --2- 172.21.15.119:0/1670221154 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 0x7f2574110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 --2- 172.21.15.119:0/1670221154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 0x7f2574109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 --2- 172.21.15.119:0/1670221154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f25741080e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 -- 172.21.15.119:0/1670221154 >> 172.21.15.119:0/1670221154 conn(0x7f2574076c60 msgr2=0x7f25740790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 -- 172.21.15.119:0/1670221154 shutdown_connections 2024-03-28T18:41:27.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.943+0000 7f257ba00640 1 -- 172.21.15.119:0/1670221154 wait complete. 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.944+0000 7f257ba00640 1 Processor -- start 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 -- start start 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 0x7f257419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 0x7f25741a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2574113940 con 0x7f2574108eb0 2024-03-28T18:41:27.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25741137c0 con 0x7f2574109870 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f257ba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2574113ac0 con 0x7f2574107cc0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f2579000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f2579000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:55716/0 (socket says 172.21.15.119:55716) 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f2579000640 1 -- 172.21.15.119:0/3637664581 learned_addr learned my addr 172.21.15.119:0/3637664581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f2573e00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 0x7f257419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.945+0000 7f2579a00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 0x7f25741a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 -- 172.21.15.119:0/3637664581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 msgr2=0x7f25741a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 0x7f25741a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 -- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 msgr2=0x7f257419e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 0x7f257419e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 -- 172.21.15.119:0/3637664581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25741a5bd0 con 0x7f2574107cc0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2579000640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2568000c00 tx=0x7f2568005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568041070 con 0x7f2574107cc0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25680044d0 con 0x7f2574107cc0 2024-03-28T18:41:27.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25741a5e60 con 0x7f2574107cc0 2024-03-28T18:41:27.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568034440 con 0x7f2574107cc0 2024-03-28T18:41:27.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.946+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25741a6270 con 0x7f2574107cc0 2024-03-28T18:41:27.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.948+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257410efe0 con 0x7f2574107cc0 2024-03-28T18:41:27.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.950+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2568044070 con 0x7f2574107cc0 2024-03-28T18:41:27.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.951+0000 7f2571600640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 0x7f255c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:27.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.951+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f25680f04f0 con 0x7f2574107cc0 2024-03-28T18:41:27.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.951+0000 7f2573e00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 0x7f255c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:27.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.952+0000 7f2573e00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 0x7f255c0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f257419f4c0 tx=0x7f256000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:27.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:27.955+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f25680bcff0 con 0x7f2574107cc0 2024-03-28T18:41:28.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.192+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2574061b70 con 0x7f255c0a2800 2024-03-28T18:41:28.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.202+0000 7f2571600640 1 -- 172.21.15.119:0/3637664581 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f2574061b70 con 0x7f255c0a2800 2024-03-28T18:41:28.202 INFO:teuthology.orchestra.run.smithi119.stdout:/volumes/qa/sv_0/79fc8c60-d19b-4438-89d7-6afa57462be1 2024-03-28T18:41:28.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 msgr2=0x7f255c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 0x7f255c0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f257419f4c0 tx=0x7f256000b410 comp rx=0 tx=0).stop 2024-03-28T18:41:28.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 msgr2=0x7f257419dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2568000c00 tx=0x7f2568005d50 comp rx=0 tx=0).stop 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 shutdown_connections 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2800 0x7f255c0a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2574109870 0x7f25741a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2574108eb0 0x7f257419e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 --2- 172.21.15.119:0/3637664581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2574107cc0 0x7f257419dfb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.205+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 >> 172.21.15.119:0/3637664581 conn(0x7f2574076c60 msgr2=0x7f257410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.206+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 shutdown_connections 2024-03-28T18:41:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.206+0000 7f257ba00640 1 -- 172.21.15.119:0/3637664581 wait complete. 2024-03-28T18:41:28.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T18:41:28.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 -- 172.21.15.119:0/2966422047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c108eb0 msgr2=0x7fc03c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 --2- 172.21.15.119:0/2966422047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c108eb0 0x7fc03c109330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc024009900 tx=0x7fc02402bc40 comp rx=0 tx=0).stop 2024-03-28T18:41:28.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 -- 172.21.15.119:0/2966422047 shutdown_connections 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 --2- 172.21.15.119:0/2966422047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c109870 0x7fc03c110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 --2- 172.21.15.119:0/2966422047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c108eb0 0x7fc03c109330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 --2- 172.21.15.119:0/2966422047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 -- 172.21.15.119:0/2966422047 >> 172.21.15.119:0/2966422047 conn(0x7fc03c076c60 msgr2=0x7fc03c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 -- 172.21.15.119:0/2966422047 shutdown_connections 2024-03-28T18:41:28.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.414+0000 7fc041a00640 1 -- 172.21.15.119:0/2966422047 wait complete. 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.415+0000 7fc041a00640 1 Processor -- start 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.415+0000 7fc041a00640 1 -- start start 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c108eb0 0x7fc03c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c109870 0x7fc03c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c113990 con 0x7fc03c108eb0 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c113810 con 0x7fc03c109870 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc041a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c113b10 con 0x7fc03c107cc0 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:55740/0 (socket says 172.21.15.119:55740) 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 -- 172.21.15.119:0/120802540 learned_addr learned my addr 172.21.15.119:0/120802540 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc032000640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c108eb0 0x7fc03c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03b400640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c109870 0x7fc03c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 -- 172.21.15.119:0/120802540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c109870 msgr2=0x7fc03c1a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c109870 0x7fc03c1a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 -- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c108eb0 msgr2=0x7fc03c19e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c108eb0 0x7fc03c19e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.416+0000 7fc03aa00640 1 -- 172.21.15.119:0/120802540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c1a5bd0 con 0x7fc03c107cc0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc03aa00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00e9c0 tx=0x7fc02c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00cd80 con 0x7fc03c107cc0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc02c0035f0 con 0x7fc03c107cc0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c1a5ec0 con 0x7fc03c107cc0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c010690 con 0x7fc03c107cc0 2024-03-28T18:41:28.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.417+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c1a62d0 con 0x7fc03c107cc0 2024-03-28T18:41:28.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.418+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c10ef80 con 0x7fc03c107cc0 2024-03-28T18:41:28.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.420+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc02c0037a0 con 0x7fc03c107cc0 2024-03-28T18:41:28.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.422+0000 7fc033e00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.422+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc02c0cedb0 con 0x7fc03c107cc0 2024-03-28T18:41:28.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.422+0000 7fc032000640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.423+0000 7fc032000640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc02402c250 tx=0x7fc02402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:28.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.428+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc02c09bf00 con 0x7fc03c107cc0 2024-03-28T18:41:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: pgmap v315: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-28T18:41:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:active 2024-03-28T18:41:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:41:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:41:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:41:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-28T18:41:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T18:41:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:28 smithi196 ceph-mon[36300]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: pgmap v315: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:active 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:41:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:41:28.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-28T18:41:28.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T18:41:28.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:28 smithi131 ceph-mon[36065]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-28T18:41:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: pgmap v315: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-28T18:41:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] up:active 2024-03-28T18:41:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T18:41:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T18:41:28.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:41:28.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-28T18:41:28.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-28T18:41:28.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:28 smithi119 ceph-mon[31813]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-28T18:41:28.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.657+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fc03c061ae0 con 0x7fc0200a2800 2024-03-28T18:41:28.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.688+0000 7fc033e00640 1 -- 172.21.15.119:0/120802540 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc03c061ae0 con 0x7fc0200a2800 2024-03-28T18:41:28.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 msgr2=0x7fc0200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc02402c250 tx=0x7fc02402cf90 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 msgr2=0x7fc03c19dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00e9c0 tx=0x7fc02c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 shutdown_connections 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc03c109870 0x7fc03c1a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03c108eb0 0x7fc03c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 --2- 172.21.15.119:0/120802540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc03c107cc0 0x7fc03c19dfb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.691+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 >> 172.21.15.119:0/120802540 conn(0x7fc03c076c60 msgr2=0x7fc03c10c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.692+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 shutdown_connections 2024-03-28T18:41:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.692+0000 7fc041a00640 1 -- 172.21.15.119:0/120802540 wait complete. 2024-03-28T18:41:28.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 -- 172.21.15.119:0/1982748908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 msgr2=0x7f8540104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 --2- 172.21.15.119:0/1982748908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f8540104f50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8528009900 tx=0x7f852802bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 -- 172.21.15.119:0/1982748908 shutdown_connections 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 --2- 172.21.15.119:0/1982748908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 0x7f854010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 --2- 172.21.15.119:0/1982748908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f8540104f50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 --2- 172.21.15.119:0/1982748908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 -- 172.21.15.119:0/1982748908 >> 172.21.15.119:0/1982748908 conn(0x7f85400fee50 msgr2=0x7f8540101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:28.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 -- 172.21.15.119:0/1982748908 shutdown_connections 2024-03-28T18:41:28.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.930+0000 7f8547400640 1 -- 172.21.15.119:0/1982748908 wait complete. 2024-03-28T18:41:28.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 Processor -- start 2024-03-28T18:41:28.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 -- start start 2024-03-28T18:41:28.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f854019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 0x7f85401a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854010f5b0 con 0x7f8540105490 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854010f430 con 0x7f8540104ad0 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8547400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854010f730 con 0x7f85401038e0 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8544a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8544a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39782/0 (socket says 172.21.15.119:39782) 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8544a00640 1 -- 172.21.15.119:0/4091250719 learned_addr learned my addr 172.21.15.119:0/4091250719 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8545400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 0x7f85401a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.931+0000 7f8537e00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f854019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 -- 172.21.15.119:0/4091250719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 msgr2=0x7f854019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f854019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 -- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 msgr2=0x7f85401a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 0x7f85401a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:28.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 -- 172.21.15.119:0/4091250719 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85401a1830 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8544a00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f853000e9c0 tx=0x7f853000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853000cd80 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85401a1b20 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85300035f0 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.932+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8530010690 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.933+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85401a2020 con 0x7f85401038e0 2024-03-28T18:41:28.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.934+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854010ac00 con 0x7f85401038e0 2024-03-28T18:41:28.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.935+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f85300037a0 con 0x7f85401038e0 2024-03-28T18:41:28.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.937+0000 7f853e000640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 0x7f85240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:28.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.937+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f85300ced70 con 0x7f85401038e0 2024-03-28T18:41:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.937+0000 7f8537e00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 0x7f85240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.938+0000 7f8537e00640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 0x7f85240a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f852802c240 tx=0x7f852802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:28.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:28.940+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f853009bec0 con 0x7f85401038e0 2024-03-28T18:41:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.176+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8540061ae0 con 0x7f85240a2800 2024-03-28T18:41:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.188+0000 7f853e000640 1 -- 172.21.15.119:0/4091250719 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f8540061ae0 con 0x7f85240a2800 2024-03-28T18:41:29.188 INFO:teuthology.orchestra.run.smithi119.stdout:/volumes/qa/sv_1/a7a67cbb-052a-445c-b58b-3818e042bf37 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 msgr2=0x7f85240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 0x7f85240a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f852802c240 tx=0x7f852802df90 comp rx=0 tx=0).stop 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 msgr2=0x7f8540199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f853000e9c0 tx=0x7f853000ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 shutdown_connections 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f85240a2800 0x7f85240a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8540105490 0x7f85401a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8540104ad0 0x7f854019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 --2- 172.21.15.119:0/4091250719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85401038e0 0x7f8540199c20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 >> 172.21.15.119:0/4091250719 conn(0x7f85400fee50 msgr2=0x7f8540107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 shutdown_connections 2024-03-28T18:41:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.191+0000 7f8547400640 1 -- 172.21.15.119:0/4091250719 wait complete. 2024-03-28T18:41:29.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 -- 172.21.15.119:0/778769127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40076060 msgr2=0x7fbd40076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 --2- 172.21.15.119:0/778769127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40076060 0x7fbd40076480 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009900 tx=0x7fbd3402bc30 comp rx=0 tx=0).stop 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 -- 172.21.15.119:0/778769127 shutdown_connections 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 --2- 172.21.15.119:0/778769127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40077c40 0x7fbd400784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 --2- 172.21.15.119:0/778769127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd40077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 --2- 172.21.15.119:0/778769127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40076060 0x7fbd40076480 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 -- 172.21.15.119:0/778769127 >> 172.21.15.119:0/778769127 conn(0x7fbd400fee50 msgr2=0x7fbd40101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 -- 172.21.15.119:0/778769127 shutdown_connections 2024-03-28T18:41:29.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.404+0000 7fbd47a00640 1 -- 172.21.15.119:0/778769127 wait complete. 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 Processor -- start 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 -- start start 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 0x7fbd4019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd401139b0 con 0x7fbd40076060 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40113830 con 0x7fbd40077c40 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.405+0000 7fbd47a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40113b30 con 0x7fbd400772a0 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 0x7fbd4019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46262/0 (socket says 172.21.15.119:46262) 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 0x7fbd4019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40924/0 (socket says 172.21.15.119:40924) 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45a00640 1 -- 172.21.15.119:0/1805839470 learned_addr learned my addr 172.21.15.119:0/1805839470 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39796/0 (socket says 172.21.15.119:39796) 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 -- 172.21.15.119:0/1805839470 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 msgr2=0x7fbd401a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 -- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 msgr2=0x7fbd4019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 0x7fbd4019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 -- 172.21.15.119:0/1805839470 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd401a5c20 con 0x7fbd400772a0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd45a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.406+0000 7fbd3fe00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9f0 tx=0x7fbd3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.407+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000ccc0 con 0x7fbd400772a0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.407+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd30005020 con 0x7fbd400772a0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.407+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd401a5f10 con 0x7fbd400772a0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.407+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300049e0 con 0x7fbd400772a0 2024-03-28T18:41:29.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.407+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd401a6380 con 0x7fbd400772a0 2024-03-28T18:41:29.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.408+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd401103f0 con 0x7fbd400772a0 2024-03-28T18:41:29.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.410+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbd30005560 con 0x7fbd400772a0 2024-03-28T18:41:29.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.411+0000 7fbd3d600640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 0x7fbd280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.411+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fbd300cebd0 con 0x7fbd400772a0 2024-03-28T18:41:29.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.412+0000 7fbd45000640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 0x7fbd280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.412+0000 7fbd45000640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 0x7fbd280a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbd3402c240 tx=0x7fbd3402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.418+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd300051d0 con 0x7fbd400772a0 2024-03-28T18:41:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:active 2024-03-28T18:41:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T18:41:29.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-28T18:41:29.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:29 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-28T18:41:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:active 2024-03-28T18:41:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-28T18:41:29.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:29 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-28T18:41:29.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.658+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd4002d2c0 con 0x7fbd400772a0 2024-03-28T18:41:29.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.659+0000 7fbd3d600640 1 -- 172.21.15.119:0/1805839470 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fbd3009bb30 con 0x7fbd400772a0 2024-03-28T18:41:29.660 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:29.660 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:41:29.660 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:41:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 msgr2=0x7fbd280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 0x7fbd280a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbd3402c240 tx=0x7fbd3402df90 comp rx=0 tx=0).stop 2024-03-28T18:41:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 msgr2=0x7fbd4019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9f0 tx=0x7fbd3000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 shutdown_connections 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fbd280a2890 0x7fbd280a4d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd40077c40 0x7fbd401a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd400772a0 0x7fbd4019e540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 --2- 172.21.15.119:0/1805839470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd40076060 0x7fbd4019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 >> 172.21.15.119:0/1805839470 conn(0x7fbd400fee50 msgr2=0x7fbd401005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 shutdown_connections 2024-03-28T18:41:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.662+0000 7fbd47a00640 1 -- 172.21.15.119:0/1805839470 wait complete. 2024-03-28T18:41:29.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:41:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] up:active 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-03-28T18:41:29.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:29 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.888+0000 7f34e1400640 1 -- 172.21.15.119:0/2200800915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc109890 msgr2=0x7f34dc110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.888+0000 7f34e1400640 1 --2- 172.21.15.119:0/2200800915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc109890 0x7f34dc110120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009ca0 tx=0x7f34d402c100 comp rx=0 tx=0).stop 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 -- 172.21.15.119:0/2200800915 shutdown_connections 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 --2- 172.21.15.119:0/2200800915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc109890 0x7f34dc110120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 --2- 172.21.15.119:0/2200800915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 0x7f34dc109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 --2- 172.21.15.119:0/2200800915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc107ce0 0x7f34dc108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 -- 172.21.15.119:0/2200800915 >> 172.21.15.119:0/2200800915 conn(0x7f34dc076cb0 msgr2=0x7f34dc0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 -- 172.21.15.119:0/2200800915 shutdown_connections 2024-03-28T18:41:29.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.889+0000 7f34e1400640 1 -- 172.21.15.119:0/2200800915 wait complete. 2024-03-28T18:41:29.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 Processor -- start 2024-03-28T18:41:29.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 -- start start 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 0x7f34dc19e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc109890 0x7f34dc1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc113960 con 0x7f34dc108ed0 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc1137e0 con 0x7f34dc109890 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34e1400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc113ae0 con 0x7f34dc107ce0 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34da800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34da800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39816/0 (socket says 172.21.15.119:39816) 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34da800640 1 -- 172.21.15.119:0/4100932948 learned_addr learned my addr 172.21.15.119:0/4100932948 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34d1e00640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 0x7f34dc19e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.890+0000 7f34db200640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc109890 0x7f34dc1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 -- 172.21.15.119:0/4100932948 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc109890 msgr2=0x7f34dc1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc109890 0x7f34dc1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 -- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 msgr2=0x7f34dc19e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 0x7f34dc19e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 -- 172.21.15.119:0/4100932948 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc1a5b90 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34da800640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00e9c0 tx=0x7f34cc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00cd80 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34cc0035f0 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc010690 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34dc1a5e80 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.891+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc1a6290 con 0x7f34dc107ce0 2024-03-28T18:41:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.892+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc10efa0 con 0x7f34dc107ce0 2024-03-28T18:41:29.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.894+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f34cc0037a0 con 0x7f34dc107ce0 2024-03-28T18:41:29.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.895+0000 7f34d3e00640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 0x7f34c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:29.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.895+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f34cc0cefe0 con 0x7f34dc107ce0 2024-03-28T18:41:29.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.895+0000 7f34d1e00640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 0x7f34c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:29.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.896+0000 7f34d1e00640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 0x7f34c00a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34c4005fd0 tx=0x7f34c4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:29.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:29.899+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34cc09beb0 con 0x7f34dc107ce0 2024-03-28T18:41:30.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.098+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34dc0623e0 con 0x7f34dc107ce0 2024-03-28T18:41:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.100+0000 7f34d3e00640 1 -- 172.21.15.119:0/4100932948 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+22776 (secure 0 0 0) 0x7f34cc09b7b0 con 0x7f34dc107ce0 2024-03-28T18:41:30.100 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:30.100 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":87,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:41:29.196114+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:30.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.102+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 msgr2=0x7f34c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.102+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 0x7f34c00a4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34c4005fd0 tx=0x7f34c4005850 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.102+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 msgr2=0x7f34dc19df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.102+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00e9c0 tx=0x7f34cc00ee80 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 shutdown_connections 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f34c00a2910 0x7f34c00a4dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34dc109890 0x7f34dc1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34dc108ed0 0x7f34dc19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 --2- 172.21.15.119:0/4100932948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34dc107ce0 0x7f34dc19df70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 >> 172.21.15.119:0/4100932948 conn(0x7f34dc076cb0 msgr2=0x7f34dc10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 shutdown_connections 2024-03-28T18:41:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.103+0000 7f34e1400640 1 -- 172.21.15.119:0/4100932948 wait complete. 2024-03-28T18:41:30.121 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T18:41:30.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:41:30.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 -- 172.21.15.119:0/3754104010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 msgr2=0x7fd7540784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 --2- 172.21.15.119:0/3754104010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7540784e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009ca0 tx=0x7fd74c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 -- 172.21.15.119:0/3754104010 shutdown_connections 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 --2- 172.21.15.119:0/3754104010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7540784e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 --2- 172.21.15.119:0/3754104010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd754077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 --2- 172.21.15.119:0/3754104010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 0x7fd754076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.331+0000 7fd758e00640 1 -- 172.21.15.119:0/3754104010 >> 172.21.15.119:0/3754104010 conn(0x7fd7540fee50 msgr2=0x7fd754101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.332+0000 7fd758e00640 1 -- 172.21.15.119:0/3754104010 shutdown_connections 2024-03-28T18:41:30.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.332+0000 7fd758e00640 1 -- 172.21.15.119:0/3754104010 wait complete. 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 Processor -- start 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 -- start start 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 0x7fd75419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7541a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754113960 con 0x7fd754076060 2024-03-28T18:41:30.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7541137e0 con 0x7fd7540772a0 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd758e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754113ae0 con 0x7fd754077c40 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd751400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd751e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 0x7fd75419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd752800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7541a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd751400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46286/0 (socket says 172.21.15.119:46286) 2024-03-28T18:41:30.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.333+0000 7fd751400640 1 -- 172.21.15.119:0/1542017170 learned_addr learned my addr 172.21.15.119:0/1542017170 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 -- 172.21.15.119:0/1542017170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 msgr2=0x7fd7541a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7541a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 -- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 msgr2=0x7fd75419df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 0x7fd75419df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 -- 172.21.15.119:0/1542017170 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7541a5b70 con 0x7fd7540772a0 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd752800640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7541a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd751400640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e9f0 tx=0x7fd74000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74000cd80 con 0x7fd7540772a0 2024-03-28T18:41:30.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7541a5e60 con 0x7fd7540772a0 2024-03-28T18:41:30.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd740005500 con 0x7fd7540772a0 2024-03-28T18:41:30.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.334+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740010690 con 0x7fd7540772a0 2024-03-28T18:41:30.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.335+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7541a6320 con 0x7fd7540772a0 2024-03-28T18:41:30.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.336+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7541103a0 con 0x7fd7540772a0 2024-03-28T18:41:30.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.338+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd7400046c0 con 0x7fd7540772a0 2024-03-28T18:41:30.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.339+0000 7fd74aa00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 0x7fd7380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.340+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fd7400ceef0 con 0x7fd7540772a0 2024-03-28T18:41:30.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.340+0000 7fd751e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 0x7fd7380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.340+0000 7fd751e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 0x7fd7380a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd744005d00 tx=0x7fd74400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.348+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd74009bdc0 con 0x7fd7540772a0 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: 2024-03-28T18:41:30.020+0000 7f0c83c48700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: 2024-03-28T18:41:30.020+0000 7f0c83c48700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: pgmap v316: 97 pgs: 97 active+clean; 590 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s wr, 21 op/s 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]': finished 2024-03-28T18:41:30.392 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1805839470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:30 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4100932948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.546+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd754061ae0 con 0x7fd7380a2910 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: pgmap v316: 97 pgs: 97 active+clean; 590 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s wr, 21 op/s 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-03-28T18:41:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1805839470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4100932948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.574+0000 7fd74aa00640 1 -- 172.21.15.119:0/1542017170 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258898 (secure 0 0 0) 0x7fd754061ae0 con 0x7fd7380a2910 2024-03-28T18:41:30.576 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:30.579 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 msgr2=0x7fd7380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 0x7fd7380a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd744005d00 tx=0x7fd74400b410 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 msgr2=0x7fd75419e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e9f0 tx=0x7fd74000edf0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 shutdown_connections 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7380a2910 0x7fd7380a4dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd754077c40 0x7fd7541a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7540772a0 0x7fd75419e490 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 --2- 172.21.15.119:0/1542017170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd754076060 0x7fd75419df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 >> 172.21.15.119:0/1542017170 conn(0x7fd7540fee50 msgr2=0x7fd754100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:30.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 shutdown_connections 2024-03-28T18:41:30.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.581+0000 7fd758e00640 1 -- 172.21.15.119:0/1542017170 wait complete. 2024-03-28T18:41:30.601 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-03-28T18:41:30.210049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605022,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":365,"num_write_kb":3007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360468,"kb_used_data":11308,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581356,"statfs":{"total":1151940427776,"available":1151571308544,"internally_reserved":0,"allocated":11579392,"data_stored":7121329,"data_compressed":110325,"data_compressed_allocated":139264,"data_compressed_original":1777664,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14654,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":184,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.135048"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902971+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:41:22.902971+0000","last_peered":"2024-03-28T18:41:22.902971+0000","last_clean":"2024-03-28T18:41:22.902971+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:41:22.902971+0000","last_undegraded":"2024-03-28T18:41:22.902971+0000","last_fullsized":"2024-03-28T18:41:22.902971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721874+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:41:22.721874+0000","last_peered":"2024-03-28T18:41:22.721874+0000","last_clean":"2024-03-28T18:41:22.721874+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:41:22.721874+0000","last_undegraded":"2024-03-28T18:41:22.721874+0000","last_fullsized":"2024-03-28T18:41:22.721874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721500+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:41:22.721500+0000","last_peered":"2024-03-28T18:41:22.721500+0000","last_clean":"2024-03-28T18:41:22.721500+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:41:22.721500+0000","last_undegraded":"2024-03-28T18:41:22.721500+0000","last_fullsized":"2024-03-28T18:41:22.721500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214247+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:41:29.214247+0000","last_peered":"2024-03-28T18:41:29.214247+0000","last_clean":"2024-03-28T18:41:29.214247+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:41:29.214247+0000","last_undegraded":"2024-03-28T18:41:29.214247+0000","last_fullsized":"2024-03-28T18:41:29.214247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114546+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:41:22.114546+0000","last_peered":"2024-03-28T18:41:22.114546+0000","last_clean":"2024-03-28T18:41:22.114546+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:41:22.114546+0000","last_undegraded":"2024-03-28T18:41:22.114546+0000","last_fullsized":"2024-03-28T18:41:22.114546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720137+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:41:22.720137+0000","last_peered":"2024-03-28T18:41:22.720137+0000","last_clean":"2024-03-28T18:41:22.720137+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:41:22.720137+0000","last_undegraded":"2024-03-28T18:41:22.720137+0000","last_fullsized":"2024-03-28T18:41:22.720137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718960+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:41:22.718960+0000","last_peered":"2024-03-28T18:41:22.718960+0000","last_clean":"2024-03-28T18:41:22.718960+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:41:22.718960+0000","last_undegraded":"2024-03-28T18:41:22.718960+0000","last_fullsized":"2024-03-28T18:41:22.718960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077097+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:41:22.077097+0000","last_peered":"2024-03-28T18:41:22.077097+0000","last_clean":"2024-03-28T18:41:22.077097+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:41:22.077097+0000","last_undegraded":"2024-03-28T18:41:22.077097+0000","last_fullsized":"2024-03-28T18:41:22.077097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434539+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:41:22.434539+0000","last_peered":"2024-03-28T18:41:22.434539+0000","last_clean":"2024-03-28T18:41:22.434539+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:41:22.434539+0000","last_undegraded":"2024-03-28T18:41:22.434539+0000","last_fullsized":"2024-03-28T18:41:22.434539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720496+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:41:22.720496+0000","last_peered":"2024-03-28T18:41:22.720496+0000","last_clean":"2024-03-28T18:41:22.720496+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:41:22.720496+0000","last_undegraded":"2024-03-28T18:41:22.720496+0000","last_fullsized":"2024-03-28T18:41:22.720496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719448+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:41:22.719448+0000","last_peered":"2024-03-28T18:41:22.719448+0000","last_clean":"2024-03-28T18:41:22.719448+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:41:22.719448+0000","last_undegraded":"2024-03-28T18:41:22.719448+0000","last_fullsized":"2024-03-28T18:41:22.719448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214154+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:41:29.214154+0000","last_peered":"2024-03-28T18:41:29.214154+0000","last_clean":"2024-03-28T18:41:29.214154+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:41:29.214154+0000","last_undegraded":"2024-03-28T18:41:29.214154+0000","last_fullsized":"2024-03-28T18:41:29.214154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.080006+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:41:22.080006+0000","last_peered":"2024-03-28T18:41:22.080006+0000","last_clean":"2024-03-28T18:41:22.080006+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:41:22.080006+0000","last_undegraded":"2024-03-28T18:41:22.080006+0000","last_fullsized":"2024-03-28T18:41:22.080006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720845+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:41:22.720845+0000","last_peered":"2024-03-28T18:41:22.720845+0000","last_clean":"2024-03-28T18:41:22.720845+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:41:22.720845+0000","last_undegraded":"2024-03-28T18:41:22.720845+0000","last_fullsized":"2024-03-28T18:41:22.720845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720785+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:41:22.720785+0000","last_peered":"2024-03-28T18:41:22.720785+0000","last_clean":"2024-03-28T18:41:22.720785+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:41:22.720785+0000","last_undegraded":"2024-03-28T18:41:22.720785+0000","last_fullsized":"2024-03-28T18:41:22.720785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434173+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:41:22.434173+0000","last_peered":"2024-03-28T18:41:22.434173+0000","last_clean":"2024-03-28T18:41:22.434173+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:41:22.434173+0000","last_undegraded":"2024-03-28T18:41:22.434173+0000","last_fullsized":"2024-03-28T18:41:22.434173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902956+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:41:22.902956+0000","last_peered":"2024-03-28T18:41:22.902956+0000","last_clean":"2024-03-28T18:41:22.902956+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:41:22.902956+0000","last_undegraded":"2024-03-28T18:41:22.902956+0000","last_fullsized":"2024-03-28T18:41:22.902956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.111908+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:41:22.111908+0000","last_peered":"2024-03-28T18:41:22.111908+0000","last_clean":"2024-03-28T18:41:22.111908+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:41:22.111908+0000","last_undegraded":"2024-03-28T18:41:22.111908+0000","last_fullsized":"2024-03-28T18:41:22.111908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719361+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:41:22.719361+0000","last_peered":"2024-03-28T18:41:22.719361+0000","last_clean":"2024-03-28T18:41:22.719361+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:41:22.719361+0000","last_undegraded":"2024-03-28T18:41:22.719361+0000","last_fullsized":"2024-03-28T18:41:22.719361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.158395+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:41:25.158395+0000","last_peered":"2024-03-28T18:41:25.158395+0000","last_clean":"2024-03-28T18:41:25.158395+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:41:25.158395+0000","last_undegraded":"2024-03-28T18:41:25.158395+0000","last_fullsized":"2024-03-28T18:41:25.158395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077994+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:41:22.077994+0000","last_peered":"2024-03-28T18:41:22.077994+0000","last_clean":"2024-03-28T18:41:22.077994+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:41:22.077994+0000","last_undegraded":"2024-03-28T18:41:22.077994+0000","last_fullsized":"2024-03-28T18:41:22.077994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:23.499460+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:41:23.499460+0000","last_peered":"2024-03-28T18:41:23.499460+0000","last_clean":"2024-03-28T18:41:23.499460+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:41:23.499460+0000","last_undegraded":"2024-03-28T18:41:23.499460+0000","last_fullsized":"2024-03-28T18:41:23.499460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721706+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:41:22.721706+0000","last_peered":"2024-03-28T18:41:22.721706+0000","last_clean":"2024-03-28T18:41:22.721706+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:41:22.721706+0000","last_undegraded":"2024-03-28T18:41:22.721706+0000","last_fullsized":"2024-03-28T18:41:22.721706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.167291+0000","last_change":"2024-03-28T18:41:20.067522+0000","last_active":"2024-03-28T18:41:26.167291+0000","last_peered":"2024-03-28T18:41:26.167291+0000","last_clean":"2024-03-28T18:41:26.167291+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:41:26.167291+0000","last_undegraded":"2024-03-28T18:41:26.167291+0000","last_fullsized":"2024-03-28T18:41:26.167291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:47.011842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434474+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:41:22.434474+0000","last_peered":"2024-03-28T18:41:22.434474+0000","last_clean":"2024-03-28T18:41:22.434474+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:41:22.434474+0000","last_undegraded":"2024-03-28T18:41:22.434474+0000","last_fullsized":"2024-03-28T18:41:22.434474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114558+0000","last_change":"2024-03-28T18:41:20.069156+0000","last_active":"2024-03-28T18:41:22.114558+0000","last_peered":"2024-03-28T18:41:22.114558+0000","last_clean":"2024-03-28T18:41:22.114558+0000","last_became_active":"2024-03-28T18:41:20.068839+0000","last_became_peered":"2024-03-28T18:41:20.068839+0000","last_unstale":"2024-03-28T18:41:22.114558+0000","last_undegraded":"2024-03-28T18:41:22.114558+0000","last_fullsized":"2024-03-28T18:41:22.114558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:55.493517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076763+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:41:22.076763+0000","last_peered":"2024-03-28T18:41:22.076763+0000","last_clean":"2024-03-28T18:41:22.076763+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:41:22.076763+0000","last_undegraded":"2024-03-28T18:41:22.076763+0000","last_fullsized":"2024-03-28T18:41:22.076763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.146054+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:41:24.146054+0000","last_peered":"2024-03-28T18:41:24.146054+0000","last_clean":"2024-03-28T18:41:24.146054+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:41:24.146054+0000","last_undegraded":"2024-03-28T18:41:24.146054+0000","last_fullsized":"2024-03-28T18:41:24.146054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114307+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:41:22.114307+0000","last_peered":"2024-03-28T18:41:22.114307+0000","last_clean":"2024-03-28T18:41:22.114307+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:41:22.114307+0000","last_undegraded":"2024-03-28T18:41:22.114307+0000","last_fullsized":"2024-03-28T18:41:22.114307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.174594+0000","last_change":"2024-03-28T18:41:20.065845+0000","last_active":"2024-03-28T18:41:26.174594+0000","last_peered":"2024-03-28T18:41:26.174594+0000","last_clean":"2024-03-28T18:41:26.174594+0000","last_became_active":"2024-03-28T18:41:20.065495+0000","last_became_peered":"2024-03-28T18:41:20.065495+0000","last_unstale":"2024-03-28T18:41:26.174594+0000","last_undegraded":"2024-03-28T18:41:26.174594+0000","last_fullsized":"2024-03-28T18:41:26.174594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:11.078525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076799+0000","last_change":"2024-03-28T18:41:21.115518+0000","last_active":"2024-03-28T18:41:22.076799+0000","last_peered":"2024-03-28T18:41:22.076799+0000","last_clean":"2024-03-28T18:41:22.076799+0000","last_became_active":"2024-03-28T18:41:21.115164+0000","last_became_peered":"2024-03-28T18:41:21.115164+0000","last_unstale":"2024-03-28T18:41:22.076799+0000","last_undegraded":"2024-03-28T18:41:22.076799+0000","last_fullsized":"2024-03-28T18:41:22.076799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:24:30.331524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.161607+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:41:25.161607+0000","last_peered":"2024-03-28T18:41:25.161607+0000","last_clean":"2024-03-28T18:41:25.161607+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:41:25.161607+0000","last_undegraded":"2024-03-28T18:41:25.161607+0000","last_fullsized":"2024-03-28T18:41:25.161607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076978+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:41:22.076978+0000","last_peered":"2024-03-28T18:41:22.076978+0000","last_clean":"2024-03-28T18:41:22.076978+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:41:22.076978+0000","last_undegraded":"2024-03-28T18:41:22.076978+0000","last_fullsized":"2024-03-28T18:41:22.076978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.165830+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:41:26.165830+0000","last_peered":"2024-03-28T18:41:26.165830+0000","last_clean":"2024-03-28T18:41:26.165830+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:41:26.165830+0000","last_undegraded":"2024-03-28T18:41:26.165830+0000","last_fullsized":"2024-03-28T18:41:26.165830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.111849+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:41:22.111849+0000","last_peered":"2024-03-28T18:41:22.111849+0000","last_clean":"2024-03-28T18:41:22.111849+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:41:22.111849+0000","last_undegraded":"2024-03-28T18:41:22.111849+0000","last_fullsized":"2024-03-28T18:41:22.111849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.145174+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:41:24.145174+0000","last_peered":"2024-03-28T18:41:24.145174+0000","last_clean":"2024-03-28T18:41:24.145174+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:41:24.145174+0000","last_undegraded":"2024-03-28T18:41:24.145174+0000","last_fullsized":"2024-03-28T18:41:24.145174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434459+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:41:22.434459+0000","last_peered":"2024-03-28T18:41:22.434459+0000","last_clean":"2024-03-28T18:41:22.434459+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:41:22.434459+0000","last_undegraded":"2024-03-28T18:41:22.434459+0000","last_fullsized":"2024-03-28T18:41:22.434459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.188968+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:41:27.188968+0000","last_peered":"2024-03-28T18:41:27.188968+0000","last_clean":"2024-03-28T18:41:27.188968+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:41:27.188968+0000","last_undegraded":"2024-03-28T18:41:27.188968+0000","last_fullsized":"2024-03-28T18:41:27.188968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721883+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:41:22.721883+0000","last_peered":"2024-03-28T18:41:22.721883+0000","last_clean":"2024-03-28T18:41:22.721883+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:41:22.721883+0000","last_undegraded":"2024-03-28T18:41:22.721883+0000","last_fullsized":"2024-03-28T18:41:22.721883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.146273+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:41:24.146273+0000","last_peered":"2024-03-28T18:41:24.146273+0000","last_clean":"2024-03-28T18:41:24.146273+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:41:24.146273+0000","last_undegraded":"2024-03-28T18:41:24.146273+0000","last_fullsized":"2024-03-28T18:41:24.146273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434633+0000","last_change":"2024-03-28T18:41:21.094009+0000","last_active":"2024-03-28T18:41:22.434633+0000","last_peered":"2024-03-28T18:41:22.434633+0000","last_clean":"2024-03-28T18:41:22.434633+0000","last_became_active":"2024-03-28T18:41:21.093693+0000","last_became_peered":"2024-03-28T18:41:21.093693+0000","last_unstale":"2024-03-28T18:41:22.434633+0000","last_undegraded":"2024-03-28T18:41:22.434633+0000","last_fullsized":"2024-03-28T18:41:22.434633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:53.427462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'13","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.188386+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:41:27.188386+0000","last_peered":"2024-03-28T18:41:27.188386+0000","last_clean":"2024-03-28T18:41:27.188386+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:41:27.188386+0000","last_undegraded":"2024-03-28T18:41:27.188386+0000","last_fullsized":"2024-03-28T18:41:27.188386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114560+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:41:22.114560+0000","last_peered":"2024-03-28T18:41:22.114560+0000","last_clean":"2024-03-28T18:41:22.114560+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:41:22.114560+0000","last_undegraded":"2024-03-28T18:41:22.114560+0000","last_fullsized":"2024-03-28T18:41:22.114560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.159524+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:41:25.159524+0000","last_peered":"2024-03-28T18:41:25.159524+0000","last_clean":"2024-03-28T18:41:25.159524+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:41:25.159524+0000","last_undegraded":"2024-03-28T18:41:25.159524+0000","last_fullsized":"2024-03-28T18:41:25.159524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079807+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:41:22.079807+0000","last_peered":"2024-03-28T18:41:22.079807+0000","last_clean":"2024-03-28T18:41:22.079807+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:41:22.079807+0000","last_undegraded":"2024-03-28T18:41:22.079807+0000","last_fullsized":"2024-03-28T18:41:22.079807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.171233+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:41:26.171233+0000","last_peered":"2024-03-28T18:41:26.171233+0000","last_clean":"2024-03-28T18:41:26.171233+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:41:26.171233+0000","last_undegraded":"2024-03-28T18:41:26.171233+0000","last_fullsized":"2024-03-28T18:41:26.171233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719217+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:41:22.719217+0000","last_peered":"2024-03-28T18:41:22.719217+0000","last_clean":"2024-03-28T18:41:22.719217+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:41:22.719217+0000","last_undegraded":"2024-03-28T18:41:22.719217+0000","last_fullsized":"2024-03-28T18:41:22.719217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214507+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:41:29.214507+0000","last_peered":"2024-03-28T18:41:29.214507+0000","last_clean":"2024-03-28T18:41:29.214507+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:41:29.214507+0000","last_undegraded":"2024-03-28T18:41:29.214507+0000","last_fullsized":"2024-03-28T18:41:29.214507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718705+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:41:22.718705+0000","last_peered":"2024-03-28T18:41:22.718705+0000","last_clean":"2024-03-28T18:41:22.718705+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:41:22.718705+0000","last_undegraded":"2024-03-28T18:41:22.718705+0000","last_fullsized":"2024-03-28T18:41:22.718705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.158372+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:41:25.158372+0000","last_peered":"2024-03-28T18:41:25.158372+0000","last_clean":"2024-03-28T18:41:25.158372+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:41:25.158372+0000","last_undegraded":"2024-03-28T18:41:25.158372+0000","last_fullsized":"2024-03-28T18:41:25.158372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076618+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:41:22.076618+0000","last_peered":"2024-03-28T18:41:22.076618+0000","last_clean":"2024-03-28T18:41:22.076618+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:41:22.076618+0000","last_undegraded":"2024-03-28T18:41:22.076618+0000","last_fullsized":"2024-03-28T18:41:22.076618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.180141+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:41:22.180141+0000","last_peered":"2024-03-28T18:41:22.180141+0000","last_clean":"2024-03-28T18:41:22.180141+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:41:22.180141+0000","last_undegraded":"2024-03-28T18:41:22.180141+0000","last_fullsized":"2024-03-28T18:41:22.180141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902783+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:41:22.902783+0000","last_peered":"2024-03-28T18:41:22.902783+0000","last_clean":"2024-03-28T18:41:22.902783+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:41:22.902783+0000","last_undegraded":"2024-03-28T18:41:22.902783+0000","last_fullsized":"2024-03-28T18:41:22.902783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.111761+0000","last_change":"2024-03-28T18:41:20.082452+0000","last_active":"2024-03-28T18:41:22.111761+0000","last_peered":"2024-03-28T18:41:22.111761+0000","last_clean":"2024-03-28T18:41:22.111761+0000","last_became_active":"2024-03-28T18:41:20.081083+0000","last_became_peered":"2024-03-28T18:41:20.081083+0000","last_unstale":"2024-03-28T18:41:22.111761+0000","last_undegraded":"2024-03-28T18:41:22.111761+0000","last_fullsized":"2024-03-28T18:41:22.111761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:57:49.530663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720417+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:41:22.720417+0000","last_peered":"2024-03-28T18:41:22.720417+0000","last_clean":"2024-03-28T18:41:22.720417+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:41:22.720417+0000","last_undegraded":"2024-03-28T18:41:22.720417+0000","last_fullsized":"2024-03-28T18:41:22.720417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.193538+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:41:27.193538+0000","last_peered":"2024-03-28T18:41:27.193538+0000","last_clean":"2024-03-28T18:41:27.193538+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:41:27.193538+0000","last_undegraded":"2024-03-28T18:41:27.193538+0000","last_fullsized":"2024-03-28T18:41:27.193538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720296+0000","last_change":"2024-03-28T18:41:21.119477+0000","last_active":"2024-03-28T18:41:22.720296+0000","last_peered":"2024-03-28T18:41:22.720296+0000","last_clean":"2024-03-28T18:41:22.720296+0000","last_became_active":"2024-03-28T18:41:21.119121+0000","last_became_peered":"2024-03-28T18:41:21.119121+0000","last_unstale":"2024-03-28T18:41:22.720296+0000","last_undegraded":"2024-03-28T18:41:22.720296+0000","last_fullsized":"2024-03-28T18:41:22.720296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:14:06.510122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076758+0000","last_change":"2024-03-28T18:41:21.097489+0000","last_active":"2024-03-28T18:41:22.076758+0000","last_peered":"2024-03-28T18:41:22.076758+0000","last_clean":"2024-03-28T18:41:22.076758+0000","last_became_active":"2024-03-28T18:41:21.097079+0000","last_became_peered":"2024-03-28T18:41:21.097079+0000","last_unstale":"2024-03-28T18:41:22.076758+0000","last_undegraded":"2024-03-28T18:41:22.076758+0000","last_fullsized":"2024-03-28T18:41:22.076758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:49.130304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.163837+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:41:25.163837+0000","last_peered":"2024-03-28T18:41:25.163837+0000","last_clean":"2024-03-28T18:41:25.163837+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:41:25.163837+0000","last_undegraded":"2024-03-28T18:41:25.163837+0000","last_fullsized":"2024-03-28T18:41:25.163837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078015+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:41:22.078015+0000","last_peered":"2024-03-28T18:41:22.078015+0000","last_clean":"2024-03-28T18:41:22.078015+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:41:22.078015+0000","last_undegraded":"2024-03-28T18:41:22.078015+0000","last_fullsized":"2024-03-28T18:41:22.078015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214319+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:41:29.214319+0000","last_peered":"2024-03-28T18:41:29.214319+0000","last_clean":"2024-03-28T18:41:29.214319+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:41:29.214319+0000","last_undegraded":"2024-03-28T18:41:29.214319+0000","last_fullsized":"2024-03-28T18:41:29.214319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718819+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:41:22.718819+0000","last_peered":"2024-03-28T18:41:22.718819+0000","last_clean":"2024-03-28T18:41:22.718819+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:41:22.718819+0000","last_undegraded":"2024-03-28T18:41:22.718819+0000","last_fullsized":"2024-03-28T18:41:22.718819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077332+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:41:22.077332+0000","last_peered":"2024-03-28T18:41:22.077332+0000","last_clean":"2024-03-28T18:41:22.077332+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:41:22.077332+0000","last_undegraded":"2024-03-28T18:41:22.077332+0000","last_fullsized":"2024-03-28T18:41:22.077332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.154441+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:41:25.154441+0000","last_peered":"2024-03-28T18:41:25.154441+0000","last_clean":"2024-03-28T18:41:25.154441+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:41:25.154441+0000","last_undegraded":"2024-03-28T18:41:25.154441+0000","last_fullsized":"2024-03-28T18:41:25.154441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.169515+0000","last_change":"2024-03-28T18:41:20.079909+0000","last_active":"2024-03-28T18:41:26.169515+0000","last_peered":"2024-03-28T18:41:26.169515+0000","last_clean":"2024-03-28T18:41:26.169515+0000","last_became_active":"2024-03-28T18:41:20.079658+0000","last_became_peered":"2024-03-28T18:41:20.079658+0000","last_unstale":"2024-03-28T18:41:26.169515+0000","last_undegraded":"2024-03-28T18:41:26.169515+0000","last_fullsized":"2024-03-28T18:41:26.169515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:14.211815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079821+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:41:22.079821+0000","last_peered":"2024-03-28T18:41:22.079821+0000","last_clean":"2024-03-28T18:41:22.079821+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:41:22.079821+0000","last_undegraded":"2024-03-28T18:41:22.079821+0000","last_fullsized":"2024-03-28T18:41:22.079821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.159473+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:41:25.159473+0000","last_peered":"2024-03-28T18:41:25.159473+0000","last_clean":"2024-03-28T18:41:25.159473+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:41:25.159473+0000","last_undegraded":"2024-03-28T18:41:25.159473+0000","last_fullsized":"2024-03-28T18:41:25.159473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078255+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:41:22.078255+0000","last_peered":"2024-03-28T18:41:22.078255+0000","last_clean":"2024-03-28T18:41:22.078255+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:41:22.078255+0000","last_undegraded":"2024-03-28T18:41:22.078255+0000","last_fullsized":"2024-03-28T18:41:22.078255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.213454+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:41:29.213454+0000","last_peered":"2024-03-28T18:41:29.213454+0000","last_clean":"2024-03-28T18:41:29.213454+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:41:29.213454+0000","last_undegraded":"2024-03-28T18:41:29.213454+0000","last_fullsized":"2024-03-28T18:41:29.213454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721926+0000","last_change":"2024-03-28T18:41:21.125927+0000","last_active":"2024-03-28T18:41:22.721926+0000","last_peered":"2024-03-28T18:41:22.721926+0000","last_clean":"2024-03-28T18:41:22.721926+0000","last_became_active":"2024-03-28T18:41:21.125516+0000","last_became_peered":"2024-03-28T18:41:21.125516+0000","last_unstale":"2024-03-28T18:41:22.721926+0000","last_undegraded":"2024-03-28T18:41:22.721926+0000","last_fullsized":"2024-03-28T18:41:22.721926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:20.224459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.145579+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:41:24.145579+0000","last_peered":"2024-03-28T18:41:24.145579+0000","last_clean":"2024-03-28T18:41:24.145579+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:41:24.145579+0000","last_undegraded":"2024-03-28T18:41:24.145579+0000","last_fullsized":"2024-03-28T18:41:24.145579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114360+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:41:22.114360+0000","last_peered":"2024-03-28T18:41:22.114360+0000","last_clean":"2024-03-28T18:41:22.114360+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:41:22.114360+0000","last_undegraded":"2024-03-28T18:41:22.114360+0000","last_fullsized":"2024-03-28T18:41:22.114360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.199496+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:41:27.199496+0000","last_peered":"2024-03-28T18:41:27.199496+0000","last_clean":"2024-03-28T18:41:27.199496+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:41:27.199496+0000","last_undegraded":"2024-03-28T18:41:27.199496+0000","last_fullsized":"2024-03-28T18:41:27.199496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902566+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:41:22.902566+0000","last_peered":"2024-03-28T18:41:22.902566+0000","last_clean":"2024-03-28T18:41:22.902566+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:41:22.902566+0000","last_undegraded":"2024-03-28T18:41:22.902566+0000","last_fullsized":"2024-03-28T18:41:22.902566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.140419+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:41:26.140419+0000","last_peered":"2024-03-28T18:41:26.140419+0000","last_clean":"2024-03-28T18:41:26.140419+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:41:26.140419+0000","last_undegraded":"2024-03-28T18:41:26.140419+0000","last_fullsized":"2024-03-28T18:41:26.140419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078223+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:41:22.078223+0000","last_peered":"2024-03-28T18:41:22.078223+0000","last_clean":"2024-03-28T18:41:22.078223+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:41:22.078223+0000","last_undegraded":"2024-03-28T18:41:22.078223+0000","last_fullsized":"2024-03-28T18:41:22.078223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.143807+0000","last_change":"2024-03-28T18:41:20.067104+0000","last_active":"2024-03-28T18:41:24.143807+0000","last_peered":"2024-03-28T18:41:24.143807+0000","last_clean":"2024-03-28T18:41:24.143807+0000","last_became_active":"2024-03-28T18:41:20.066686+0000","last_became_peered":"2024-03-28T18:41:20.066686+0000","last_unstale":"2024-03-28T18:41:24.143807+0000","last_undegraded":"2024-03-28T18:41:24.143807+0000","last_fullsized":"2024-03-28T18:41:24.143807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:33.049170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719391+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:41:22.719391+0000","last_peered":"2024-03-28T18:41:22.719391+0000","last_clean":"2024-03-28T18:41:22.719391+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:41:22.719391+0000","last_undegraded":"2024-03-28T18:41:22.719391+0000","last_fullsized":"2024-03-28T18:41:22.719391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.168589+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:41:26.168589+0000","last_peered":"2024-03-28T18:41:26.168589+0000","last_clean":"2024-03-28T18:41:26.168589+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:41:26.168589+0000","last_undegraded":"2024-03-28T18:41:26.168589+0000","last_fullsized":"2024-03-28T18:41:26.168589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719556+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:41:22.719556+0000","last_peered":"2024-03-28T18:41:22.719556+0000","last_clean":"2024-03-28T18:41:22.719556+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:41:22.719556+0000","last_undegraded":"2024-03-28T18:41:22.719556+0000","last_fullsized":"2024-03-28T18:41:22.719556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.172298+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:41:26.172298+0000","last_peered":"2024-03-28T18:41:26.172298+0000","last_clean":"2024-03-28T18:41:26.172298+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:41:26.172298+0000","last_undegraded":"2024-03-28T18:41:26.172298+0000","last_fullsized":"2024-03-28T18:41:26.172298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718970+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:41:22.718970+0000","last_peered":"2024-03-28T18:41:22.718970+0000","last_clean":"2024-03-28T18:41:22.718970+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:41:22.718970+0000","last_undegraded":"2024-03-28T18:41:22.718970+0000","last_fullsized":"2024-03-28T18:41:22.718970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.200827+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:41:27.200827+0000","last_peered":"2024-03-28T18:41:27.200827+0000","last_clean":"2024-03-28T18:41:27.200827+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:41:27.200827+0000","last_undegraded":"2024-03-28T18:41:27.200827+0000","last_fullsized":"2024-03-28T18:41:27.200827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719260+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:41:22.719260+0000","last_peered":"2024-03-28T18:41:22.719260+0000","last_clean":"2024-03-28T18:41:22.719260+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:41:22.719260+0000","last_undegraded":"2024-03-28T18:41:22.719260+0000","last_fullsized":"2024-03-28T18:41:22.719260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902746+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:41:22.902746+0000","last_peered":"2024-03-28T18:41:22.902746+0000","last_clean":"2024-03-28T18:41:22.902746+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:41:22.902746+0000","last_undegraded":"2024-03-28T18:41:22.902746+0000","last_fullsized":"2024-03-28T18:41:22.902746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719476+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:41:22.719476+0000","last_peered":"2024-03-28T18:41:22.719476+0000","last_clean":"2024-03-28T18:41:22.719476+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:41:22.719476+0000","last_undegraded":"2024-03-28T18:41:22.719476+0000","last_fullsized":"2024-03-28T18:41:22.719476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719163+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:41:22.719163+0000","last_peered":"2024-03-28T18:41:22.719163+0000","last_clean":"2024-03-28T18:41:22.719163+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:41:22.719163+0000","last_undegraded":"2024-03-28T18:41:22.719163+0000","last_fullsized":"2024-03-28T18:41:22.719163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721223+0000","last_change":"2024-03-28T18:41:21.124291+0000","last_active":"2024-03-28T18:41:22.721223+0000","last_peered":"2024-03-28T18:41:22.721223+0000","last_clean":"2024-03-28T18:41:22.721223+0000","last_became_active":"2024-03-28T18:41:21.118429+0000","last_became_peered":"2024-03-28T18:41:21.118429+0000","last_unstale":"2024-03-28T18:41:22.721223+0000","last_undegraded":"2024-03-28T18:41:22.721223+0000","last_fullsized":"2024-03-28T18:41:22.721223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:40.151407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079161+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:41:22.079161+0000","last_peered":"2024-03-28T18:41:22.079161+0000","last_clean":"2024-03-28T18:41:22.079161+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:41:22.079161+0000","last_undegraded":"2024-03-28T18:41:22.079161+0000","last_fullsized":"2024-03-28T18:41:22.079161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902785+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:41:22.902785+0000","last_peered":"2024-03-28T18:41:22.902785+0000","last_clean":"2024-03-28T18:41:22.902785+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:41:22.902785+0000","last_undegraded":"2024-03-28T18:41:22.902785+0000","last_fullsized":"2024-03-28T18:41:22.902785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719613+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:41:22.719613+0000","last_peered":"2024-03-28T18:41:22.719613+0000","last_clean":"2024-03-28T18:41:22.719613+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:41:22.719613+0000","last_undegraded":"2024-03-28T18:41:22.719613+0000","last_fullsized":"2024-03-28T18:41:22.719613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078002+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:41:22.078002+0000","last_peered":"2024-03-28T18:41:22.078002+0000","last_clean":"2024-03-28T18:41:22.078002+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:41:22.078002+0000","last_undegraded":"2024-03-28T18:41:22.078002+0000","last_fullsized":"2024-03-28T18:41:22.078002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076729+0000","last_change":"2024-03-28T18:41:21.122177+0000","last_active":"2024-03-28T18:41:22.076729+0000","last_peered":"2024-03-28T18:41:22.076729+0000","last_clean":"2024-03-28T18:41:22.076729+0000","last_became_active":"2024-03-28T18:41:21.121680+0000","last_became_peered":"2024-03-28T18:41:21.121680+0000","last_unstale":"2024-03-28T18:41:22.076729+0000","last_undegraded":"2024-03-28T18:41:22.076729+0000","last_fullsized":"2024-03-28T18:41:22.076729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:43.866475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076890+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:41:22.076890+0000","last_peered":"2024-03-28T18:41:22.076890+0000","last_clean":"2024-03-28T18:41:22.076890+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:41:22.076890+0000","last_undegraded":"2024-03-28T18:41:22.076890+0000","last_fullsized":"2024-03-28T18:41:22.076890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434403+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:41:22.434403+0000","last_peered":"2024-03-28T18:41:22.434403+0000","last_clean":"2024-03-28T18:41:22.434403+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:41:22.434403+0000","last_undegraded":"2024-03-28T18:41:22.434403+0000","last_fullsized":"2024-03-28T18:41:22.434403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079214+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:41:22.079214+0000","last_peered":"2024-03-28T18:41:22.079214+0000","last_clean":"2024-03-28T18:41:22.079214+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:41:22.079214+0000","last_undegraded":"2024-03-28T18:41:22.079214+0000","last_fullsized":"2024-03-28T18:41:22.079214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902606+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:41:22.902606+0000","last_peered":"2024-03-28T18:41:22.902606+0000","last_clean":"2024-03-28T18:41:22.902606+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:41:22.902606+0000","last_undegraded":"2024-03-28T18:41:22.902606+0000","last_fullsized":"2024-03-28T18:41:22.902606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14654,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":184,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":68563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162,"ondisk_log_size":162,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":442764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1035645,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":1034404,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":441684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":464219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":443750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":445547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":442119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":451380,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035151,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":39945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":10339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:41:30.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:41:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='client.24694 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: pgmap v316: 97 pgs: 97 active+clean; 590 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s wr, 21 op/s 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='client.24700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [5, 4]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-03-28T18:41:30.798 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1805839470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.799 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:30 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4100932948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:30.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 -- 172.21.15.119:0/1263810800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 msgr2=0x7ff4a410bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 --2- 172.21.15.119:0/1263810800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a410bd00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009ca0 tx=0x7ff49c02c100 comp rx=0 tx=0).stop 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 -- 172.21.15.119:0/1263810800 shutdown_connections 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 --2- 172.21.15.119:0/1263810800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a410bd00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 --2- 172.21.15.119:0/1263810800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 --2- 172.21.15.119:0/1263810800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a41038c0 0x7ff4a4103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.857+0000 7ff4ac600640 1 -- 172.21.15.119:0/1263810800 >> 172.21.15.119:0/1263810800 conn(0x7ff4a40fee50 msgr2=0x7ff4a4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.858+0000 7ff4ac600640 1 -- 172.21.15.119:0/1263810800 shutdown_connections 2024-03-28T18:41:30.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.858+0000 7ff4ac600640 1 -- 172.21.15.119:0/1263810800 wait complete. 2024-03-28T18:41:30.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 Processor -- start 2024-03-28T18:41:30.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 -- start start 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a41a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a410f540 con 0x7ff4a41038c0 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a410f3c0 con 0x7ff4a4104ab0 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.859+0000 7ff4ac600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a410f6c0 con 0x7ff4a4105470 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40956/0 (socket says 172.21.15.119:40956) 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9c00640 1 -- 172.21.15.119:0/2467648100 learned_addr learned my addr 172.21.15.119:0/2467648100 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a41a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9200640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 -- 172.21.15.119:0/2467648100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 msgr2=0x7ff4a4199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 -- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 msgr2=0x7ff4a4199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 -- 172.21.15.119:0/2467648100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a41a16b0 con 0x7ff4a4105470 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9c00640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4a9200640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.860+0000 7ff4aa600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a41a1040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff49c037040 tx=0x7ff49c035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.861+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c039040 con 0x7ff4a4105470 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.861+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff49c044070 con 0x7ff4a4105470 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.861+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c004370 con 0x7ff4a4105470 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.861+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a41a1940 con 0x7ff4a4105470 2024-03-28T18:41:30.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.861+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a41a1db0 con 0x7ff4a4105470 2024-03-28T18:41:30.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.862+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a410ab10 con 0x7ff4a4105470 2024-03-28T18:41:30.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.865+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff49c004510 con 0x7ff4a4105470 2024-03-28T18:41:30.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.866+0000 7ff4a2000640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 0x7ff48c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:30.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.866+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7ff49c0f1070 con 0x7ff4a4105470 2024-03-28T18:41:30.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.866+0000 7ff4a9c00640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 0x7ff48c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:30.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.867+0000 7ff4a9c00640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 0x7ff48c0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4980058c0 tx=0x7ff498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:30.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:30.874+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff49c0be0b0 con 0x7ff4a4105470 2024-03-28T18:41:31.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.071+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4a4061ae0 con 0x7ff48c0a2910 2024-03-28T18:41:31.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.095+0000 7ff4a2000640 1 -- 172.21.15.119:0/2467648100 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258921 (secure 0 0 0) 0x7ff4a4061ae0 con 0x7ff48c0a2910 2024-03-28T18:41:31.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:31.102 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 msgr2=0x7ff48c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 0x7ff48c0a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4980058c0 tx=0x7ff498005f90 comp rx=0 tx=0).stop 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 msgr2=0x7ff4a41a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a41a1040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff49c037040 tx=0x7ff49c035d10 comp rx=0 tx=0).stop 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 shutdown_connections 2024-03-28T18:41:31.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff48c0a2910 0x7ff48c0a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4105470 0x7ff4a41a1040 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff4a4104ab0 0x7ff4a4199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 --2- 172.21.15.119:0/2467648100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a41038c0 0x7ff4a4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 >> 172.21.15.119:0/2467648100 conn(0x7ff4a40fee50 msgr2=0x7ff4a4107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 shutdown_connections 2024-03-28T18:41:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:31.101+0000 7ff4ac600640 1 -- 172.21.15.119:0/2467648100 wait complete. 2024-03-28T18:41:31.121 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-03-28T18:41:30.962033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605112,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":149,"num_read_kb":130,"num_write":381,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":311,"ondisk_log_size":311,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360500,"kb_used_data":11340,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581324,"statfs":{"total":1151940427776,"available":1151571275776,"internally_reserved":0,"allocated":11612160,"data_stored":7130384,"data_compressed":113214,"data_compressed_allocated":143360,"data_compressed_original":1785856,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13076,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":168,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003557"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902971+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:41:22.902971+0000","last_peered":"2024-03-28T18:41:22.902971+0000","last_clean":"2024-03-28T18:41:22.902971+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:41:22.902971+0000","last_undegraded":"2024-03-28T18:41:22.902971+0000","last_fullsized":"2024-03-28T18:41:22.902971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721874+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:41:22.721874+0000","last_peered":"2024-03-28T18:41:22.721874+0000","last_clean":"2024-03-28T18:41:22.721874+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:41:22.721874+0000","last_undegraded":"2024-03-28T18:41:22.721874+0000","last_fullsized":"2024-03-28T18:41:22.721874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721500+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:41:22.721500+0000","last_peered":"2024-03-28T18:41:22.721500+0000","last_clean":"2024-03-28T18:41:22.721500+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:41:22.721500+0000","last_undegraded":"2024-03-28T18:41:22.721500+0000","last_fullsized":"2024-03-28T18:41:22.721500+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214247+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:41:29.214247+0000","last_peered":"2024-03-28T18:41:29.214247+0000","last_clean":"2024-03-28T18:41:29.214247+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:41:29.214247+0000","last_undegraded":"2024-03-28T18:41:29.214247+0000","last_fullsized":"2024-03-28T18:41:29.214247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114546+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:41:22.114546+0000","last_peered":"2024-03-28T18:41:22.114546+0000","last_clean":"2024-03-28T18:41:22.114546+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:41:22.114546+0000","last_undegraded":"2024-03-28T18:41:22.114546+0000","last_fullsized":"2024-03-28T18:41:22.114546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720137+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:41:22.720137+0000","last_peered":"2024-03-28T18:41:22.720137+0000","last_clean":"2024-03-28T18:41:22.720137+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:41:22.720137+0000","last_undegraded":"2024-03-28T18:41:22.720137+0000","last_fullsized":"2024-03-28T18:41:22.720137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718960+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:41:22.718960+0000","last_peered":"2024-03-28T18:41:22.718960+0000","last_clean":"2024-03-28T18:41:22.718960+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:41:22.718960+0000","last_undegraded":"2024-03-28T18:41:22.718960+0000","last_fullsized":"2024-03-28T18:41:22.718960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077097+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:41:22.077097+0000","last_peered":"2024-03-28T18:41:22.077097+0000","last_clean":"2024-03-28T18:41:22.077097+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:41:22.077097+0000","last_undegraded":"2024-03-28T18:41:22.077097+0000","last_fullsized":"2024-03-28T18:41:22.077097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434539+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:41:22.434539+0000","last_peered":"2024-03-28T18:41:22.434539+0000","last_clean":"2024-03-28T18:41:22.434539+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:41:22.434539+0000","last_undegraded":"2024-03-28T18:41:22.434539+0000","last_fullsized":"2024-03-28T18:41:22.434539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720496+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:41:22.720496+0000","last_peered":"2024-03-28T18:41:22.720496+0000","last_clean":"2024-03-28T18:41:22.720496+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:41:22.720496+0000","last_undegraded":"2024-03-28T18:41:22.720496+0000","last_fullsized":"2024-03-28T18:41:22.720496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719448+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:41:22.719448+0000","last_peered":"2024-03-28T18:41:22.719448+0000","last_clean":"2024-03-28T18:41:22.719448+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:41:22.719448+0000","last_undegraded":"2024-03-28T18:41:22.719448+0000","last_fullsized":"2024-03-28T18:41:22.719448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214154+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:41:29.214154+0000","last_peered":"2024-03-28T18:41:29.214154+0000","last_clean":"2024-03-28T18:41:29.214154+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:41:29.214154+0000","last_undegraded":"2024-03-28T18:41:29.214154+0000","last_fullsized":"2024-03-28T18:41:29.214154+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.080006+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:41:22.080006+0000","last_peered":"2024-03-28T18:41:22.080006+0000","last_clean":"2024-03-28T18:41:22.080006+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:41:22.080006+0000","last_undegraded":"2024-03-28T18:41:22.080006+0000","last_fullsized":"2024-03-28T18:41:22.080006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720845+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:41:22.720845+0000","last_peered":"2024-03-28T18:41:22.720845+0000","last_clean":"2024-03-28T18:41:22.720845+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:41:22.720845+0000","last_undegraded":"2024-03-28T18:41:22.720845+0000","last_fullsized":"2024-03-28T18:41:22.720845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720785+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:41:22.720785+0000","last_peered":"2024-03-28T18:41:22.720785+0000","last_clean":"2024-03-28T18:41:22.720785+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:41:22.720785+0000","last_undegraded":"2024-03-28T18:41:22.720785+0000","last_fullsized":"2024-03-28T18:41:22.720785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434173+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:41:22.434173+0000","last_peered":"2024-03-28T18:41:22.434173+0000","last_clean":"2024-03-28T18:41:22.434173+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:41:22.434173+0000","last_undegraded":"2024-03-28T18:41:22.434173+0000","last_fullsized":"2024-03-28T18:41:22.434173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902956+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:41:22.902956+0000","last_peered":"2024-03-28T18:41:22.902956+0000","last_clean":"2024-03-28T18:41:22.902956+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:41:22.902956+0000","last_undegraded":"2024-03-28T18:41:22.902956+0000","last_fullsized":"2024-03-28T18:41:22.902956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T18:41:30.212596+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:41:30.212596+0000","last_peered":"2024-03-28T18:41:30.212596+0000","last_clean":"2024-03-28T18:41:30.212596+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:41:30.212596+0000","last_undegraded":"2024-03-28T18:41:30.212596+0000","last_fullsized":"2024-03-28T18:41:30.212596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719361+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:41:22.719361+0000","last_peered":"2024-03-28T18:41:22.719361+0000","last_clean":"2024-03-28T18:41:22.719361+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:41:22.719361+0000","last_undegraded":"2024-03-28T18:41:22.719361+0000","last_fullsized":"2024-03-28T18:41:22.719361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T18:41:30.212836+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:41:30.212836+0000","last_peered":"2024-03-28T18:41:30.212836+0000","last_clean":"2024-03-28T18:41:30.212836+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:41:30.212836+0000","last_undegraded":"2024-03-28T18:41:30.212836+0000","last_fullsized":"2024-03-28T18:41:30.212836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077994+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:41:22.077994+0000","last_peered":"2024-03-28T18:41:22.077994+0000","last_clean":"2024-03-28T18:41:22.077994+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:41:22.077994+0000","last_undegraded":"2024-03-28T18:41:22.077994+0000","last_fullsized":"2024-03-28T18:41:22.077994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:23.499460+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:41:23.499460+0000","last_peered":"2024-03-28T18:41:23.499460+0000","last_clean":"2024-03-28T18:41:23.499460+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:41:23.499460+0000","last_undegraded":"2024-03-28T18:41:23.499460+0000","last_fullsized":"2024-03-28T18:41:23.499460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721706+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:41:22.721706+0000","last_peered":"2024-03-28T18:41:22.721706+0000","last_clean":"2024-03-28T18:41:22.721706+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:41:22.721706+0000","last_undegraded":"2024-03-28T18:41:22.721706+0000","last_fullsized":"2024-03-28T18:41:22.721706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.167291+0000","last_change":"2024-03-28T18:41:20.067522+0000","last_active":"2024-03-28T18:41:26.167291+0000","last_peered":"2024-03-28T18:41:26.167291+0000","last_clean":"2024-03-28T18:41:26.167291+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:41:26.167291+0000","last_undegraded":"2024-03-28T18:41:26.167291+0000","last_fullsized":"2024-03-28T18:41:26.167291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:16:47.011842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434474+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:41:22.434474+0000","last_peered":"2024-03-28T18:41:22.434474+0000","last_clean":"2024-03-28T18:41:22.434474+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:41:22.434474+0000","last_undegraded":"2024-03-28T18:41:22.434474+0000","last_fullsized":"2024-03-28T18:41:22.434474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114558+0000","last_change":"2024-03-28T18:41:20.069156+0000","last_active":"2024-03-28T18:41:22.114558+0000","last_peered":"2024-03-28T18:41:22.114558+0000","last_clean":"2024-03-28T18:41:22.114558+0000","last_became_active":"2024-03-28T18:41:20.068839+0000","last_became_peered":"2024-03-28T18:41:20.068839+0000","last_unstale":"2024-03-28T18:41:22.114558+0000","last_undegraded":"2024-03-28T18:41:22.114558+0000","last_fullsized":"2024-03-28T18:41:22.114558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:39:55.493517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076763+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:41:22.076763+0000","last_peered":"2024-03-28T18:41:22.076763+0000","last_clean":"2024-03-28T18:41:22.076763+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:41:22.076763+0000","last_undegraded":"2024-03-28T18:41:22.076763+0000","last_fullsized":"2024-03-28T18:41:22.076763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.146054+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:41:24.146054+0000","last_peered":"2024-03-28T18:41:24.146054+0000","last_clean":"2024-03-28T18:41:24.146054+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:41:24.146054+0000","last_undegraded":"2024-03-28T18:41:24.146054+0000","last_fullsized":"2024-03-28T18:41:24.146054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114307+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:41:22.114307+0000","last_peered":"2024-03-28T18:41:22.114307+0000","last_clean":"2024-03-28T18:41:22.114307+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:41:22.114307+0000","last_undegraded":"2024-03-28T18:41:22.114307+0000","last_fullsized":"2024-03-28T18:41:22.114307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.174594+0000","last_change":"2024-03-28T18:41:20.065845+0000","last_active":"2024-03-28T18:41:26.174594+0000","last_peered":"2024-03-28T18:41:26.174594+0000","last_clean":"2024-03-28T18:41:26.174594+0000","last_became_active":"2024-03-28T18:41:20.065495+0000","last_became_peered":"2024-03-28T18:41:20.065495+0000","last_unstale":"2024-03-28T18:41:26.174594+0000","last_undegraded":"2024-03-28T18:41:26.174594+0000","last_fullsized":"2024-03-28T18:41:26.174594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:53:11.078525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076799+0000","last_change":"2024-03-28T18:41:21.115518+0000","last_active":"2024-03-28T18:41:22.076799+0000","last_peered":"2024-03-28T18:41:22.076799+0000","last_clean":"2024-03-28T18:41:22.076799+0000","last_became_active":"2024-03-28T18:41:21.115164+0000","last_became_peered":"2024-03-28T18:41:21.115164+0000","last_unstale":"2024-03-28T18:41:22.076799+0000","last_undegraded":"2024-03-28T18:41:22.076799+0000","last_fullsized":"2024-03-28T18:41:22.076799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:24:30.331524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.161607+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:41:25.161607+0000","last_peered":"2024-03-28T18:41:25.161607+0000","last_clean":"2024-03-28T18:41:25.161607+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:41:25.161607+0000","last_undegraded":"2024-03-28T18:41:25.161607+0000","last_fullsized":"2024-03-28T18:41:25.161607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076978+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:41:22.076978+0000","last_peered":"2024-03-28T18:41:22.076978+0000","last_clean":"2024-03-28T18:41:22.076978+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:41:22.076978+0000","last_undegraded":"2024-03-28T18:41:22.076978+0000","last_fullsized":"2024-03-28T18:41:22.076978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.165830+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:41:26.165830+0000","last_peered":"2024-03-28T18:41:26.165830+0000","last_clean":"2024-03-28T18:41:26.165830+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:41:26.165830+0000","last_undegraded":"2024-03-28T18:41:26.165830+0000","last_fullsized":"2024-03-28T18:41:26.165830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T18:41:30.211838+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:41:30.211838+0000","last_peered":"2024-03-28T18:41:30.211838+0000","last_clean":"2024-03-28T18:41:30.211838+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:41:30.211838+0000","last_undegraded":"2024-03-28T18:41:30.211838+0000","last_fullsized":"2024-03-28T18:41:30.211838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.145174+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:41:24.145174+0000","last_peered":"2024-03-28T18:41:24.145174+0000","last_clean":"2024-03-28T18:41:24.145174+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:41:24.145174+0000","last_undegraded":"2024-03-28T18:41:24.145174+0000","last_fullsized":"2024-03-28T18:41:24.145174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434459+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:41:22.434459+0000","last_peered":"2024-03-28T18:41:22.434459+0000","last_clean":"2024-03-28T18:41:22.434459+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:41:22.434459+0000","last_undegraded":"2024-03-28T18:41:22.434459+0000","last_fullsized":"2024-03-28T18:41:22.434459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.188968+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:41:27.188968+0000","last_peered":"2024-03-28T18:41:27.188968+0000","last_clean":"2024-03-28T18:41:27.188968+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:41:27.188968+0000","last_undegraded":"2024-03-28T18:41:27.188968+0000","last_fullsized":"2024-03-28T18:41:27.188968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721883+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:41:22.721883+0000","last_peered":"2024-03-28T18:41:22.721883+0000","last_clean":"2024-03-28T18:41:22.721883+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:41:22.721883+0000","last_undegraded":"2024-03-28T18:41:22.721883+0000","last_fullsized":"2024-03-28T18:41:22.721883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.146273+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:41:24.146273+0000","last_peered":"2024-03-28T18:41:24.146273+0000","last_clean":"2024-03-28T18:41:24.146273+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:41:24.146273+0000","last_undegraded":"2024-03-28T18:41:24.146273+0000","last_fullsized":"2024-03-28T18:41:24.146273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434633+0000","last_change":"2024-03-28T18:41:21.094009+0000","last_active":"2024-03-28T18:41:22.434633+0000","last_peered":"2024-03-28T18:41:22.434633+0000","last_clean":"2024-03-28T18:41:22.434633+0000","last_became_active":"2024-03-28T18:41:21.093693+0000","last_became_peered":"2024-03-28T18:41:21.093693+0000","last_unstale":"2024-03-28T18:41:22.434633+0000","last_undegraded":"2024-03-28T18:41:22.434633+0000","last_fullsized":"2024-03-28T18:41:22.434633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:53:53.427462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'13","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.188386+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:41:27.188386+0000","last_peered":"2024-03-28T18:41:27.188386+0000","last_clean":"2024-03-28T18:41:27.188386+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:41:27.188386+0000","last_undegraded":"2024-03-28T18:41:27.188386+0000","last_fullsized":"2024-03-28T18:41:27.188386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114560+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:41:22.114560+0000","last_peered":"2024-03-28T18:41:22.114560+0000","last_clean":"2024-03-28T18:41:22.114560+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:41:22.114560+0000","last_undegraded":"2024-03-28T18:41:22.114560+0000","last_fullsized":"2024-03-28T18:41:22.114560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.159524+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:41:25.159524+0000","last_peered":"2024-03-28T18:41:25.159524+0000","last_clean":"2024-03-28T18:41:25.159524+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:41:25.159524+0000","last_undegraded":"2024-03-28T18:41:25.159524+0000","last_fullsized":"2024-03-28T18:41:25.159524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079807+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:41:22.079807+0000","last_peered":"2024-03-28T18:41:22.079807+0000","last_clean":"2024-03-28T18:41:22.079807+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:41:22.079807+0000","last_undegraded":"2024-03-28T18:41:22.079807+0000","last_fullsized":"2024-03-28T18:41:22.079807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.171233+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:41:26.171233+0000","last_peered":"2024-03-28T18:41:26.171233+0000","last_clean":"2024-03-28T18:41:26.171233+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:41:26.171233+0000","last_undegraded":"2024-03-28T18:41:26.171233+0000","last_fullsized":"2024-03-28T18:41:26.171233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719217+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:41:22.719217+0000","last_peered":"2024-03-28T18:41:22.719217+0000","last_clean":"2024-03-28T18:41:22.719217+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:41:22.719217+0000","last_undegraded":"2024-03-28T18:41:22.719217+0000","last_fullsized":"2024-03-28T18:41:22.719217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214507+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:41:29.214507+0000","last_peered":"2024-03-28T18:41:29.214507+0000","last_clean":"2024-03-28T18:41:29.214507+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:41:29.214507+0000","last_undegraded":"2024-03-28T18:41:29.214507+0000","last_fullsized":"2024-03-28T18:41:29.214507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718705+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:41:22.718705+0000","last_peered":"2024-03-28T18:41:22.718705+0000","last_clean":"2024-03-28T18:41:22.718705+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:41:22.718705+0000","last_undegraded":"2024-03-28T18:41:22.718705+0000","last_fullsized":"2024-03-28T18:41:22.718705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.158372+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:41:25.158372+0000","last_peered":"2024-03-28T18:41:25.158372+0000","last_clean":"2024-03-28T18:41:25.158372+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:41:25.158372+0000","last_undegraded":"2024-03-28T18:41:25.158372+0000","last_fullsized":"2024-03-28T18:41:25.158372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076618+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:41:22.076618+0000","last_peered":"2024-03-28T18:41:22.076618+0000","last_clean":"2024-03-28T18:41:22.076618+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:41:22.076618+0000","last_undegraded":"2024-03-28T18:41:22.076618+0000","last_fullsized":"2024-03-28T18:41:22.076618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.180141+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:41:22.180141+0000","last_peered":"2024-03-28T18:41:22.180141+0000","last_clean":"2024-03-28T18:41:22.180141+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:41:22.180141+0000","last_undegraded":"2024-03-28T18:41:22.180141+0000","last_fullsized":"2024-03-28T18:41:22.180141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902783+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:41:22.902783+0000","last_peered":"2024-03-28T18:41:22.902783+0000","last_clean":"2024-03-28T18:41:22.902783+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:41:22.902783+0000","last_undegraded":"2024-03-28T18:41:22.902783+0000","last_fullsized":"2024-03-28T18:41:22.902783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T18:41:30.220354+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:41:30.220354+0000","last_peered":"2024-03-28T18:41:30.220354+0000","last_clean":"2024-03-28T18:41:30.220354+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:41:30.220354+0000","last_undegraded":"2024-03-28T18:41:30.220354+0000","last_fullsized":"2024-03-28T18:41:30.220354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720417+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:41:22.720417+0000","last_peered":"2024-03-28T18:41:22.720417+0000","last_clean":"2024-03-28T18:41:22.720417+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:41:22.720417+0000","last_undegraded":"2024-03-28T18:41:22.720417+0000","last_fullsized":"2024-03-28T18:41:22.720417+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.193538+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:41:27.193538+0000","last_peered":"2024-03-28T18:41:27.193538+0000","last_clean":"2024-03-28T18:41:27.193538+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:41:27.193538+0000","last_undegraded":"2024-03-28T18:41:27.193538+0000","last_fullsized":"2024-03-28T18:41:27.193538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.720296+0000","last_change":"2024-03-28T18:41:21.119477+0000","last_active":"2024-03-28T18:41:22.720296+0000","last_peered":"2024-03-28T18:41:22.720296+0000","last_clean":"2024-03-28T18:41:22.720296+0000","last_became_active":"2024-03-28T18:41:21.119121+0000","last_became_peered":"2024-03-28T18:41:21.119121+0000","last_unstale":"2024-03-28T18:41:22.720296+0000","last_undegraded":"2024-03-28T18:41:22.720296+0000","last_fullsized":"2024-03-28T18:41:22.720296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:14:06.510122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076758+0000","last_change":"2024-03-28T18:41:21.097489+0000","last_active":"2024-03-28T18:41:22.076758+0000","last_peered":"2024-03-28T18:41:22.076758+0000","last_clean":"2024-03-28T18:41:22.076758+0000","last_became_active":"2024-03-28T18:41:21.097079+0000","last_became_peered":"2024-03-28T18:41:21.097079+0000","last_unstale":"2024-03-28T18:41:22.076758+0000","last_undegraded":"2024-03-28T18:41:22.076758+0000","last_fullsized":"2024-03-28T18:41:22.076758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:49.130304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.163837+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:41:25.163837+0000","last_peered":"2024-03-28T18:41:25.163837+0000","last_clean":"2024-03-28T18:41:25.163837+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:41:25.163837+0000","last_undegraded":"2024-03-28T18:41:25.163837+0000","last_fullsized":"2024-03-28T18:41:25.163837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078015+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:41:22.078015+0000","last_peered":"2024-03-28T18:41:22.078015+0000","last_clean":"2024-03-28T18:41:22.078015+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:41:22.078015+0000","last_undegraded":"2024-03-28T18:41:22.078015+0000","last_fullsized":"2024-03-28T18:41:22.078015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.214319+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:41:29.214319+0000","last_peered":"2024-03-28T18:41:29.214319+0000","last_clean":"2024-03-28T18:41:29.214319+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:41:29.214319+0000","last_undegraded":"2024-03-28T18:41:29.214319+0000","last_fullsized":"2024-03-28T18:41:29.214319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718819+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:41:22.718819+0000","last_peered":"2024-03-28T18:41:22.718819+0000","last_clean":"2024-03-28T18:41:22.718819+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:41:22.718819+0000","last_undegraded":"2024-03-28T18:41:22.718819+0000","last_fullsized":"2024-03-28T18:41:22.718819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.077332+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:41:22.077332+0000","last_peered":"2024-03-28T18:41:22.077332+0000","last_clean":"2024-03-28T18:41:22.077332+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:41:22.077332+0000","last_undegraded":"2024-03-28T18:41:22.077332+0000","last_fullsized":"2024-03-28T18:41:22.077332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:25.154441+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:41:25.154441+0000","last_peered":"2024-03-28T18:41:25.154441+0000","last_clean":"2024-03-28T18:41:25.154441+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:41:25.154441+0000","last_undegraded":"2024-03-28T18:41:25.154441+0000","last_fullsized":"2024-03-28T18:41:25.154441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.169515+0000","last_change":"2024-03-28T18:41:20.079909+0000","last_active":"2024-03-28T18:41:26.169515+0000","last_peered":"2024-03-28T18:41:26.169515+0000","last_clean":"2024-03-28T18:41:26.169515+0000","last_became_active":"2024-03-28T18:41:20.079658+0000","last_became_peered":"2024-03-28T18:41:20.079658+0000","last_unstale":"2024-03-28T18:41:26.169515+0000","last_undegraded":"2024-03-28T18:41:26.169515+0000","last_fullsized":"2024-03-28T18:41:26.169515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:14.211815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079821+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:41:22.079821+0000","last_peered":"2024-03-28T18:41:22.079821+0000","last_clean":"2024-03-28T18:41:22.079821+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:41:22.079821+0000","last_undegraded":"2024-03-28T18:41:22.079821+0000","last_fullsized":"2024-03-28T18:41:22.079821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-28T18:41:30.212772+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:41:30.212772+0000","last_peered":"2024-03-28T18:41:30.212772+0000","last_clean":"2024-03-28T18:41:30.212772+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:41:30.212772+0000","last_undegraded":"2024-03-28T18:41:30.212772+0000","last_fullsized":"2024-03-28T18:41:30.212772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078255+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:41:22.078255+0000","last_peered":"2024-03-28T18:41:22.078255+0000","last_clean":"2024-03-28T18:41:22.078255+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:41:22.078255+0000","last_undegraded":"2024-03-28T18:41:22.078255+0000","last_fullsized":"2024-03-28T18:41:22.078255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-28T18:41:29.213454+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:41:29.213454+0000","last_peered":"2024-03-28T18:41:29.213454+0000","last_clean":"2024-03-28T18:41:29.213454+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:41:29.213454+0000","last_undegraded":"2024-03-28T18:41:29.213454+0000","last_fullsized":"2024-03-28T18:41:29.213454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721926+0000","last_change":"2024-03-28T18:41:21.125927+0000","last_active":"2024-03-28T18:41:22.721926+0000","last_peered":"2024-03-28T18:41:22.721926+0000","last_clean":"2024-03-28T18:41:22.721926+0000","last_became_active":"2024-03-28T18:41:21.125516+0000","last_became_peered":"2024-03-28T18:41:21.125516+0000","last_unstale":"2024-03-28T18:41:22.721926+0000","last_undegraded":"2024-03-28T18:41:22.721926+0000","last_fullsized":"2024-03-28T18:41:22.721926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:20.224459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.145579+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:41:24.145579+0000","last_peered":"2024-03-28T18:41:24.145579+0000","last_clean":"2024-03-28T18:41:24.145579+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:41:24.145579+0000","last_undegraded":"2024-03-28T18:41:24.145579+0000","last_fullsized":"2024-03-28T18:41:24.145579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.114360+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:41:22.114360+0000","last_peered":"2024-03-28T18:41:22.114360+0000","last_clean":"2024-03-28T18:41:22.114360+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:41:22.114360+0000","last_undegraded":"2024-03-28T18:41:22.114360+0000","last_fullsized":"2024-03-28T18:41:22.114360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.199496+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:41:27.199496+0000","last_peered":"2024-03-28T18:41:27.199496+0000","last_clean":"2024-03-28T18:41:27.199496+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:41:27.199496+0000","last_undegraded":"2024-03-28T18:41:27.199496+0000","last_fullsized":"2024-03-28T18:41:27.199496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902566+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:41:22.902566+0000","last_peered":"2024-03-28T18:41:22.902566+0000","last_clean":"2024-03-28T18:41:22.902566+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:41:22.902566+0000","last_undegraded":"2024-03-28T18:41:22.902566+0000","last_fullsized":"2024-03-28T18:41:22.902566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.140419+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:41:26.140419+0000","last_peered":"2024-03-28T18:41:26.140419+0000","last_clean":"2024-03-28T18:41:26.140419+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:41:26.140419+0000","last_undegraded":"2024-03-28T18:41:26.140419+0000","last_fullsized":"2024-03-28T18:41:26.140419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078223+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:41:22.078223+0000","last_peered":"2024-03-28T18:41:22.078223+0000","last_clean":"2024-03-28T18:41:22.078223+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:41:22.078223+0000","last_undegraded":"2024-03-28T18:41:22.078223+0000","last_fullsized":"2024-03-28T18:41:22.078223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:24.143807+0000","last_change":"2024-03-28T18:41:20.067104+0000","last_active":"2024-03-28T18:41:24.143807+0000","last_peered":"2024-03-28T18:41:24.143807+0000","last_clean":"2024-03-28T18:41:24.143807+0000","last_became_active":"2024-03-28T18:41:20.066686+0000","last_became_peered":"2024-03-28T18:41:20.066686+0000","last_unstale":"2024-03-28T18:41:24.143807+0000","last_undegraded":"2024-03-28T18:41:24.143807+0000","last_fullsized":"2024-03-28T18:41:24.143807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:20:33.049170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719391+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:41:22.719391+0000","last_peered":"2024-03-28T18:41:22.719391+0000","last_clean":"2024-03-28T18:41:22.719391+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:41:22.719391+0000","last_undegraded":"2024-03-28T18:41:22.719391+0000","last_fullsized":"2024-03-28T18:41:22.719391+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.168589+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:41:26.168589+0000","last_peered":"2024-03-28T18:41:26.168589+0000","last_clean":"2024-03-28T18:41:26.168589+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:41:26.168589+0000","last_undegraded":"2024-03-28T18:41:26.168589+0000","last_fullsized":"2024-03-28T18:41:26.168589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719556+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:41:22.719556+0000","last_peered":"2024-03-28T18:41:22.719556+0000","last_clean":"2024-03-28T18:41:22.719556+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:41:22.719556+0000","last_undegraded":"2024-03-28T18:41:22.719556+0000","last_fullsized":"2024-03-28T18:41:22.719556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:26.172298+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:41:26.172298+0000","last_peered":"2024-03-28T18:41:26.172298+0000","last_clean":"2024-03-28T18:41:26.172298+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:41:26.172298+0000","last_undegraded":"2024-03-28T18:41:26.172298+0000","last_fullsized":"2024-03-28T18:41:26.172298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.718970+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:41:22.718970+0000","last_peered":"2024-03-28T18:41:22.718970+0000","last_clean":"2024-03-28T18:41:22.718970+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:41:22.718970+0000","last_undegraded":"2024-03-28T18:41:22.718970+0000","last_fullsized":"2024-03-28T18:41:22.718970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:27.200827+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:41:27.200827+0000","last_peered":"2024-03-28T18:41:27.200827+0000","last_clean":"2024-03-28T18:41:27.200827+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:41:27.200827+0000","last_undegraded":"2024-03-28T18:41:27.200827+0000","last_fullsized":"2024-03-28T18:41:27.200827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719260+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:41:22.719260+0000","last_peered":"2024-03-28T18:41:22.719260+0000","last_clean":"2024-03-28T18:41:22.719260+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:41:22.719260+0000","last_undegraded":"2024-03-28T18:41:22.719260+0000","last_fullsized":"2024-03-28T18:41:22.719260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902746+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:41:22.902746+0000","last_peered":"2024-03-28T18:41:22.902746+0000","last_clean":"2024-03-28T18:41:22.902746+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:41:22.902746+0000","last_undegraded":"2024-03-28T18:41:22.902746+0000","last_fullsized":"2024-03-28T18:41:22.902746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719476+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:41:22.719476+0000","last_peered":"2024-03-28T18:41:22.719476+0000","last_clean":"2024-03-28T18:41:22.719476+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:41:22.719476+0000","last_undegraded":"2024-03-28T18:41:22.719476+0000","last_fullsized":"2024-03-28T18:41:22.719476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719163+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:41:22.719163+0000","last_peered":"2024-03-28T18:41:22.719163+0000","last_clean":"2024-03-28T18:41:22.719163+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:41:22.719163+0000","last_undegraded":"2024-03-28T18:41:22.719163+0000","last_fullsized":"2024-03-28T18:41:22.719163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.721223+0000","last_change":"2024-03-28T18:41:21.124291+0000","last_active":"2024-03-28T18:41:22.721223+0000","last_peered":"2024-03-28T18:41:22.721223+0000","last_clean":"2024-03-28T18:41:22.721223+0000","last_became_active":"2024-03-28T18:41:21.118429+0000","last_became_peered":"2024-03-28T18:41:21.118429+0000","last_unstale":"2024-03-28T18:41:22.721223+0000","last_undegraded":"2024-03-28T18:41:22.721223+0000","last_fullsized":"2024-03-28T18:41:22.721223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:40.151407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079161+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:41:22.079161+0000","last_peered":"2024-03-28T18:41:22.079161+0000","last_clean":"2024-03-28T18:41:22.079161+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:41:22.079161+0000","last_undegraded":"2024-03-28T18:41:22.079161+0000","last_fullsized":"2024-03-28T18:41:22.079161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902785+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:41:22.902785+0000","last_peered":"2024-03-28T18:41:22.902785+0000","last_clean":"2024-03-28T18:41:22.902785+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:41:22.902785+0000","last_undegraded":"2024-03-28T18:41:22.902785+0000","last_fullsized":"2024-03-28T18:41:22.902785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.719613+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:41:22.719613+0000","last_peered":"2024-03-28T18:41:22.719613+0000","last_clean":"2024-03-28T18:41:22.719613+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:41:22.719613+0000","last_undegraded":"2024-03-28T18:41:22.719613+0000","last_fullsized":"2024-03-28T18:41:22.719613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.078002+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:41:22.078002+0000","last_peered":"2024-03-28T18:41:22.078002+0000","last_clean":"2024-03-28T18:41:22.078002+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:41:22.078002+0000","last_undegraded":"2024-03-28T18:41:22.078002+0000","last_fullsized":"2024-03-28T18:41:22.078002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076729+0000","last_change":"2024-03-28T18:41:21.122177+0000","last_active":"2024-03-28T18:41:22.076729+0000","last_peered":"2024-03-28T18:41:22.076729+0000","last_clean":"2024-03-28T18:41:22.076729+0000","last_became_active":"2024-03-28T18:41:21.121680+0000","last_became_peered":"2024-03-28T18:41:21.121680+0000","last_unstale":"2024-03-28T18:41:22.076729+0000","last_undegraded":"2024-03-28T18:41:22.076729+0000","last_fullsized":"2024-03-28T18:41:22.076729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:43.866475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.076890+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:41:22.076890+0000","last_peered":"2024-03-28T18:41:22.076890+0000","last_clean":"2024-03-28T18:41:22.076890+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:41:22.076890+0000","last_undegraded":"2024-03-28T18:41:22.076890+0000","last_fullsized":"2024-03-28T18:41:22.076890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.434403+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:41:22.434403+0000","last_peered":"2024-03-28T18:41:22.434403+0000","last_clean":"2024-03-28T18:41:22.434403+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:41:22.434403+0000","last_undegraded":"2024-03-28T18:41:22.434403+0000","last_fullsized":"2024-03-28T18:41:22.434403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.079214+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:41:22.079214+0000","last_peered":"2024-03-28T18:41:22.079214+0000","last_clean":"2024-03-28T18:41:22.079214+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:41:22.079214+0000","last_undegraded":"2024-03-28T18:41:22.079214+0000","last_fullsized":"2024-03-28T18:41:22.079214+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-28T18:41:22.902606+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:41:22.902606+0000","last_peered":"2024-03-28T18:41:22.902606+0000","last_clean":"2024-03-28T18:41:22.902606+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:41:22.902606+0000","last_undegraded":"2024-03-28T18:41:22.902606+0000","last_fullsized":"2024-03-28T18:41:22.902606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14744,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":13,"num_read_kb":11,"num_write":200,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":68867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174,"ondisk_log_size":174,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":442764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1035645,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776154,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":451411,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":1034404,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":441684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":464219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":443750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":445547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":442119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":451380,"data_compressed":2889,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":1035151,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":39945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":10339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:41:31.127 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T18:41:31.127 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T18:41:31.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T18:41:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:31 smithi119 ceph-mon[31813]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:41:31.616 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:31.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:31 smithi196 ceph-mon[36300]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:41:31.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:31 smithi131 ceph-mon[36065]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T18:41:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:32 smithi119 ceph-mon[31813]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:32 smithi119 ceph-mon[31813]: pgmap v319: 97 pgs: 97 active+clean; 591 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-03-28T18:41:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:32 smithi119 ceph-mon[31813]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:32 smithi196 ceph-mon[36300]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:32 smithi196 ceph-mon[36300]: pgmap v319: 97 pgs: 97 active+clean; 591 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-03-28T18:41:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:32 smithi196 ceph-mon[36300]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:32 smithi131 ceph-mon[36065]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:32 smithi131 ceph-mon[36065]: pgmap v319: 97 pgs: 97 active+clean; 591 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.7 KiB/s wr, 22 op/s 2024-03-28T18:41:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:32 smithi131 ceph-mon[36065]: from='client.24724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:41:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.051+0000 7f65b7200700 1 -- 172.21.15.119:0/954302232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0105610 msgr2=0x7f65b0105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.051+0000 7f65b7200700 1 --2- 172.21.15.119:0/954302232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0105610 0x7f65b0105a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f659c009990 tx=0x7f659c009ca0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 -- 172.21.15.119:0/954302232 shutdown_connections 2024-03-28T18:41:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 --2- 172.21.15.119:0/954302232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 0x7f65b010c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 --2- 172.21.15.119:0/954302232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0104410 0x7f65b0104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 --2- 172.21.15.119:0/954302232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0105610 0x7f65b0105a70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 -- 172.21.15.119:0/954302232 >> 172.21.15.119:0/954302232 conn(0x7f65b00ff9b0 msgr2=0x7f65b0101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 -- 172.21.15.119:0/954302232 shutdown_connections 2024-03-28T18:41:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.052+0000 7f65b7200700 1 -- 172.21.15.119:0/954302232 wait complete. 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 Processor -- start 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 -- start start 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 0x7f65b019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 0x7f65b01a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b0110030 con 0x7f65b0105fb0 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b010feb0 con 0x7f65b0105610 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.053+0000 7f65b7200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01101b0 con 0x7f65b0104410 2024-03-28T18:41:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65afe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 0x7f65b019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46322/0 (socket says 172.21.15.119:46322) 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65afe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 0x7f65b019a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39880/0 (socket says 172.21.15.119:39880) 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 -- 172.21.15.119:0/1939875330 learned_addr learned my addr 172.21.15.119:0/1939875330 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65b5200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 0x7f65b01a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 -- 172.21.15.119:0/1939875330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 msgr2=0x7f65b019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 0x7f65b019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 -- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 msgr2=0x7f65b01a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 0x7f65b01a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 -- 172.21.15.119:0/1939875330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b01a2250 con 0x7f65b0105610 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.054+0000 7f65af400700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f659c005910 tx=0x7f659c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.055+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c026070 con 0x7f65b0105610 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.055+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f659c00fde0 con 0x7f65b0105610 2024-03-28T18:41:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.055+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b01a24e0 con 0x7f65b0105610 2024-03-28T18:41:34.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.055+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c017650 con 0x7f65b0105610 2024-03-28T18:41:34.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.055+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b01a2990 con 0x7f65b0105610 2024-03-28T18:41:34.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.056+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b00506a0 con 0x7f65b0105610 2024-03-28T18:41:34.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.058+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f659c029070 con 0x7f65b0105610 2024-03-28T18:41:34.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.059+0000 7f65acc00700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 0x7f65940a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:34.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.059+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f659c0cfb20 con 0x7f65b0105610 2024-03-28T18:41:34.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.059+0000 7f65afe00700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 0x7f65940a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:34.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.060+0000 7f65afe00700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 0x7f65940a4f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f65b0105470 tx=0x7f65a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:34.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.068+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f659c05d740 con 0x7f65b0105610 2024-03-28T18:41:34.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.379+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f65b002da80 con 0x7f65b0105610 2024-03-28T18:41:34.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.387+0000 7f65acc00700 1 -- 172.21.15.119:0/1939875330 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f659c013070 con 0x7f65b0105610 2024-03-28T18:41:34.387 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-03-28T18:41:34.387 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDeuQVmXVHGFhAA8/FmyN2b6F10fa/+OKJNiw== 2024-03-28T18:41:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 msgr2=0x7f65940a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 0x7f65940a4f10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f65b0105470 tx=0x7f65a0009260 comp rx=0 tx=0).stop 2024-03-28T18:41:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 msgr2=0x7f65b019ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f659c005910 tx=0x7f659c00f690 comp rx=0 tx=0).stop 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 shutdown_connections 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f65940a2a50 0x7f65940a4f10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65b0105fb0 0x7f65b01a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65b0104410 0x7f65b019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 --2- 172.21.15.119:0/1939875330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65b0105610 0x7f65b019ab30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.390+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 >> 172.21.15.119:0/1939875330 conn(0x7f65b00ff9b0 msgr2=0x7f65b0108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.391+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 shutdown_connections 2024-03-28T18:41:34.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:34.391+0000 7f65b7200700 1 -- 172.21.15.119:0/1939875330 wait complete. 2024-03-28T18:41:34.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:34 smithi119 ceph-mon[31813]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 31 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:34 smithi196 ceph-mon[36300]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 31 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:34 smithi131 ceph-mon[36065]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 31 op/s; 72 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:35.293 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:41:35.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T18:41:35.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T18:41:35.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:35 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1939875330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:35 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:35 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:41:35.444 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T18:41:35.444 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T18:41:35.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T18:41:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:35 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1939875330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:35 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:35 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:41:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:35 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1939875330' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:35 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T18:41:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:35 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T18:41:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:36.401 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:36 smithi119 ceph-mon[31813]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 29 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:36 smithi196 ceph-mon[36300]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 29 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:36 smithi131 ceph-mon[36065]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 29 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:38.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 -- 172.21.15.119:0/675062063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102df0 msgr2=0x7f6d90106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:38.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 --2- 172.21.15.119:0/675062063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102df0 0x7f6d90106e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009990 tx=0x7f6d78009ca0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 -- 172.21.15.119:0/675062063 shutdown_connections 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 --2- 172.21.15.119:0/675062063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90107510 0x7f6d9010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 --2- 172.21.15.119:0/675062063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90102440 0x7f6d90102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 --2- 172.21.15.119:0/675062063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102df0 0x7f6d90106e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 -- 172.21.15.119:0/675062063 >> 172.21.15.119:0/675062063 conn(0x7f6d900792f0 msgr2=0x7f6d90079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 -- 172.21.15.119:0/675062063 shutdown_connections 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.311+0000 7f6d94c00700 1 -- 172.21.15.119:0/675062063 wait complete. 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.312+0000 7f6d94c00700 1 Processor -- start 2024-03-28T18:41:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.312+0000 7f6d94c00700 1 -- start start 2024-03-28T18:41:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 0x7f6d901a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 0x7f6d901a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90107510 0x7f6d9019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90112520 con 0x7f6d90107510 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901123a0 con 0x7f6d90102440 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d94c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d901126a0 con 0x7f6d90102df0 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d8de00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 0x7f6d901a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d87e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 0x7f6d901a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d8de00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 0x7f6d901a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46340/0 (socket says 172.21.15.119:46340) 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d8de00700 1 -- 172.21.15.119:0/1269945814 learned_addr learned my addr 172.21.15.119:0/1269945814 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.313+0000 7f6d8e800700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90107510 0x7f6d9019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 -- 172.21.15.119:0/1269945814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 msgr2=0x7f6d901a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 0x7f6d901a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 -- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90107510 msgr2=0x7f6d9019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:38.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90107510 0x7f6d9019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 -- 172.21.15.119:0/1269945814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9019b1c0 con 0x7f6d90102df0 2024-03-28T18:41:38.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d87e00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 0x7f6d901a0d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6d780098e0 tx=0x7f6d7800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78027070 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d7800fd10 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9019b450 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780173f0 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.314+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d901a7720 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.315+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d90103910 con 0x7f6d90102df0 2024-03-28T18:41:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.317+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6d7802a070 con 0x7f6d90102df0 2024-03-28T18:41:38.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.318+0000 7f6d86a00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 0x7f6d740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:38.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.318+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f6d780cfa90 con 0x7f6d90102df0 2024-03-28T18:41:38.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.318+0000 7f6d8de00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 0x7f6d740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:38 smithi119 ceph-mon[31813]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:38.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.319+0000 7f6d8de00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 0x7f6d740a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6d90108900 tx=0x7f6d80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.322+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d7805e740 con 0x7f6d90102df0 2024-03-28T18:41:38.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.600+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6d9019bf40 con 0x7f6d90102df0 2024-03-28T18:41:38.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.606+0000 7f6d86a00700 1 -- 172.21.15.119:0/1269945814 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262665 (secure 0 0 0) 0x7f6d78013070 con 0x7f6d90102df0 2024-03-28T18:41:38.612 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:38.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.613+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 msgr2=0x7f6d740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:38.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.613+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 0x7f6d740a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6d90108900 tx=0x7f6d80009260 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.613+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 msgr2=0x7f6d901a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.613+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 0x7f6d901a0d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6d780098e0 tx=0x7f6d7800b840 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 shutdown_connections 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6d740a2920 0x7f6d740a4de0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d90107510 0x7f6d9019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6d90102440 0x7f6d901a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 --2- 172.21.15.119:0/1269945814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d90102df0 0x7f6d901a0d90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 >> 172.21.15.119:0/1269945814 conn(0x7f6d900792f0 msgr2=0x7f6d9010d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 shutdown_connections 2024-03-28T18:41:38.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:38.614+0000 7f6d94c00700 1 -- 172.21.15.119:0/1269945814 wait complete. 2024-03-28T18:41:38.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:38 smithi196 ceph-mon[36300]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:38 smithi131 ceph-mon[36065]: pgmap v322: 97 pgs: 97 active+clean; 630 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s; 86 B/s, 1 keys/s, 2 objects/s recovering 2024-03-28T18:41:39.399 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2493078203},{"type":"v1","addr":"172.21.15.119:6801","nonce":2493078203}]},"active_addr":"172.21.15.119:6801/2493078203","active_change":"2024-03-28T18:33:28.825337+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.119:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":1763050303}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":748679560}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":393583864}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2790763787}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":4162827}]}]} 2024-03-28T18:41:39.404 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T18:41:39.404 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T18:41:39.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:41:39.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:39 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1269945814' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:41:39.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:39 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1269945814' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:41:39.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:39 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1269945814' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T18:41:39.900 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:40.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:40 smithi119 ceph-mon[31813]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 19 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:40 smithi196 ceph-mon[36300]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 19 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:40 smithi131 ceph-mon[36065]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 19 op/s; 71 B/s, 1 keys/s, 1 objects/s recovering 2024-03-28T18:41:41.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:41 smithi119 ceph-mon[31813]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 16 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:41.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:41 smithi196 ceph-mon[36300]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 16 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:41 smithi131 ceph-mon[36065]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 16 op/s; 64 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 -- 172.21.15.119:0/1807124262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f681021e0 msgr2=0x7f1f68102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 --2- 172.21.15.119:0/1807124262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f681021e0 0x7f1f68102660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1f580098f0 tx=0x7f1f58009c00 comp rx=0 tx=0).stop 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 -- 172.21.15.119:0/1807124262 shutdown_connections 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 --2- 172.21.15.119:0/1807124262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68102ba0 0x7f1f6810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 --2- 172.21.15.119:0/1807124262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 0x7f1f681085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 --2- 172.21.15.119:0/1807124262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f681021e0 0x7f1f68102660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 -- 172.21.15.119:0/1807124262 >> 172.21.15.119:0/1807124262 conn(0x7f1f680792c0 msgr2=0x7f1f680796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.135+0000 7f1f70800700 1 -- 172.21.15.119:0/1807124262 shutdown_connections 2024-03-28T18:41:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.136+0000 7f1f70800700 1 -- 172.21.15.119:0/1807124262 wait complete. 2024-03-28T18:41:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.136+0000 7f1f70800700 1 Processor -- start 2024-03-28T18:41:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 -- start start 2024-03-28T18:41:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 0x7f1f68074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 0x7f1f68074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f681124b0 con 0x7f1f681021e0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f68112330 con 0x7f1f681081c0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f70800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f68112630 con 0x7f1f68102ba0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6d400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 0x7f1f68074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6e800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 0x7f1f68074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6d400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45216/0 (socket says 172.21.15.119:45216) 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 0x7f1f68074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60544/0 (socket says 172.21.15.119:60544) 2024-03-28T18:41:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.137+0000 7f1f6d400700 1 -- 172.21.15.119:0/2721943623 learned_addr learned my addr 172.21.15.119:0/2721943623 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:42.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 -- 172.21.15.119:0/2721943623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 msgr2=0x7f1f68074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:42.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 0x7f1f68074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 -- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 msgr2=0x7f1f68074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:42.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 0x7f1f68074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 -- 172.21.15.119:0/2721943623 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f681a6e60 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f6d400700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009fd0 tx=0x7f1f580094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58025070 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f5800ba40 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f681a70f0 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58004b40 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.138+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f681a7540 con 0x7f1f68102ba0 2024-03-28T18:41:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.140+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f68068e50 con 0x7f1f68102ba0 2024-03-28T18:41:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.142+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1f58028070 con 0x7f1f68102ba0 2024-03-28T18:41:42.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.143+0000 7f1f62a00700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 0x7f1f500a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.143+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f1f580cf380 con 0x7f1f68102ba0 2024-03-28T18:41:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.143+0000 7f1f6de00700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 0x7f1f500a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.144+0000 7f1f6de00700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 0x7f1f500a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005fd0 tx=0x7f1f5c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:42.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.147+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1f5805c740 con 0x7f1f68102ba0 2024-03-28T18:41:42.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.370+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f6802dc90 con 0x7f1f68102ba0 2024-03-28T18:41:42.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.372+0000 7f1f62a00700 1 -- 172.21.15.119:0/2721943623 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22776 (secure 0 0 0) 0x7f1f58013030 con 0x7f1f68102ba0 2024-03-28T18:41:42.372 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:42.373 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":88,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:41:30.207288+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:42.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 msgr2=0x7f1f500a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:42.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 0x7f1f500a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005fd0 tx=0x7f1f5c009310 comp rx=0 tx=0).stop 2024-03-28T18:41:42.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 msgr2=0x7f1f680749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:42.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009fd0 tx=0x7f1f580094b0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 shutdown_connections 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1f500a2920 0x7f1f500a4de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f681081c0 0x7f1f68074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f681021e0 0x7f1f68074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 --2- 172.21.15.119:0/2721943623 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f68102ba0 0x7f1f680749d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.375+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 >> 172.21.15.119:0/2721943623 conn(0x7f1f680792c0 msgr2=0x7f1f68100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.376+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 shutdown_connections 2024-03-28T18:41:42.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:42.376+0000 7f1f70800700 1 -- 172.21.15.119:0/2721943623 wait complete. 2024-03-28T18:41:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:42 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2721943623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:42 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2721943623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:42.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:42 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2721943623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:43.311 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T18:41:43.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T18:41:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:43 smithi119 ceph-mon[31813]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.8 KiB/s wr, 16 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:43.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:43 smithi196 ceph-mon[36300]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.8 KiB/s wr, 16 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:43.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:43 smithi131 ceph-mon[36065]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.8 KiB/s wr, 16 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-28T18:41:43.771 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:45.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 -- 172.21.15.119:0/1317331610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce4109160 msgr2=0x7f7ce4115670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:45.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 --2- 172.21.15.119:0/1317331610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce4109160 0x7f7ce4115670 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00b0a0 tx=0x7f7cdc00b3b0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 -- 172.21.15.119:0/1317331610 shutdown_connections 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 --2- 172.21.15.119:0/1317331610 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce4109160 0x7f7ce4115670 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 --2- 172.21.15.119:0/1317331610 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce410e7a0 0x7f7ce410eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 --2- 172.21.15.119:0/1317331610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41087a0 0x7f7ce4108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 -- 172.21.15.119:0/1317331610 >> 172.21.15.119:0/1317331610 conn(0x7f7ce4075960 msgr2=0x7f7ce4077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 -- 172.21.15.119:0/1317331610 shutdown_connections 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.954+0000 7f7ceb200700 1 -- 172.21.15.119:0/1317331610 wait complete. 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.955+0000 7f7ceb200700 1 Processor -- start 2024-03-28T18:41:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.955+0000 7f7ceb200700 1 -- start start 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce4109160 0x7f7ce419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce410e7a0 0x7f7ce41a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4118b80 con 0x7f7ce4109160 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4118a00 con 0x7f7ce41087a0 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ceb200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4118d00 con 0x7f7ce410e7a0 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ce3e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ce3e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33828/0 (socket says 172.21.15.119:33828) 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ce3e00700 1 -- 172.21.15.119:0/215910616 learned_addr learned my addr 172.21.15.119:0/215910616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ce3400700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce4109160 0x7f7ce419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.956+0000 7f7ce9200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce410e7a0 0x7f7ce41a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 -- 172.21.15.119:0/215910616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce410e7a0 msgr2=0x7f7ce41a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce410e7a0 0x7f7ce41a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 -- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce4109160 msgr2=0x7f7ce419f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce4109160 0x7f7ce419f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 -- 172.21.15.119:0/215910616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce41a3ee0 con 0x7f7ce41087a0 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce3e00700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400e9a0 tx=0x7f7cd400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400ca10 con 0x7f7ce41087a0 2024-03-28T18:41:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd400cbb0 con 0x7f7ce41087a0 2024-03-28T18:41:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4018680 con 0x7f7ce41087a0 2024-03-28T18:41:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.957+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce41a41d0 con 0x7f7ce41087a0 2024-03-28T18:41:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.958+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41aba50 con 0x7f7ce41087a0 2024-03-28T18:41:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.959+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce4068e50 con 0x7f7ce41087a0 2024-03-28T18:41:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.960+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7cd4010430 con 0x7f7ce41087a0 2024-03-28T18:41:45.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.962+0000 7f7ce0c00700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 0x7f7cc80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:45.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.962+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f7cd40cedb0 con 0x7f7ce41087a0 2024-03-28T18:41:45.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.962+0000 7f7ce3400700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 0x7f7cc80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:45.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.963+0000 7f7ce3400700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 0x7f7cc80a4e60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7ce410dd60 tx=0x7f7cd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:45.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:45.966+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7cd40d3070 con 0x7f7ce41087a0 2024-03-28T18:41:46.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.187+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ce402db50 con 0x7f7ce41087a0 2024-03-28T18:41:46.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.189+0000 7f7ce0c00700 1 -- 172.21.15.119:0/215910616 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22776 (secure 0 0 0) 0x7f7cd409bc80 con 0x7f7ce41087a0 2024-03-28T18:41:46.190 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:41:46.190 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":88,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:41:30.207288+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.192+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 msgr2=0x7f7cc80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.192+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 0x7f7cc80a4e60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7ce410dd60 tx=0x7f7cd0009260 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.192+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 msgr2=0x7f7ce419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.192+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400e9a0 tx=0x7f7cd400ecb0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 shutdown_connections 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f7cc80a29a0 0x7f7cc80a4e60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ce410e7a0 0x7f7ce41a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7ce41087a0 0x7f7ce419eed0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 --2- 172.21.15.119:0/215910616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce4109160 0x7f7ce419f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 >> 172.21.15.119:0/215910616 conn(0x7f7ce4075960 msgr2=0x7f7ce4075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 shutdown_connections 2024-03-28T18:41:46.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:46.193+0000 7f7ceb200700 1 -- 172.21.15.119:0/215910616 wait complete. 2024-03-28T18:41:46.209 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:46 smithi119 ceph-mon[31813]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T18:41:46.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:46 smithi196 ceph-mon[36300]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T18:41:46.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:46 smithi131 ceph-mon[36065]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T18:41:47.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T18:41:47.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T18:41:47.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T18:41:47.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T18:41:47.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T18:41:47.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T18:41:47.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T18:41:47.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T18:41:47.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T18:41:47.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T18:41:47.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T18:41:47.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T18:41:47.215 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:47 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/215910616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:47.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:47 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/215910616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:47.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:47 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/215910616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:41:48.230 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.231 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.231 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.245 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.245 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.252 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.257 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.258 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.267 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.268 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.268 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.273 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:41:48.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:48 smithi119 ceph-mon[31813]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:41:48.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:48 smithi196 ceph-mon[36300]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:41:48.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:48 smithi131 ceph-mon[36065]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T18:41:50.312 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:50 smithi196 ceph-mon[36300]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-28T18:41:50.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:50 smithi119 ceph-mon[31813]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-28T18:41:50.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:50 smithi131 ceph-mon[36065]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-28T18:41:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:52 smithi119 ceph-mon[31813]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-03-28T18:41:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:52 smithi196 ceph-mon[36300]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-03-28T18:41:52.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:52 smithi131 ceph-mon[36065]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 9 op/s 2024-03-28T18:41:54.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:54 smithi119 ceph-mon[31813]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-28T18:41:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:54 smithi196 ceph-mon[36300]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-28T18:41:54.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:54 smithi131 ceph-mon[36065]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-28T18:41:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:56 smithi119 ceph-mon[31813]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T18:41:56.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:56 smithi196 ceph-mon[36300]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T18:41:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:56 smithi131 ceph-mon[36065]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T18:41:57.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 -- 172.21.15.119:0/1438212725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 msgr2=0x7fd2cc06a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 --2- 172.21.15.119:0/1438212725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 0x7fd2cc06a2a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40098f0 tx=0x7fd2b4009c00 comp rx=0 tx=0).stop 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 -- 172.21.15.119:0/1438212725 shutdown_connections 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 --2- 172.21.15.119:0/1438212725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 --2- 172.21.15.119:0/1438212725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc1081e0 0x7fd2cc1085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 --2- 172.21.15.119:0/1438212725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 0x7fd2cc06a2a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 -- 172.21.15.119:0/1438212725 >> 172.21.15.119:0/1438212725 conn(0x7fd2cc077fc0 msgr2=0x7fd2cc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 -- 172.21.15.119:0/1438212725 shutdown_connections 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.411+0000 7fd2d1600700 1 -- 172.21.15.119:0/1438212725 wait complete. 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 Processor -- start 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 -- start start 2024-03-28T18:41:57.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 0x7fd2cc1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc1081e0 0x7fd2cc19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc112490 con 0x7fd2cc1081e0 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc112310 con 0x7fd2cc069e40 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.413+0000 7fd2d1600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc112610 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2caa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 0x7fd2cc1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc1a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:60524/0 (socket says 172.21.15.119:60524) 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 -- 172.21.15.119:0/2034549797 learned_addr learned my addr 172.21.15.119:0/2034549797 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2cb400700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc1081e0 0x7fd2cc19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 -- 172.21.15.119:0/2034549797 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 msgr2=0x7fd2cc1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2cc069e40 0x7fd2cc1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 -- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc1081e0 msgr2=0x7fd2cc19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc1081e0 0x7fd2cc19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc19b110 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2cb400700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2cc1081e0 0x7fd2cc19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c2000700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc1a0d30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009fd0 tx=0x7fd2b40094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.414+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4025070 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.415+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc19b3a0 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.415+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2b400ba40 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.415+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004c00 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.415+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc1a76c0 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.415+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2cc10dcc0 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.418+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd2b4028070 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.419+0000 7fd2c3e00700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2b00a2920 0x7fd2b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.419+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fd2b40cfbd0 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.419+0000 7fd2caa00700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2b00a2920 0x7fd2b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.420+0000 7fd2c3e00700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] conn(0x7fd2b00ac4e0 0x7fd2b00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.420+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2b00af050 con 0x7fd2b00ac4e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.420+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2b40cff80 con 0x7fd2cc06a7e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.420+0000 7fd2cb400700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] conn(0x7fd2b00ac4e0 0x7fd2b00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.420+0000 7fd2caa00700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2b00a2920 0x7fd2b00a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc19baf0 tx=0x7fd2bc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.421+0000 7fd2cb400700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] conn(0x7fd2b00ac4e0 0x7fd2b00ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.422+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== osd.0 v2:172.21.15.119:6802/532063015 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd2b00af050 con 0x7fd2b00ac4e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.439+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 --> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2cc103890 con 0x7fd2b00ac4e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2c3e00700 1 -- 172.21.15.119:0/2034549797 <== osd.0 v2:172.21.15.119:6802/532063015 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd2cc103890 con 0x7fd2b00ac4e0 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] conn(0x7fd2b00ac4e0 msgr2=0x7fd2b00ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] conn(0x7fd2b00ac4e0 0x7fd2b00ae980 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2b00a2920 msgr2=0x7fd2b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2b00a2920 0x7fd2b00a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc19baf0 tx=0x7fd2bc009310 comp rx=0 tx=0).stop 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 msgr2=0x7fd2cc1a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.440+0000 7fd2d1600700 1 --2- 172.21.15.119:0/2034549797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2cc06a7e0 0x7fd2cc1a0d30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009fd0 tx=0x7fd2b40094b0 comp rx=0 tx=0).stop 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.441+0000 7fd2cb400700 1 -- 172.21.15.119:0/2034549797 reap_dead start 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.441+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 shutdown_connections 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.441+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 >> 172.21.15.119:0/2034549797 conn(0x7fd2cc077fc0 msgr2=0x7fd2cc10d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.441+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 shutdown_connections 2024-03-28T18:41:57.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:57.441+0000 7fd2d1600700 1 -- 172.21.15.119:0/2034549797 wait complete. 2024-03-28T18:41:58.266 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:41:58 smithi119 ceph-mon[31813]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:41:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:41:58 smithi196 ceph-mon[36300]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:41:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:41:58 smithi131 ceph-mon[36065]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 -- 172.21.15.119:0/1854464529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102440 msgr2=0x7f09b4102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 --2- 172.21.15.119:0/1854464529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102440 0x7f09b4102820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f09a40099c0 tx=0x7f09a4009cd0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 -- 172.21.15.119:0/1854464529 shutdown_connections 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 --2- 172.21.15.119:0/1854464529 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09b4107510 0x7f09b410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 --2- 172.21.15.119:0/1854464529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102440 0x7f09b4102820 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 --2- 172.21.15.119:0/1854464529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09b4102df0 0x7f09b4106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.684+0000 7f09bae00700 1 -- 172.21.15.119:0/1854464529 >> 172.21.15.119:0/1854464529 conn(0x7f09b40792f0 msgr2=0x7f09b4079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:58.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.685+0000 7f09bae00700 1 -- 172.21.15.119:0/1854464529 shutdown_connections 2024-03-28T18:41:58.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.685+0000 7f09bae00700 1 -- 172.21.15.119:0/1854464529 wait complete. 2024-03-28T18:41:58.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 Processor -- start 2024-03-28T18:41:58.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 -- start start 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09b4102440 0x7f09b419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 0x7f09b419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09b4107510 0x7f09b4196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4110400 con 0x7f09b4107510 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4110280 con 0x7f09b4102440 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.686+0000 7f09bae00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4110580 con 0x7f09b4102df0 2024-03-28T18:41:58.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 0x7f09b419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 0x7f09b419da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:60574/0 (socket says 172.21.15.119:60574) 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 -- 172.21.15.119:0/812290673 learned_addr learned my addr 172.21.15.119:0/812290673 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3e00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09b4102440 0x7f09b419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 -- 172.21.15.119:0/812290673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09b4102440 msgr2=0x7f09b419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09b4102440 0x7f09b419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 -- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09b4107510 msgr2=0x7f09b4196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09b4107510 0x7f09b4196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.687+0000 7f09b3400700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4197000 con 0x7f09b4102df0 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09b3400700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 0x7f09b419da70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e970 tx=0x7f09a000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:58.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a000cb50 con 0x7f09b4102df0 2024-03-28T18:41:58.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09a0004d10 con 0x7f09b4102df0 2024-03-28T18:41:58.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b41972f0 con 0x7f09b4102df0 2024-03-28T18:41:58.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a00055c0 con 0x7f09b4102df0 2024-03-28T18:41:58.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.688+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b419b340 con 0x7f09b4102df0 2024-03-28T18:41:58.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.690+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09b419b900 con 0x7f09b4102df0 2024-03-28T18:41:58.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.692+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f09a0004750 con 0x7f09b4102df0 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.694+0000 7f09b0c00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09980a29e0 0x7f09980a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.695+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f09a00cefa0 con 0x7f09b4102df0 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.695+0000 7f09b3e00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09980a29e0 0x7f09980a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.695+0000 7f09b0c00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] conn(0x7f09980ac5a0 0x7f09980aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.695+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09a001de70 con 0x7f09980ac5a0 2024-03-28T18:41:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.695+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09a00cf350 con 0x7f09b4102df0 2024-03-28T18:41:58.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.696+0000 7f09b8e00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] conn(0x7f09980ac5a0 0x7f09980aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:58.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.696+0000 7f09b3e00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09980a29e0 0x7f09980a4ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000c00 tx=0x7f09a4009f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:58.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.697+0000 7f09b8e00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] conn(0x7f09980ac5a0 0x7f09980aea40 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:58.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.699+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== osd.2 v2:172.21.15.196:6800/1413796404 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f09a001de70 con 0x7f09980ac5a0 2024-03-28T18:41:58.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.719+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 --> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09b419bd00 con 0x7f09980ac5a0 2024-03-28T18:41:58.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09b0c00700 1 -- 172.21.15.119:0/812290673 <== osd.2 v2:172.21.15.196:6800/1413796404 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f09b419bd00 con 0x7f09980ac5a0 2024-03-28T18:41:58.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] conn(0x7f09980ac5a0 msgr2=0x7f09980aea40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] conn(0x7f09980ac5a0 0x7f09980aea40 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09980a29e0 msgr2=0x7f09980a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09980a29e0 0x7f09980a4ea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000c00 tx=0x7f09a4009f40 comp rx=0 tx=0).stop 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 msgr2=0x7f09b419da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.720+0000 7f09bae00700 1 --2- 172.21.15.119:0/812290673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09b4102df0 0x7f09b419da70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f09a000e970 tx=0x7f09a000ec80 comp rx=0 tx=0).stop 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.721+0000 7f09b8e00700 1 -- 172.21.15.119:0/812290673 reap_dead start 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.721+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 shutdown_connections 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.721+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 >> 172.21.15.119:0/812290673 conn(0x7f09b40792f0 msgr2=0x7f09b4101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.721+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 shutdown_connections 2024-03-28T18:41:58.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.721+0000 7f09bae00700 1 -- 172.21.15.119:0/812290673 wait complete. 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 -- 172.21.15.119:0/3873422434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc069e40 msgr2=0x7fafbc06a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 --2- 172.21.15.119:0/3873422434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc069e40 0x7fafbc06a2a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009920 tx=0x7fafa8009c30 comp rx=0 tx=0).stop 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 -- 172.21.15.119:0/3873422434 shutdown_connections 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 --2- 172.21.15.119:0/3873422434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafbc06a7e0 0x7fafbc10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 --2- 172.21.15.119:0/3873422434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafbc1081e0 0x7fafbc1085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 --2- 172.21.15.119:0/3873422434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc069e40 0x7fafbc06a2a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 -- 172.21.15.119:0/3873422434 >> 172.21.15.119:0/3873422434 conn(0x7fafbc077fc0 msgr2=0x7fafbc0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 -- 172.21.15.119:0/3873422434 shutdown_connections 2024-03-28T18:41:59.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.593+0000 7fafc1c00700 1 -- 172.21.15.119:0/3873422434 wait complete. 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.594+0000 7fafc1c00700 1 Processor -- start 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.594+0000 7fafc1c00700 1 -- start start 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafbc069e40 0x7fafbc1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 0x7fafbc1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafbc1081e0 0x7fafbc19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc112490 con 0x7fafbc069e40 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc112310 con 0x7fafbc1081e0 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafc1c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc112610 con 0x7fafbc06a7e0 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 0x7fafbc1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafbaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafbc069e40 0x7fafbc1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 0x7fafbc1a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:60552/0 (socket says 172.21.15.119:60552) 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 -- 172.21.15.119:0/4093017927 learned_addr learned my addr 172.21.15.119:0/4093017927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafbb400700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafbc1081e0 0x7fafbc19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:59.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 -- 172.21.15.119:0/4093017927 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafbc1081e0 msgr2=0x7fafbc19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafbc1081e0 0x7fafbc19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 -- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafbc069e40 msgr2=0x7fafbc1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafbc069e40 0x7fafbc1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.595+0000 7fafba000700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc19b170 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafba000700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 0x7fafbc1a0d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000c00 tx=0x7fafa800fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8027070 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa8005620 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8004930 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc19b400 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.596+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc1a7720 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.597+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafbc10dd50 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.600+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fafa802a070 con 0x7fafbc06a7e0 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.602+0000 7fafb3400700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fafa00a29a0 0x7fafa00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:59.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.602+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fafa80cfcd0 con 0x7fafbc06a7e0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.602+0000 7fafb3400700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] conn(0x7fafa00ac560 0x7fafa00aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.602+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafa8029e50 con 0x7fafa00ac560 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.602+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafa80d0120 con 0x7fafbc06a7e0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.603+0000 7fafbaa00700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fafa00a29a0 0x7fafa00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.603+0000 7fafbb400700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] conn(0x7fafa00ac560 0x7fafa00aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.604+0000 7fafbaa00700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fafa00a29a0 0x7fafa00a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafbc19bb50 tx=0x7fafac009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.604+0000 7fafbb400700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] conn(0x7fafa00ac560 0x7fafa00aea00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.605+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== osd.8 v2:172.21.15.196:6816/1701924552 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fafa8029e50 con 0x7fafa00ac560 2024-03-28T18:41:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.626+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 --> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafbc103890 con 0x7fafa00ac560 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.626+0000 7fafb3400700 1 -- 172.21.15.119:0/4093017927 <== osd.8 v2:172.21.15.196:6816/1701924552 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fafbc103890 con 0x7fafa00ac560 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] conn(0x7fafa00ac560 msgr2=0x7fafa00aea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] conn(0x7fafa00ac560 0x7fafa00aea00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fafa00a29a0 msgr2=0x7fafa00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fafa00a29a0 0x7fafa00a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafbc19bb50 tx=0x7fafac009310 comp rx=0 tx=0).stop 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 msgr2=0x7fafbc1a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 --2- 172.21.15.119:0/4093017927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafbc06a7e0 0x7fafbc1a0d90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000c00 tx=0x7fafa800fa70 comp rx=0 tx=0).stop 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafbb400700 1 -- 172.21.15.119:0/4093017927 reap_dead start 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 shutdown_connections 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 >> 172.21.15.119:0/4093017927 conn(0x7fafbc077fc0 msgr2=0x7fafbc10d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:41:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 shutdown_connections 2024-03-28T18:41:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:41:58.627+0000 7fafc1c00700 1 -- 172.21.15.119:0/4093017927 wait complete. 2024-03-28T18:42:00.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:00 smithi119 ceph-mon[31813]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:42:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:00 smithi196 ceph-mon[36300]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:42:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:00 smithi131 ceph-mon[36065]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:42:00.560 INFO:teuthology.orchestra.run.smithi119.stdout:34359738450 2024-03-28T18:42:00.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T18:42:02.191 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:02 smithi119 ceph-mon[31813]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:02 smithi196 ceph-mon[36300]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:02 smithi131 ceph-mon[36065]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 -- 172.21.15.119:0/1512277390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb81023d0 msgr2=0x7fedb81027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 --2- 172.21.15.119:0/1512277390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb81023d0 0x7fedb81027b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feda8009990 tx=0x7feda8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 -- 172.21.15.119:0/1512277390 shutdown_connections 2024-03-28T18:42:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 --2- 172.21.15.119:0/1512277390 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb810f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 --2- 172.21.15.119:0/1512277390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb81023d0 0x7fedb81027b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 --2- 172.21.15.119:0/1512277390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedb8102d80 0x7fedb8106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 -- 172.21.15.119:0/1512277390 >> 172.21.15.119:0/1512277390 conn(0x7fedb80fdc40 msgr2=0x7fedb8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 -- 172.21.15.119:0/1512277390 shutdown_connections 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.538+0000 7fedbf200700 1 -- 172.21.15.119:0/1512277390 wait complete. 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.539+0000 7fedbf200700 1 Processor -- start 2024-03-28T18:42:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.539+0000 7fedbf200700 1 -- start start 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedb81023d0 0x7fedb810e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb8102d80 0x7fedb810e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb81a3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8112510 con 0x7fedb81023d0 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8112390 con 0x7fedb8102d80 2024-03-28T18:42:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbf200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8112690 con 0x7fedb81074b0 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbd200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb81a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedb7400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb8102d80 0x7fedb810e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbd200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb81a3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40486/0 (socket says 172.21.15.119:40486) 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedbd200700 1 -- 172.21.15.119:0/4264004861 learned_addr learned my addr 172.21.15.119:0/4264004861 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.540+0000 7fedb7e00700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedb81023d0 0x7fedb810e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 -- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb8102d80 msgr2=0x7fedb810e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fedb8102d80 0x7fedb810e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 -- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedb81023d0 msgr2=0x7fedb810e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedb81023d0 0x7fedb810e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb81a3f80 con 0x7fedb81074b0 2024-03-28T18:42:02.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbd200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb81a3840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb000ef30 tx=0x7fedb000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb00092e0 con 0x7fedb81074b0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedb0004bb0 con 0x7fedb81074b0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb0004280 con 0x7fedb81074b0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb81a4210 con 0x7fedb81074b0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.541+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb81abaf0 con 0x7fedb81074b0 2024-03-28T18:42:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.543+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedb81001e0 con 0x7fedb81074b0 2024-03-28T18:42:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.544+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fedb001b030 con 0x7fedb81074b0 2024-03-28T18:42:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb4c00700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed9c0a2920 0x7fed9c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fedb00ce750 con 0x7fedb81074b0 2024-03-28T18:42:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb7e00700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed9c0a2920 0x7fed9c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb4c00700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] conn(0x7fed9c0ac4e0 0x7fed9c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed9c0af050 con 0x7fed9c0ac4e0 2024-03-28T18:42:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedb00d5050 con 0x7fedb81074b0 2024-03-28T18:42:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.546+0000 7fedb7400700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] conn(0x7fed9c0ac4e0 0x7fed9c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.547+0000 7fedb7e00700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed9c0a2920 0x7fed9c0a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feda800b2e0 tx=0x7feda801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.547+0000 7fedb7400700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] conn(0x7fed9c0ac4e0 0x7fed9c0ae980 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.548+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== osd.4 v2:172.21.15.131:6808/1834627175 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fed9c0af050 con 0x7fed9c0ac4e0 2024-03-28T18:42:02.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.566+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 --> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedb81038b0 con 0x7fed9c0ac4e0 2024-03-28T18:42:02.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedb4c00700 1 -- 172.21.15.119:0/4264004861 <== osd.4 v2:172.21.15.131:6808/1834627175 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fedb81038b0 con 0x7fed9c0ac4e0 2024-03-28T18:42:02.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] conn(0x7fed9c0ac4e0 msgr2=0x7fed9c0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] conn(0x7fed9c0ac4e0 0x7fed9c0ae980 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed9c0a2920 msgr2=0x7fed9c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed9c0a2920 0x7fed9c0a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7feda800b2e0 tx=0x7feda801a040 comp rx=0 tx=0).stop 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 msgr2=0x7fedb81a3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 --2- 172.21.15.119:0/4264004861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedb81074b0 0x7fedb81a3840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedb000ef30 tx=0x7fedb000c3a0 comp rx=0 tx=0).stop 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbd200700 1 -- 172.21.15.119:0/4264004861 reap_dead start 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 shutdown_connections 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.567+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 >> 172.21.15.119:0/4264004861 conn(0x7fedb80fdc40 msgr2=0x7fedb80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.568+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 shutdown_connections 2024-03-28T18:42:02.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.568+0000 7fedbf200700 1 -- 172.21.15.119:0/4264004861 wait complete. 2024-03-28T18:42:03.220 INFO:teuthology.orchestra.run.smithi119.stdout:257698037798 2024-03-28T18:42:03.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T18:42:03.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 -- 172.21.15.119:0/644910592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f0113360 msgr2=0x7f10f0115750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 --2- 172.21.15.119:0/644910592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f0113360 0x7f10f0115750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b0a0 tx=0x7f10e800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 -- 172.21.15.119:0/644910592 shutdown_connections 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 --2- 172.21.15.119:0/644910592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f0113360 0x7f10f0115750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 --2- 172.21.15.119:0/644910592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f00743c0 0x7f10f00747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 --2- 172.21.15.119:0/644910592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0074ce0 0x7f10f0112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 -- 172.21.15.119:0/644910592 >> 172.21.15.119:0/644910592 conn(0x7f10f00792f0 msgr2=0x7f10f0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 -- 172.21.15.119:0/644910592 shutdown_connections 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.412+0000 7f10f6400700 1 -- 172.21.15.119:0/644910592 wait complete. 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.413+0000 7f10f6400700 1 Processor -- start 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.413+0000 7f10f6400700 1 -- start start 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f00743c0 0x7f10f019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 0x7f10f019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0113360 0x7f10f01a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0118b40 con 0x7f10f0113360 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f01189c0 con 0x7f10f00743c0 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10f6400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f0118cc0 con 0x7f10f0074ce0 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 0x7f10f019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10ef400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f00743c0 0x7f10f019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 0x7f10f019f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40480/0 (socket says 172.21.15.119:40480) 2024-03-28T18:42:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 -- 172.21.15.119:0/3332277250 learned_addr learned my addr 172.21.15.119:0/3332277250 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10efe00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0113360 0x7f10f01a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 -- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f00743c0 msgr2=0x7f10f019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f00743c0 0x7f10f019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 -- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0113360 msgr2=0x7f10f01a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0113360 0x7f10f01a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10e7e00700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f01a3f00 con 0x7f10f0074ce0 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10efe00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10f0113360 0x7f10f01a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.414+0000 7f10ef400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10f00743c0 0x7f10f019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10e7e00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 0x7f10f019f430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e970 tx=0x7f10d800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d800cbd0 con 0x7f10f0074ce0 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10d8004500 con 0x7f10f0074ce0 2024-03-28T18:42:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f01a41f0 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8003e30 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.415+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f01aba70 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.416+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10f0112b00 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.418+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f10d8004000 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ecc00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f10d00a2920 0x7f10d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f10d80cf150 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ef400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f10d00a2920 0x7f10d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ecc00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] conn(0x7f10d00ac4e0 0x7f10d00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10d00af050 con 0x7f10d00ac4e0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10d80cf500 con 0x7f10f0074ce0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.420+0000 7f10efe00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] conn(0x7f10d00ac4e0 0x7f10d00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.421+0000 7f10ef400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f10d00a2920 0x7f10d00a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10e0011490 tx=0x7f10e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:03.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.421+0000 7f10efe00700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] conn(0x7f10d00ac4e0 0x7f10d00ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.422+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== osd.7 v2:172.21.15.131:6816/417830389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10d00af050 con 0x7f10d00ac4e0 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.439+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 --> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10f0075af0 con 0x7f10d00ac4e0 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10ecc00700 1 -- 172.21.15.119:0/3332277250 <== osd.7 v2:172.21.15.131:6816/417830389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f10f0075af0 con 0x7f10d00ac4e0 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] conn(0x7f10d00ac4e0 msgr2=0x7f10d00ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] conn(0x7f10d00ac4e0 0x7f10d00ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f10d00a2920 msgr2=0x7f10d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f10d00a2920 0x7f10d00a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f10e0011490 tx=0x7f10e0009260 comp rx=0 tx=0).stop 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 msgr2=0x7f10f019f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10f6400700 1 --2- 172.21.15.119:0/3332277250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10f0074ce0 0x7f10f019f430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10d800e970 tx=0x7f10d800ec80 comp rx=0 tx=0).stop 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.440+0000 7f10efe00700 1 -- 172.21.15.119:0/3332277250 reap_dead start 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.441+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 shutdown_connections 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.441+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 >> 172.21.15.119:0/3332277250 conn(0x7f10f00792f0 msgr2=0x7f10f0112120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.441+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 shutdown_connections 2024-03-28T18:42:03.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:02.441+0000 7f10f6400700 1 -- 172.21.15.119:0/3332277250 wait complete. 2024-03-28T18:42:04.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:04 smithi119 ceph-mon[31813]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:04.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:04 smithi196 ceph-mon[36300]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:04 smithi131 ceph-mon[36065]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.783+0000 7f46dc000700 1 -- 172.21.15.119:0/1187901369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 msgr2=0x7f46d4101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.783+0000 7f46dc000700 1 --2- 172.21.15.119:0/1187901369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 0x7f46d4101c40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f46c80099c0 tx=0x7f46c8009cd0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 -- 172.21.15.119:0/1187901369 shutdown_connections 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 --2- 172.21.15.119:0/1187901369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d4104e00 0x7f46d41052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 --2- 172.21.15.119:0/1187901369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 0x7f46d4101c40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 --2- 172.21.15.119:0/1187901369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4102180 0x7f46d4102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 -- 172.21.15.119:0/1187901369 >> 172.21.15.119:0/1187901369 conn(0x7f46d4077130 msgr2=0x7f46d4077540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 -- 172.21.15.119:0/1187901369 shutdown_connections 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.784+0000 7f46dc000700 1 -- 172.21.15.119:0/1187901369 wait complete. 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 Processor -- start 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 -- start start 2024-03-28T18:42:04.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d4102180 0x7f46d419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 0x7f46d419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 0x7f46d41a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4118c00 con 0x7f46d4102180 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4118a80 con 0x7f46d4107290 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.785+0000 7f46dc000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4118d80 con 0x7f46d4104e00 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 0x7f46d419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d9600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d4102180 0x7f46d419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 0x7f46d419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40526/0 (socket says 172.21.15.119:40526) 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 -- 172.21.15.119:0/1760559123 learned_addr learned my addr 172.21.15.119:0/1760559123 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46da000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 0x7f46d41a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 -- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 msgr2=0x7f46d41a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f46d4107290 0x7f46d41a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 -- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d4102180 msgr2=0x7f46d419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d4102180 0x7f46d419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d41a40b0 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d8c00700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 0x7f46d419f630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46c000eb30 tx=0x7f46c000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000cbd0 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.786+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d41a43a0 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.787+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c0004500 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.787+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0010430 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.787+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d41abc20 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.787+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f46d4100b00 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.790+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f46c0003f00 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d2000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46bc0a2920 0x7f46bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f46c00cf270 con 0x7f46d4104e00 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d9600700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46bc0a2920 0x7f46bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d2000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] conn(0x7f46bc0ac4e0 0x7f46bc0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46c0012ec0 con 0x7f46bc0ac4e0 2024-03-28T18:42:04.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.791+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46c00cf620 con 0x7f46d4104e00 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.792+0000 7f46da000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] conn(0x7f46bc0ac4e0 0x7f46bc0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.792+0000 7f46d9600700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46bc0a2920 0x7f46bc0a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f46c8000c00 tx=0x7f46c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.792+0000 7f46da000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] conn(0x7f46bc0ac4e0 0x7f46bc0ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.793+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== osd.1 v2:172.21.15.131:6800/1240079334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f46c0012ec0 con 0x7f46bc0ac4e0 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.812+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 --> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46d4101750 con 0x7f46bc0ac4e0 2024-03-28T18:42:04.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.812+0000 7f46d2000700 1 -- 172.21.15.119:0/1760559123 <== osd.1 v2:172.21.15.131:6800/1240079334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46d4101750 con 0x7f46bc0ac4e0 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] conn(0x7f46bc0ac4e0 msgr2=0x7f46bc0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] conn(0x7f46bc0ac4e0 0x7f46bc0ae980 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46bc0a2920 msgr2=0x7f46bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46bc0a2920 0x7f46bc0a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f46c8000c00 tx=0x7f46c8005fb0 comp rx=0 tx=0).stop 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 msgr2=0x7f46d419f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 --2- 172.21.15.119:0/1760559123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46d4104e00 0x7f46d419f630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f46c000eb30 tx=0x7f46c000ee40 comp rx=0 tx=0).stop 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46da000700 1 -- 172.21.15.119:0/1760559123 reap_dead start 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 shutdown_connections 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 >> 172.21.15.119:0/1760559123 conn(0x7f46d4077130 msgr2=0x7f46d4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 shutdown_connections 2024-03-28T18:42:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:04.813+0000 7f46dc000700 1 -- 172.21.15.119:0/1760559123 wait complete. 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.750+0000 7f5e41800700 1 -- 172.21.15.119:0/3663053094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 msgr2=0x7f5e3c115670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.750+0000 7f5e41800700 1 --2- 172.21.15.119:0/3663053094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c115670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400b0a0 tx=0x7f5e3400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 -- 172.21.15.119:0/3663053094 shutdown_connections 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 --2- 172.21.15.119:0/3663053094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c115670 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 --2- 172.21.15.119:0/3663053094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c10e7a0 0x7f5e3c10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 --2- 172.21.15.119:0/3663053094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e3c1087a0 0x7f5e3c108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 -- 172.21.15.119:0/3663053094 >> 172.21.15.119:0/3663053094 conn(0x7f5e3c075960 msgr2=0x7f5e3c077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 -- 172.21.15.119:0/3663053094 shutdown_connections 2024-03-28T18:42:05.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.751+0000 7f5e41800700 1 -- 172.21.15.119:0/3663053094 wait complete. 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.752+0000 7f5e41800700 1 Processor -- start 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.752+0000 7f5e41800700 1 -- start start 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.752+0000 7f5e41800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c1087a0 0x7f5e3c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.752+0000 7f5e41800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.752+0000 7f5e41800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e3c10e7a0 0x7f5e3c1a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e41800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c118b30 con 0x7f5e3c10e7a0 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e41800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1189b0 con 0x7f5e3c1087a0 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e41800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c118cb0 con 0x7f5e3c109160 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3aa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c1087a0 0x7f5e3c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c19f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40502/0 (socket says 172.21.15.119:40502) 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3aa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c1087a0 0x7f5e3c19eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34184/0 (socket says 172.21.15.119:34184) 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3b400700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e3c10e7a0 0x7f5e3c1a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 -- 172.21.15.119:0/2667431260 learned_addr learned my addr 172.21.15.119:0/2667431260 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:05.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 -- 172.21.15.119:0/2667431260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c1087a0 msgr2=0x7f5e3c19eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5e3c1087a0 0x7f5e3c19eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 -- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e3c10e7a0 msgr2=0x7f5e3c1a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e3c10e7a0 0x7f5e3c1a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c1a3eb0 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.753+0000 7f5e3a000700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c19f3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9a0 tx=0x7f5e2800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.754+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800ca10 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.754+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1a41a0 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.754+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e2800cbb0 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.754+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28018680 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.754+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1aba20 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.755+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e3c062920 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.757+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5e28010430 con 0x7f5e3c109160 2024-03-28T18:42:05.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.758+0000 7f5e33400700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e200a2920 0x7f5e200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.758+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f5e280cf6b0 con 0x7f5e3c109160 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.758+0000 7f5e3aa00700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e200a2920 0x7f5e200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e33400700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] conn(0x7f5e200ac4e0 0x7f5e200ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e2801bec0 con 0x7f5e200ac4e0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e280cfa60 con 0x7f5e3c109160 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e3b400700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] conn(0x7f5e200ac4e0 0x7f5e200ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e3aa00700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e200a2920 0x7f5e200a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009be0 tx=0x7f5e2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.759+0000 7f5e3b400700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] conn(0x7f5e200ac4e0 0x7f5e200ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.760+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== osd.5 v2:172.21.15.196:6808/3397729850 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5e2801bec0 con 0x7f5e200ac4e0 2024-03-28T18:42:05.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.778+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 --> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e3c062da0 con 0x7f5e200ac4e0 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e33400700 1 -- 172.21.15.119:0/2667431260 <== osd.5 v2:172.21.15.196:6808/3397729850 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e3c062da0 con 0x7f5e200ac4e0 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] conn(0x7f5e200ac4e0 msgr2=0x7f5e200ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] conn(0x7f5e200ac4e0 0x7f5e200ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e200a2920 msgr2=0x7f5e200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5e200a2920 0x7f5e200a4de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009be0 tx=0x7f5e2c009260 comp rx=0 tx=0).stop 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 msgr2=0x7f5e3c19f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e41800700 1 --2- 172.21.15.119:0/2667431260 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e3c109160 0x7f5e3c19f3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9a0 tx=0x7f5e2800ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.779+0000 7f5e3b400700 1 -- 172.21.15.119:0/2667431260 reap_dead start 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.780+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 shutdown_connections 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.780+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 >> 172.21.15.119:0/2667431260 conn(0x7f5e3c075960 msgr2=0x7f5e3c075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.780+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 shutdown_connections 2024-03-28T18:42:05.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:03.780+0000 7f5e41800700 1 -- 172.21.15.119:0/2667431260 wait complete. 2024-03-28T18:42:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:06 smithi196 ceph-mon[36300]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:06 smithi131 ceph-mon[36065]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:06.453 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:06 smithi119 ceph-mon[31813]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:07.042 INFO:teuthology.orchestra.run.smithi119.stdout:146028888126 2024-03-28T18:42:07.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T18:42:08.331 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:08 smithi119 ceph-mon[31813]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:08.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:08 smithi196 ceph-mon[36300]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:08 smithi131 ceph-mon[36065]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-28T18:42:08.812 INFO:teuthology.orchestra.run.smithi119.stdout:55834574927 2024-03-28T18:42:08.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T18:42:09.803 INFO:teuthology.orchestra.run.smithi119.stdout:223338299436 2024-03-28T18:42:09.804 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T18:42:10.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:10 smithi119 ceph-mon[31813]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:10.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 -- 172.21.15.119:0/2825760653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 msgr2=0x7fe9cc106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 --2- 172.21.15.119:0/2825760653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc106e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe9b80098f0 tx=0x7fe9b8009c00 comp rx=0 tx=0).stop 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 -- 172.21.15.119:0/2825760653 shutdown_connections 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 --2- 172.21.15.119:0/2825760653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9cc107510 0x7fe9cc10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 --2- 172.21.15.119:0/2825760653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 --2- 172.21.15.119:0/2825760653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc106e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 -- 172.21.15.119:0/2825760653 >> 172.21.15.119:0/2825760653 conn(0x7fe9cc0792f0 msgr2=0x7fe9cc079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 -- 172.21.15.119:0/2825760653 shutdown_connections 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.021+0000 7fe9d1800700 1 -- 172.21.15.119:0/2825760653 wait complete. 2024-03-28T18:42:10.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 Processor -- start 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 -- start start 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9cc107510 0x7fe9cc19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc112520 con 0x7fe9cc107510 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc1123a0 con 0x7fe9cc102df0 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9d1800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc1126a0 con 0x7fe9cc102440 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9caa00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9caa00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46222/0 (socket says 172.21.15.119:46222) 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9caa00700 1 -- 172.21.15.119:0/1264542790 learned_addr learned my addr 172.21.15.119:0/1264542790 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.023+0000 7fe9ca000700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9cb400700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9cc107510 0x7fe9cc19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 -- 172.21.15.119:0/1264542790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 msgr2=0x7fe9cc1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 -- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9cc107510 msgr2=0x7fe9cc19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9cc107510 0x7fe9cc19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc19b1b0 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9ca000700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9cc102df0 0x7fe9cc1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9caa00700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc1a0890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00ec10 tx=0x7fe9bc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.024+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00cc90 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.025+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc19b4a0 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.025+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9bc00ce30 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.025+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc010690 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.025+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc1a7760 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.025+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9cc100c20 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.028+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe9bc003680 con 0x7fe9cc102440 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.029+0000 7fe9c3400700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe9b00a2920 0x7fe9b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.029+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fe9bc0cf250 con 0x7fe9cc102440 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.029+0000 7fe9ca000700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe9b00a2920 0x7fe9b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9c3400700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] conn(0x7fe9b00ac4e0 0x7fe9b00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9bc012ec0 con 0x7fe9b00ac4e0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9bc0cf600 con 0x7fe9cc102440 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9cb400700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] conn(0x7fe9b00ac4e0 0x7fe9b00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9ca000700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe9b00a2920 0x7fe9b00a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009fd0 tx=0x7fe9b801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.030+0000 7fe9cb400700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] conn(0x7fe9b00ac4e0 0x7fe9b00ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.032+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== osd.6 v2:172.21.15.119:6818/2797182795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe9bc012ec0 con 0x7fe9b00ac4e0 2024-03-28T18:42:10.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.049+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 --> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9cc103910 con 0x7fe9b00ac4e0 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.050+0000 7fe9c3400700 1 -- 172.21.15.119:0/1264542790 <== osd.6 v2:172.21.15.119:6818/2797182795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe9cc103910 con 0x7fe9b00ac4e0 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] conn(0x7fe9b00ac4e0 msgr2=0x7fe9b00ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] conn(0x7fe9b00ac4e0 0x7fe9b00ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe9b00a2920 msgr2=0x7fe9b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe9b00a2920 0x7fe9b00a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009fd0 tx=0x7fe9b801a040 comp rx=0 tx=0).stop 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 msgr2=0x7fe9cc1a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 --2- 172.21.15.119:0/1264542790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9cc102440 0x7fe9cc1a0890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00ec10 tx=0x7fe9bc00ef20 comp rx=0 tx=0).stop 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9cb400700 1 -- 172.21.15.119:0/1264542790 reap_dead start 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 shutdown_connections 2024-03-28T18:42:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 >> 172.21.15.119:0/1264542790 conn(0x7fe9cc0792f0 msgr2=0x7fe9cc100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 shutdown_connections 2024-03-28T18:42:10.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:10.051+0000 7fe9d1800700 1 -- 172.21.15.119:0/1264542790 wait complete. 2024-03-28T18:42:10.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:10 smithi196 ceph-mon[36300]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:10 smithi131 ceph-mon[36065]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 -- 172.21.15.119:0/129051338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 msgr2=0x7f35e010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 --2- 172.21.15.119:0/129051338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e010ef70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35d800b0a0 tx=0x7f35d800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 -- 172.21.15.119:0/129051338 shutdown_connections 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 --2- 172.21.15.119:0/129051338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e010ef70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 --2- 172.21.15.119:0/129051338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e01070d0 0x7f35e01074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 --2- 172.21.15.119:0/129051338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e0100940 0x7f35e0100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 -- 172.21.15.119:0/129051338 >> 172.21.15.119:0/129051338 conn(0x7f35e00762d0 msgr2=0x7f35e00766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 -- 172.21.15.119:0/129051338 shutdown_connections 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.573+0000 7f35e5800700 1 -- 172.21.15.119:0/129051338 wait complete. 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.574+0000 7f35e5800700 1 Processor -- start 2024-03-28T18:42:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 -- start start 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e0100940 0x7f35e01a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e01a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e01070d0 0x7f35e019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e01123d0 con 0x7f35e0100940 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0112250 con 0x7f35e0101300 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e5800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0112550 con 0x7f35e01070d0 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35dfe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e0100940 0x7f35e01a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35d7e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e01a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35d7e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e01a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:34210/0 (socket says 172.21.15.119:34210) 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35d7e00700 1 -- 172.21.15.119:0/2685891352 learned_addr learned my addr 172.21.15.119:0/2685891352 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.575+0000 7f35e4e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e01070d0 0x7f35e019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 -- 172.21.15.119:0/2685891352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e01070d0 msgr2=0x7f35e019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e01070d0 0x7f35e019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 -- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e0100940 msgr2=0x7f35e01a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e0100940 0x7f35e01a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e019af80 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35e4e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35e01070d0 0x7f35e019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35dfe00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e0100940 0x7f35e01a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35d7e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e01a0b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35c800e9a0 tx=0x7f35c800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c800ca10 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35c800cbb0 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e019b270 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8018680 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.576+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e01a7480 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.577+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f35e00ff210 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.579+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f35c8010430 con 0x7f35e0101300 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.580+0000 7f35dd600700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f35c40a29a0 0x7f35c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f35c80cf670 con 0x7f35e0101300 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dfe00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f35c40a29a0 0x7f35c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dd600700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] conn(0x7f35c40ac560 0x7f35c40aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35c801bec0 con 0x7f35c40ac560 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35c80cfa20 con 0x7f35e0101300 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35e4e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] conn(0x7f35c40ac560 0x7f35c40aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.581+0000 7f35dfe00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f35c40a29a0 0x7f35c40a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f35e0074550 tx=0x7f35d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.582+0000 7f35e4e00700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] conn(0x7f35c40ac560 0x7f35c40aea00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.583+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== osd.3 v2:172.21.15.119:6810/837858906 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f35c801bec0 con 0x7f35c40ac560 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.599+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 --> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f35e00699d0 con 0x7f35c40ac560 2024-03-28T18:42:10.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.600+0000 7f35dd600700 1 -- 172.21.15.119:0/2685891352 <== osd.3 v2:172.21.15.119:6810/837858906 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f35e00699d0 con 0x7f35c40ac560 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.600+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] conn(0x7f35c40ac560 msgr2=0x7f35c40aea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.600+0000 7f35e5800700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] conn(0x7f35c40ac560 0x7f35c40aea00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f35c40a29a0 msgr2=0x7f35c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f35c40a29a0 0x7f35c40a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f35e0074550 tx=0x7f35d0009260 comp rx=0 tx=0).stop 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 msgr2=0x7f35e01a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 --2- 172.21.15.119:0/2685891352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f35e0101300 0x7f35e01a0b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35c800e9a0 tx=0x7f35c800ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e4e00700 1 -- 172.21.15.119:0/2685891352 reap_dead start 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 shutdown_connections 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 >> 172.21.15.119:0/2685891352 conn(0x7f35e00762d0 msgr2=0x7f35e00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 shutdown_connections 2024-03-28T18:42:10.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:08.601+0000 7f35e5800700 1 -- 172.21.15.119:0/2685891352 wait complete. 2024-03-28T18:42:11.615 INFO:teuthology.orchestra.run.smithi119.stdout:111669149764 2024-03-28T18:42:11.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T18:42:12.126 INFO:teuthology.orchestra.run.smithi119.stdout:77309411400 2024-03-28T18:42:12.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T18:42:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:12 smithi119 ceph-mon[31813]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T18:42:12.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:12 smithi196 ceph-mon[36300]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T18:42:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:12 smithi131 ceph-mon[36065]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-28T18:42:12.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 -- 172.21.15.119:0/282964352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f082810eb90 msgr2=0x7f0828110f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 --2- 172.21.15.119:0/282964352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f082810eb90 0x7f0828110f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f082000b0a0 tx=0x7f082000b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 -- 172.21.15.119:0/282964352 shutdown_connections 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 --2- 172.21.15.119:0/282964352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f082810eb90 0x7f0828110f80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 --2- 172.21.15.119:0/282964352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08280743c0 0x7f08280747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 --2- 172.21.15.119:0/282964352 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 0x7f082810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 -- 172.21.15.119:0/282964352 >> 172.21.15.119:0/282964352 conn(0x7f08280792f0 msgr2=0x7f0828079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:12.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 -- 172.21.15.119:0/282964352 shutdown_connections 2024-03-28T18:42:12.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.693+0000 7f082d800700 1 -- 172.21.15.119:0/282964352 wait complete. 2024-03-28T18:42:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.694+0000 7f082d800700 1 Processor -- start 2024-03-28T18:42:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.694+0000 7f082d800700 1 -- start start 2024-03-28T18:42:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 0x7f08281a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 0x7f08281a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f082810eb90 0x7f082819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0828114770 con 0x7f082810eb90 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08281145f0 con 0x7f0828074ce0 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f082d800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08281148f0 con 0x7f08280743c0 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f081fe00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 0x7f08281a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f0826a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 0x7f08281a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f081fe00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 0x7f08281a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56218/0 (socket says 172.21.15.119:56218) 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f0826a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 0x7f08281a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46234/0 (socket says 172.21.15.119:46234) 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.695+0000 7f081fe00700 1 -- 172.21.15.119:0/3831528051 learned_addr learned my addr 172.21.15.119:0/3831528051 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0827400700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f082810eb90 0x7f082819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 -- 172.21.15.119:0/3831528051 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 msgr2=0x7f08281a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0828074ce0 0x7f08281a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 -- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f082810eb90 msgr2=0x7f082819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f082810eb90 0x7f082819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082819b0f0 con 0x7f08280743c0 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f0826a00700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 0x7f08281a07d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081800cbd0 con 0x7f08280743c0 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f081800cd70 con 0x7f08280743c0 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082819b3e0 con 0x7f08280743c0 2024-03-28T18:42:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818018710 con 0x7f08280743c0 2024-03-28T18:42:12.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.696+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08281a76a0 con 0x7f08280743c0 2024-03-28T18:42:12.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.697+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f08280ffd40 con 0x7f08280743c0 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.700+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0818010430 con 0x7f08280743c0 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081f400700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f080c0a29a0 0x7f080c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f08180cf5f0 con 0x7f08280743c0 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081fe00700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f080c0a29a0 0x7f080c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081f400700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] conn(0x7f080c0ac560 0x7f080c0aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f081801bec0 con 0x7f080c0ac560 2024-03-28T18:42:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.701+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08180cf9a0 con 0x7f08280743c0 2024-03-28T18:42:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.702+0000 7f0827400700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] conn(0x7f080c0ac560 0x7f080c0aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.702+0000 7f081fe00700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f080c0a29a0 0x7f080c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0828110890 tx=0x7f0810009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.702+0000 7f0827400700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] conn(0x7f080c0ac560 0x7f080c0aea00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.703+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== osd.9 v2:172.21.15.119:6826/1454392710 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f081801bec0 con 0x7f080c0ac560 2024-03-28T18:42:12.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.721+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 --> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0828075af0 con 0x7f080c0ac560 2024-03-28T18:42:12.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f081f400700 1 -- 172.21.15.119:0/3831528051 <== osd.9 v2:172.21.15.119:6826/1454392710 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0828075af0 con 0x7f080c0ac560 2024-03-28T18:42:12.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] conn(0x7f080c0ac560 msgr2=0x7f080c0aea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] conn(0x7f080c0ac560 0x7f080c0aea00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f080c0a29a0 msgr2=0x7f080c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f080c0a29a0 0x7f080c0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0828110890 tx=0x7f0810009260 comp rx=0 tx=0).stop 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 msgr2=0x7f08281a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 --2- 172.21.15.119:0/3831528051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08280743c0 0x7f08281a07d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f081800e9f0 tx=0x7f081800ed00 comp rx=0 tx=0).stop 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f0827400700 1 -- 172.21.15.119:0/3831528051 reap_dead start 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 shutdown_connections 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.722+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 >> 172.21.15.119:0/3831528051 conn(0x7f08280792f0 msgr2=0x7f08280ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.723+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 shutdown_connections 2024-03-28T18:42:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.723+0000 7f082d800700 1 -- 172.21.15.119:0/3831528051 wait complete. 2024-03-28T18:42:12.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 -- 172.21.15.119:0/72021862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180102df0 msgr2=0x7f6180106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 --2- 172.21.15.119:0/72021862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180102df0 0x7f6180106e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6168009990 tx=0x7f6168009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 -- 172.21.15.119:0/72021862 shutdown_connections 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 --2- 172.21.15.119:0/72021862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6180107510 0x7f618010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 --2- 172.21.15.119:0/72021862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f6180102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 --2- 172.21.15.119:0/72021862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180102df0 0x7f6180106e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 -- 172.21.15.119:0/72021862 >> 172.21.15.119:0/72021862 conn(0x7f61800792f0 msgr2=0x7f6180079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 -- 172.21.15.119:0/72021862 shutdown_connections 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.930+0000 7f6185000700 1 -- 172.21.15.119:0/72021862 wait complete. 2024-03-28T18:42:12.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 Processor -- start 2024-03-28T18:42:12.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 -- start start 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f618019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6180102df0 0x7f618019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180107510 0x7f61801988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180112520 con 0x7f6180102df0 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61801123a0 con 0x7f6180107510 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.932+0000 7f6185000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61801126a0 con 0x7f6180102440 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f618019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f618019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46266/0 (socket says 172.21.15.119:46266) 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 -- 172.21.15.119:0/1568491082 learned_addr learned my addr 172.21.15.119:0/1568491082 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f6175400700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6180102df0 0x7f618019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617e800700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180107510 0x7f61801988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 -- 172.21.15.119:0/1568491082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180107510 msgr2=0x7f61801988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6180107510 0x7f61801988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 -- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6180102df0 msgr2=0x7f618019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6180102df0 0x7f618019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180199020 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.933+0000 7f617de00700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f618019e700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f617000e9a0 tx=0x7f617000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.934+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617000ca10 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.934+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f617000cbb0 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.934+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170018680 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.934+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6180199310 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.934+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61801a55d0 con 0x7f6180102440 2024-03-28T18:42:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.935+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6180062860 con 0x7f6180102440 2024-03-28T18:42:12.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.937+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6170010430 con 0x7f6180102440 2024-03-28T18:42:12.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.938+0000 7f6177400700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f61640a29a0 0x7f61640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.938+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f61700cf670 con 0x7f6180102440 2024-03-28T18:42:12.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f6175400700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f61640a29a0 0x7f61640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f6177400700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] conn(0x7f61640ac560 0x7f61640aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f617001bec0 con 0x7f61640ac560 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61700cfa20 con 0x7f6180102440 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f617e800700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] conn(0x7f61640ac560 0x7f61640aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.939+0000 7f6175400700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f61640a29a0 0x7f61640a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6180106e70 tx=0x7f616800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.940+0000 7f617e800700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] conn(0x7f61640ac560 0x7f61640aea00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:12.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.940+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== osd.11 v2:172.21.15.196:6824/2114329937 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f617001bec0 con 0x7f61640ac560 2024-03-28T18:42:12.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.959+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 --> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6180062cb0 con 0x7f61640ac560 2024-03-28T18:42:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.959+0000 7f6177400700 1 -- 172.21.15.119:0/1568491082 <== osd.11 v2:172.21.15.196:6824/2114329937 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6180062cb0 con 0x7f61640ac560 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.959+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] conn(0x7f61640ac560 msgr2=0x7f61640aea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.959+0000 7f6185000700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] conn(0x7f61640ac560 0x7f61640aea00 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f61640a29a0 msgr2=0x7f61640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f61640a29a0 0x7f61640a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6180106e70 tx=0x7f616800b260 comp rx=0 tx=0).stop 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 msgr2=0x7f618019e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 --2- 172.21.15.119:0/1568491082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6180102440 0x7f618019e700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f617000e9a0 tx=0x7f617000ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f617e800700 1 -- 172.21.15.119:0/1568491082 reap_dead start 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 shutdown_connections 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 >> 172.21.15.119:0/1568491082 conn(0x7f61800792f0 msgr2=0x7f61801002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 shutdown_connections 2024-03-28T18:42:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:12.960+0000 7f6185000700 1 -- 172.21.15.119:0/1568491082 wait complete. 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 -- 172.21.15.119:0/2507794737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f681096e0 msgr2=0x7f6f681112b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/2507794737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f681096e0 0x7f6f681112b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f60009f90 tx=0x7f6f6000b330 comp rx=0 tx=0).stop 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 -- 172.21.15.119:0/2507794737 shutdown_connections 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/2507794737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f681096e0 0x7f6f681112b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/2507794737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104600 0x7f6f681049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/2507794737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f68104fb0 0x7f6f68109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 -- 172.21.15.119:0/2507794737 >> 172.21.15.119:0/2507794737 conn(0x7f6f680ffe90 msgr2=0x7f6f681022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 -- 172.21.15.119:0/2507794737 shutdown_connections 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.158+0000 7f6f6fe00700 1 -- 172.21.15.119:0/2507794737 wait complete. 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.159+0000 7f6f6fe00700 1 Processor -- start 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.159+0000 7f6f6fe00700 1 -- start start 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 0x7f6f681a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104fb0 0x7f6f681a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f681096e0 0x7f6f6819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f68114720 con 0x7f6f68104fb0 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f681145a0 con 0x7f6f68104600 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6fe00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f681148a0 con 0x7f6f681096e0 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 0x7f6f681a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 0x7f6f681a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56254/0 (socket says 172.21.15.119:56254) 2024-03-28T18:42:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 -- 172.21.15.119:0/4162328121 learned_addr learned my addr 172.21.15.119:0/4162328121 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6ca00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104fb0 0x7f6f681a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6de00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f681096e0 0x7f6f6819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 -- 172.21.15.119:0/4162328121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f681096e0 msgr2=0x7f6f6819a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f681096e0 0x7f6f6819a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 -- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104fb0 msgr2=0x7f6f681a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104fb0 0x7f6f681a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6d400700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6819b080 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.160+0000 7f6f6ca00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f68104fb0 0x7f6f681a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f6d400700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 0x7f6f681a0760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00e9a0 tx=0x7f6f5c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c00cbd0 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f5c00cd70 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6819b370 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c018750 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.161+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f681a7630 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.162+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f68101300 con 0x7f6f68104600 2024-03-28T18:42:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.164+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6f5c010430 con 0x7f6f68104600 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f66000700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f500a29a0 0x7f6f500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f6f5c0ceea0 con 0x7f6f68104600 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f6ca00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f500a29a0 0x7f6f500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f66000700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] conn(0x7f6f500ac560 0x7f6f500aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f5c01ee50 con 0x7f6f500ac560 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.165+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f5c0d14c0 con 0x7f6f68104600 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.166+0000 7f6f6de00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] conn(0x7f6f500ac560 0x7f6f500aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.166+0000 7f6f6ca00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f500a29a0 0x7f6f500a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f6810aba0 tx=0x7f6f54009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.166+0000 7f6f6de00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] conn(0x7f6f500ac560 0x7f6f500aea00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.167+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== osd.10 v2:172.21.15.131:6824/2982886262 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6f5c01ee50 con 0x7f6f500ac560 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.185+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 --> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f68105ae0 con 0x7f6f500ac560 2024-03-28T18:42:13.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.185+0000 7f6f66000700 1 -- 172.21.15.119:0/4162328121 <== osd.10 v2:172.21.15.131:6824/2982886262 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6f68105ae0 con 0x7f6f500ac560 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] conn(0x7f6f500ac560 msgr2=0x7f6f500aea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] conn(0x7f6f500ac560 0x7f6f500aea00 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f500a29a0 msgr2=0x7f6f500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f500a29a0 0x7f6f500a4e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6f6810aba0 tx=0x7f6f54009310 comp rx=0 tx=0).stop 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 msgr2=0x7f6f681a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 --2- 172.21.15.119:0/4162328121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f68104600 0x7f6f681a0760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00e9a0 tx=0x7f6f5c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6de00700 1 -- 172.21.15.119:0/4162328121 reap_dead start 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 shutdown_connections 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 >> 172.21.15.119:0/4162328121 conn(0x7f6f680ffe90 msgr2=0x7f6f681008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.186+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 shutdown_connections 2024-03-28T18:42:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:13.187+0000 7f6f6fe00700 1 -- 172.21.15.119:0/4162328121 wait complete. 2024-03-28T18:42:14.021 INFO:teuthology.orchestra.run.smithi119.stdout:167503724601 2024-03-28T18:42:14.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T18:42:14.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:14 smithi119 ceph-mon[31813]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-28T18:42:14.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:14 smithi196 ceph-mon[36300]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-28T18:42:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:14 smithi131 ceph-mon[36065]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-03-28T18:42:15.193 INFO:teuthology.orchestra.run.smithi119.stdout:313532612637 2024-03-28T18:42:15.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T18:42:15.334 INFO:teuthology.orchestra.run.smithi119.stdout:201863462963 2024-03-28T18:42:15.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T18:42:15.851 INFO:teuthology.orchestra.run.smithi119.stdout:347892351001 2024-03-28T18:42:15.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T18:42:16.282 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:16.283 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:16 smithi119 ceph-mon[31813]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-28T18:42:16.367 INFO:teuthology.orchestra.run.smithi119.stdout:292057776163 2024-03-28T18:42:16.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T18:42:16.412 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:16.434 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:16 smithi196 ceph-mon[36300]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-28T18:42:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:16 smithi131 ceph-mon[36065]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-28T18:42:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:17.411 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:17.414 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:17.420 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:18.428 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:18 smithi119 ceph-mon[31813]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:18 smithi196 ceph-mon[36300]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:18.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:18 smithi131 ceph-mon[36065]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:20.210 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:20 smithi119 ceph-mon[31813]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:20.210 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:20 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:20 smithi196 ceph-mon[36300]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:20.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:20 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:20 smithi131 ceph-mon[36065]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:20 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:42:20.864 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:22.191 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:22 smithi119 ceph-mon[31813]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:22.191 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.192 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.192 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.192 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:22 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 -- 172.21.15.119:0/1785835151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b28069e40 msgr2=0x7f0b2806a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1785835151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b28069e40 0x7f0b2806a220 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009990 tx=0x7f0b18009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 -- 172.21.15.119:0/1785835151 shutdown_connections 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1785835151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2810ec40 0x7f0b28111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1785835151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b28069e40 0x7f0b2806a220 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1785835151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b2806a760 0x7f0b2810e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 -- 172.21.15.119:0/1785835151 >> 172.21.15.119:0/1785835151 conn(0x7f0b28077fc0 msgr2=0x7f0b280783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 -- 172.21.15.119:0/1785835151 shutdown_connections 2024-03-28T18:42:22.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.223+0000 7f0b2e200700 1 -- 172.21.15.119:0/1785835151 wait complete. 2024-03-28T18:42:22.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 Processor -- start 2024-03-28T18:42:22.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 -- start start 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2806a760 0x7f0b281a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 0x7f0b2819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281146a0 con 0x7f0b2806a760 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28114520 con 0x7f0b2810ec40 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.224+0000 7f0b2e200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28114820 con 0x7f0b28069e40 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58430/0 (socket says 172.21.15.119:58430) 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27400700 1 -- 172.21.15.119:0/1731472255 learned_addr learned my addr 172.21.15.119:0/1731472255 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 0x7f0b2819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:22.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b26a00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2806a760 0x7f0b281a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:22.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 -- 172.21.15.119:0/1731472255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 msgr2=0x7f0b281a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:22.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 -- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2806a760 msgr2=0x7f0b281a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2806a760 0x7f0b281a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 -- 172.21.15.119:0/1731472255 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2819b100 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27400700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.225+0000 7f0b27e00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 0x7f0b2819a9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000ef90 tx=0x7f0b2000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.226+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200092e0 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.226+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b20004bb0 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.226+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20004280 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.226+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2819b390 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.226+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b281a7660 con 0x7f0b2810ec40 2024-03-28T18:42:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.227+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28068e50 con 0x7f0b2810ec40 2024-03-28T18:42:22.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.229+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0b2001b030 con 0x7f0b2810ec40 2024-03-28T18:42:22.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.230+0000 7f0b1fe00700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 0x7f0b0c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:22.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.230+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f0b200ce770 con 0x7f0b2810ec40 2024-03-28T18:42:22.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.230+0000 7f0b27400700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 0x7f0b0c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:22.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.231+0000 7f0b27400700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 0x7f0b0c0a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009750 tx=0x7f0b1800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:22.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.234+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0b20022020 con 0x7f0b2810ec40 2024-03-28T18:42:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:22 smithi196 ceph-mon[36300]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:22 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:22 smithi131 ceph-mon[36065]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:22 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:22.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.474+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0b2802da30 con 0x7f0b2810ec40 2024-03-28T18:42:22.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.475+0000 7f0b1fe00700 1 -- 172.21.15.119:0/1731472255 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0b20010040 con 0x7f0b2810ec40 2024-03-28T18:42:22.475 INFO:teuthology.orchestra.run.smithi119.stdout:55834574930 2024-03-28T18:42:22.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.477+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 msgr2=0x7f0b0c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.477+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 0x7f0b0c0a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009750 tx=0x7f0b1800b280 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.477+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 msgr2=0x7f0b2819a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.477+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 0x7f0b2819a9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000ef90 tx=0x7f0b2000c3a0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 shutdown_connections 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0b0c0a29a0 0x7f0b0c0a4e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0b2810ec40 0x7f0b2819a9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b28069e40 0x7f0b281a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 --2- 172.21.15.119:0/1731472255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b2806a760 0x7f0b281a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 >> 172.21.15.119:0/1731472255 conn(0x7f0b28077fc0 msgr2=0x7f0b281100d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 shutdown_connections 2024-03-28T18:42:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:22.478+0000 7f0b2e200700 1 -- 172.21.15.119:0/1731472255 wait complete. 2024-03-28T18:42:23.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:23 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1731472255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:42:23.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:23 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1731472255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:42:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:23 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1731472255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T18:42:24.405 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:24 smithi119 ceph-mon[31813]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T18:42:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:24 smithi196 ceph-mon[36300]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T18:42:24.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:24 smithi131 ceph-mon[36065]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-28T18:42:24.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.163+0000 7fb41f600700 1 -- 172.21.15.119:0/2234491288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 msgr2=0x7fb418106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.163+0000 7fb41f600700 1 --2- 172.21.15.119:0/2234491288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb418106e20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb404009990 tx=0x7fb404009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 -- 172.21.15.119:0/2234491288 shutdown_connections 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 --2- 172.21.15.119:0/2234491288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418107550 0x7fb41810f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 --2- 172.21.15.119:0/2234491288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418100d30 0x7fb418101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 --2- 172.21.15.119:0/2234491288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb418106e20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 -- 172.21.15.119:0/2234491288 >> 172.21.15.119:0/2234491288 conn(0x7fb4180763d0 msgr2=0x7fb4180767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 -- 172.21.15.119:0/2234491288 shutdown_connections 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.164+0000 7fb41f600700 1 -- 172.21.15.119:0/2234491288 wait complete. 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 Processor -- start 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 -- start start 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418100d30 0x7fb4181a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181124f0 con 0x7fb418100d30 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418112370 con 0x7fb4181016e0 2024-03-28T18:42:24.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.165+0000 7fb41f600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418112670 con 0x7fb418107550 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb417e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41cc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418100d30 0x7fb4181a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58452/0 (socket says 172.21.15.119:58452) 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb417e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46324/0 (socket says 172.21.15.119:46324) 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 -- 172.21.15.119:0/595185728 learned_addr learned my addr 172.21.15.119:0/595185728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 -- 172.21.15.119:0/595185728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 msgr2=0x7fb4181a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 -- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418100d30 msgr2=0x7fb4181a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418100d30 0x7fb4181a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 -- 172.21.15.119:0/595185728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41819b170 con 0x7fb418107550 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb417e00700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:24.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb41d600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c370 tx=0x7fb41000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.166+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4100092e0 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.167+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb410003a00 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.167+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41819b400 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.167+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4100047c0 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.167+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4181a7720 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.168+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418068e50 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.170+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb41001a020 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.171+0000 7fb415600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 0x7fb4000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.171+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fb4100ce460 con 0x7fb418107550 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.171+0000 7fb41cc00700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 0x7fb4000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.172+0000 7fb41cc00700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 0x7fb4000a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb41819bb50 tx=0x7fb40c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.179+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb410010040 con 0x7fb418107550 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stdout:34359738455 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.404+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb41802da30 con 0x7fb418107550 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.404+0000 7fb415600700 1 -- 172.21.15.119:0/595185728 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb41009b330 con 0x7fb418107550 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 msgr2=0x7fb4000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 0x7fb4000a4e60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb41819bb50 tx=0x7fb40c009260 comp rx=0 tx=0).stop 2024-03-28T18:42:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 msgr2=0x7fb41819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c370 tx=0x7fb41000c680 comp rx=0 tx=0).stop 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 shutdown_connections 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb4000a29a0 0x7fb4000a4e60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb418107550 0x7fb41819aa10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb418100d30 0x7fb4181a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 --2- 172.21.15.119:0/595185728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb4181016e0 0x7fb4181a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.407+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 >> 172.21.15.119:0/595185728 conn(0x7fb4180763d0 msgr2=0x7fb4181086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.408+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 shutdown_connections 2024-03-28T18:42:24.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:24.408+0000 7fb41f600700 1 -- 172.21.15.119:0/595185728 wait complete. 2024-03-28T18:42:25.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:25 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/595185728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:42:25.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:25 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/595185728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:42:25.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:25 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/595185728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T18:42:26.287 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:26 smithi119 ceph-mon[31813]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-03-28T18:42:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:26 smithi196 ceph-mon[36300]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-03-28T18:42:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:26 smithi131 ceph-mon[36065]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-03-28T18:42:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:28 smithi196 ceph-mon[36300]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:42:28.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:28 smithi131 ceph-mon[36065]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:42:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:28 smithi119 ceph-mon[31813]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:42:29.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 -- 172.21.15.119:0/836874401 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 msgr2=0x7f9b78103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/836874401 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78103610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800b0a0 tx=0x7f9b6800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 -- 172.21.15.119:0/836874401 shutdown_connections 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/836874401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b7810c660 0x7f9b7810eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/836874401 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78103610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/836874401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78103b50 0x7f9b7810c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 -- 172.21.15.119:0/836874401 >> 172.21.15.119:0/836874401 conn(0x7f9b780fca60 msgr2=0x7f9b780fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 -- 172.21.15.119:0/836874401 shutdown_connections 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.544+0000 7f9b7ec00700 1 -- 172.21.15.119:0/836874401 wait complete. 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.545+0000 7f9b7ec00700 1 Processor -- start 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 -- start start 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b78103b50 0x7f9b781013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:29.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 0x7f9b78101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781123c0 con 0x7f9b7810c660 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78112240 con 0x7f9b78103230 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ec00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78112540 con 0x7f9b78103b50 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7d800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7d800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57516/0 (socket says 172.21.15.119:57516) 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7d800700 1 -- 172.21.15.119:0/1248785903 learned_addr learned my addr 172.21.15.119:0/1248785903 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7ce00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b78103b50 0x7f9b781013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.546+0000 7f9b7e200700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 0x7f9b78101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 -- 172.21.15.119:0/1248785903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b78103b50 msgr2=0x7f9b781013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b78103b50 0x7f9b781013d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 -- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 msgr2=0x7f9b78101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 0x7f9b78101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 -- 172.21.15.119:0/1248785903 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b781a6d50 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7e200700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 0x7f9b78101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7d800700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b680062a0 tx=0x7f9b68004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68021070 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b68007b80 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b680052a0 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781a6fe0 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.547+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781a7440 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.548+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78068e50 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.550+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9b68024070 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.552+0000 7f9b76000700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 0x7f9b5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.552+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f9b6800f020 con 0x7f9b78103230 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.552+0000 7f9b7ce00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 0x7f9b5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.553+0000 7f9b7ce00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 0x7f9b5c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009680 tx=0x7f9b64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:29.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.560+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b6802b030 con 0x7f9b78103230 2024-03-28T18:42:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.783+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9b7802dbf0 con 0x7f9b78103230 2024-03-28T18:42:29.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.784+0000 7f9b76000700 1 -- 172.21.15.119:0/1248785903 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9b6809cea0 con 0x7f9b78103230 2024-03-28T18:42:29.784 INFO:teuthology.orchestra.run.smithi119.stdout:201863462967 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 msgr2=0x7f9b5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 0x7f9b5c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009680 tx=0x7f9b64009260 comp rx=0 tx=0).stop 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 msgr2=0x7f9b78100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b680062a0 tx=0x7f9b68004ce0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 shutdown_connections 2024-03-28T18:42:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9b5c0a2920 0x7f9b5c0a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b7810c660 0x7f9b78101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9b78103230 0x7f9b78100e90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 --2- 172.21.15.119:0/1248785903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9b78103b50 0x7f9b781013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 >> 172.21.15.119:0/1248785903 conn(0x7f9b780fca60 msgr2=0x7f9b780fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 shutdown_connections 2024-03-28T18:42:29.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:29.787+0000 7f9b7ec00700 1 -- 172.21.15.119:0/1248785903 wait complete. 2024-03-28T18:42:30.086 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:30.093 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:30.118 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574930 for osd.1 2024-03-28T18:42:30.118 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:30.349 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:30.349 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T18:42:30.349 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-28T18:42:30.349 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-28T18:42:30.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T18:42:30.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1248785903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:42:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T18:42:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-28T18:42:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-28T18:42:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T18:42:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:30 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1248785903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:42:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-03-28T18:42:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-28T18:42:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-28T18:42:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-28T18:42:30.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:30 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1248785903' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T18:42:31.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T18:42:31.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-28T18:42:31.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-28T18:42:31.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:31 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T18:42:31.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:31 smithi119 ceph-mon[31813]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:42:31.408 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738455 for osd.0 2024-03-28T18:42:31.408 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T18:42:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-28T18:42:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-28T18:42:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:31 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T18:42:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:31 smithi196 ceph-mon[36300]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:42:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-03-28T18:42:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-28T18:42:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-28T18:42:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:31 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-28T18:42:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:31 smithi131 ceph-mon[36065]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T18:42:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 -- 172.21.15.119:0/3565690447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 msgr2=0x7f6ac41085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/3565690447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac41085a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009990 tx=0x7f6ab4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 -- 172.21.15.119:0/3565690447 shutdown_connections 2024-03-28T18:42:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/3565690447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/3565690447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac41085a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/3565690447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 -- 172.21.15.119:0/3565690447 >> 172.21.15.119:0/3565690447 conn(0x7f6ac40792c0 msgr2=0x7f6ac40796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 -- 172.21.15.119:0/3565690447 shutdown_connections 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.272+0000 7f6ac8e00700 1 -- 172.21.15.119:0/3565690447 wait complete. 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.273+0000 7f6ac8e00700 1 Processor -- start 2024-03-28T18:42:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 -- start start 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4112490 con 0x7f6ac41021e0 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4112310 con 0x7f6ac4102ba0 2024-03-28T18:42:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac8e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4112610 con 0x7f6ac41081c0 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac1e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac1400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac2800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac1e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40570/0 (socket says 172.21.15.119:40570) 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac2800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48696/0 (socket says 172.21.15.119:48696) 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac1400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57542/0 (socket says 172.21.15.119:57542) 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.274+0000 7f6ac1e00700 1 -- 172.21.15.119:0/2069104571 learned_addr learned my addr 172.21.15.119:0/2069104571 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:32.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 -- 172.21.15.119:0/2069104571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 msgr2=0x7f6ac4074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 -- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 msgr2=0x7f6ac4074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 -- 172.21.15.119:0/2069104571 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac41a6ea0 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac2800700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac1400700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e9f0 tx=0x7f6ab000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000cb70 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ab000ee90 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0003dc0 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.275+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac41a7190 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.276+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac41a7680 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.277+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4068e50 con 0x7f6ac4102ba0 2024-03-28T18:42:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.279+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6ab00438f0 con 0x7f6ac4102ba0 2024-03-28T18:42:32.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.280+0000 7f6abaa00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 0x7f6aa80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:32.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.280+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f6ab00ced60 con 0x7f6ac4102ba0 2024-03-28T18:42:32.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.280+0000 7f6ac1e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 0x7f6aa80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:32.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.281+0000 7f6ac1e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 0x7f6aa80a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009750 tx=0x7f6ab400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:32.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.282+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ab009bbc0 con 0x7f6ac4102ba0 2024-03-28T18:42:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:32 smithi196 ceph-mon[36300]: pgmap v350: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:32 smithi196 ceph-mon[36300]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-28T18:42:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:32 smithi196 ceph-mon[36300]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:42:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:32 smithi131 ceph-mon[36065]: pgmap v350: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:32 smithi131 ceph-mon[36065]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-28T18:42:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:32 smithi131 ceph-mon[36065]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:42:32.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.506+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f6ac402da30 con 0x7f6ac4102ba0 2024-03-28T18:42:32.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:32 smithi119 ceph-mon[31813]: pgmap v350: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:32.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:32 smithi119 ceph-mon[31813]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-28T18:42:32.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:32 smithi119 ceph-mon[31813]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T18:42:32.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.507+0000 7f6abaa00700 1 -- 172.21.15.119:0/2069104571 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6ab009b4c0 con 0x7f6ac4102ba0 2024-03-28T18:42:32.507 INFO:teuthology.orchestra.run.smithi119.stdout:257698037804 2024-03-28T18:42:32.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 msgr2=0x7f6aa80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:32.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 0x7f6aa80a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009750 tx=0x7f6ab400b280 comp rx=0 tx=0).stop 2024-03-28T18:42:32.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 msgr2=0x7f6ac40749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:32.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e9f0 tx=0x7f6ab000ed00 comp rx=0 tx=0).stop 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 shutdown_connections 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6aa80a29a0 0x7f6aa80a4e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ac41081c0 0x7f6ac4074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ac41021e0 0x7f6ac4074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 --2- 172.21.15.119:0/2069104571 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6ac4102ba0 0x7f6ac40749d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 >> 172.21.15.119:0/2069104571 conn(0x7f6ac40792c0 msgr2=0x7f6ac4100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.510+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 shutdown_connections 2024-03-28T18:42:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:32.511+0000 7f6ac8e00700 1 -- 172.21.15.119:0/2069104571 wait complete. 2024-03-28T18:42:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:33 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2069104571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:42:33.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:33 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2069104571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:42:33.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:33 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2069104571' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 -- 172.21.15.119:0/2771014395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd8102c80 msgr2=0x7fdbd8106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 --2- 172.21.15.119:0/2771014395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd8102c80 0x7fdbd8106cd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40099c0 tx=0x7fdbc4009cd0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 -- 172.21.15.119:0/2771014395 shutdown_connections 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 --2- 172.21.15.119:0/2771014395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81073b0 0x7fdbd810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 --2- 172.21.15.119:0/2771014395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd81022d0 0x7fdbd81026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 --2- 172.21.15.119:0/2771014395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd8102c80 0x7fdbd8106cd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 -- 172.21.15.119:0/2771014395 >> 172.21.15.119:0/2771014395 conn(0x7fdbd80fdb60 msgr2=0x7fdbd80fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 -- 172.21.15.119:0/2771014395 shutdown_connections 2024-03-28T18:42:33.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.457+0000 7fdbdea00700 1 -- 172.21.15.119:0/2771014395 wait complete. 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.458+0000 7fdbdea00700 1 Processor -- start 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 -- start start 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 0x7fdbd81a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd81073b0 0x7fdbd819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd8112370 con 0x7fdbd81022d0 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd81121f0 con 0x7fdbd8102c80 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdea00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd81124f0 con 0x7fdbd81073b0 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdcc00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdcc00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57560/0 (socket says 172.21.15.119:57560) 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbde000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd81073b0 0x7fdbd819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:33.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdcc00700 1 -- 172.21.15.119:0/873961931 learned_addr learned my addr 172.21.15.119:0/873961931 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.459+0000 7fdbdd600700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 0x7fdbd81a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 -- 172.21.15.119:0/873961931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd81073b0 msgr2=0x7fdbd819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd81073b0 0x7fdbd819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 -- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 msgr2=0x7fdbd81a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 0x7fdbd81a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 -- 172.21.15.119:0/873961931 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd819b000 con 0x7fdbd8102c80 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdd600700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 0x7fdbd81a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdcc00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b580 tx=0x7fdbc4003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4003680 con 0x7fdbd8102c80 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbc40055a0 con 0x7fdbd8102c80 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400fe30 con 0x7fdbd8102c80 2024-03-28T18:42:33.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.460+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd819b290 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.461+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd81a75b0 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.462+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd8068e50 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.464+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fdbc400f4f0 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.465+0000 7fdbd6000700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 0x7fdbc00a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.465+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fdbc40cfb60 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.465+0000 7fdbdd600700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 0x7fdbc00a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.466+0000 7fdbdd600700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 0x7fdbc00a5000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdbd819b9e0 tx=0x7fdbcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.467+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbc4013070 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.693+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdbd81a7780 con 0x7fdbd8102c80 2024-03-28T18:42:33.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.693+0000 7fdbd6000700 1 -- 172.21.15.119:0/873961931 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdbc409ca50 con 0x7fdbd8102c80 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stdout:111669149768 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 msgr2=0x7fdbc00a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 0x7fdbc00a5000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdbd819b9e0 tx=0x7fdbcc009260 comp rx=0 tx=0).stop 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 msgr2=0x7fdbd81a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400b580 tx=0x7fdbc4003a90 comp rx=0 tx=0).stop 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 shutdown_connections 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdbc00a2b40 0x7fdbc00a5000 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdbd81073b0 0x7fdbd819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdbd81022d0 0x7fdbd81a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 --2- 172.21.15.119:0/873961931 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdbd8102c80 0x7fdbd81a0c20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:33.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.696+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 >> 172.21.15.119:0/873961931 conn(0x7fdbd80fdb60 msgr2=0x7fdbd810d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:33.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.697+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 shutdown_connections 2024-03-28T18:42:33.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.697+0000 7fdbdea00700 1 -- 172.21.15.119:0/873961931 wait complete. 2024-03-28T18:42:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:34 smithi119 ceph-mon[31813]: pgmap v352: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-28T18:42:34.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/873961931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:42:34.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2135698552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:42:34.679 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462963 got 201863462967 for osd.6 2024-03-28T18:42:34.679 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:34 smithi196 ceph-mon[36300]: pgmap v352: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-28T18:42:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/873961931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:42:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2135698552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:42:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:34 smithi131 ceph-mon[36065]: pgmap v352: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-28T18:42:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/873961931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T18:42:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2135698552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T18:42:35.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 -- 172.21.15.119:0/693359001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104fb0 msgr2=0x7f0ca4109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:35.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 --2- 172.21.15.119:0/693359001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104fb0 0x7f0ca4109000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0099d0 tx=0x7f0c8c009ce0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 -- 172.21.15.119:0/693359001 shutdown_connections 2024-03-28T18:42:35.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 --2- 172.21.15.119:0/693359001 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 0x7f0ca41112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 --2- 172.21.15.119:0/693359001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104600 0x7f0ca41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 --2- 172.21.15.119:0/693359001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104fb0 0x7f0ca4109000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.212+0000 7f0caaa00700 1 -- 172.21.15.119:0/693359001 >> 172.21.15.119:0/693359001 conn(0x7f0ca40ffe70 msgr2=0x7f0ca4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.213+0000 7f0caaa00700 1 -- 172.21.15.119:0/693359001 shutdown_connections 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.213+0000 7f0caaa00700 1 -- 172.21.15.119:0/693359001 wait complete. 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 Processor -- start 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 -- start start 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104fb0 0x7f0ca41a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 0x7f0ca419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41146f0 con 0x7f0ca4104fb0 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4114570 con 0x7f0ca41096e0 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.214+0000 7f0caaa00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4114870 con 0x7f0ca4104600 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48762/0 (socket says 172.21.15.119:48762) 2024-03-28T18:42:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 -- 172.21.15.119:0/1222787400 learned_addr learned my addr 172.21.15.119:0/1222787400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0c9b400700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104fb0 0x7f0ca41a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca8a00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 0x7f0ca419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 -- 172.21.15.119:0/1222787400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 msgr2=0x7f0ca419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 0x7f0ca419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 -- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104fb0 msgr2=0x7f0ca41a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stdout:347892351006 2024-03-28T18:42:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104fb0 0x7f0ca41a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 -- 172.21.15.119:0/1222787400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca419b130 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.215+0000 7f0ca3e00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ef10 tx=0x7f0c9400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.216+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9400c9c0 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.216+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c94019070 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.216+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94007e60 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.216+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca419b3c0 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.216+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41a7690 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.217+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca4105ae0 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.219+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0c94012050 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.220+0000 7f0ca1600700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 0x7f0c880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.220+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0c940cebb0 con 0x7f0ca4104600 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.220+0000 7f0c9b400700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 0x7f0c880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.221+0000 7f0c9b400700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 0x7f0c880a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0099a0 tx=0x7f0c8c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:35.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.229+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0c94010040 con 0x7f0ca4104600 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.477+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f0ca402dbf0 con 0x7f0ca4104600 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.477+0000 7f0ca1600700 1 -- 172.21.15.119:0/1222787400 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f0c9409ba10 con 0x7f0ca4104600 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 msgr2=0x7f0c880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 0x7f0c880a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0099a0 tx=0x7f0c8c01a040 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 msgr2=0x7f0ca41a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ef10 tx=0x7f0c9400c3a0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 shutdown_connections 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c880a29a0 0x7f0c880a4e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0ca41096e0 0x7f0ca419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ca4104600 0x7f0ca41a07c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 --2- 172.21.15.119:0/1222787400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca4104fb0 0x7f0ca41a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 >> 172.21.15.119:0/1222787400 conn(0x7f0ca40ffe70 msgr2=0x7f0ca41009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 shutdown_connections 2024-03-28T18:42:35.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:35.481+0000 7f0caaa00700 1 -- 172.21.15.119:0/1222787400 wait complete. 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 -- 172.21.15.119:0/1882575121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1340743c0 msgr2=0x7fc1340747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 --2- 172.21.15.119:0/1882575121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1340743c0 0x7fc1340747a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc1280099c0 tx=0x7fc128009cd0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 -- 172.21.15.119:0/1882575121 shutdown_connections 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 --2- 172.21.15.119:0/1882575121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc134113360 0x7fc134115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 --2- 172.21.15.119:0/1882575121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1340743c0 0x7fc1340747a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 --2- 172.21.15.119:0/1882575121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134074ce0 0x7fc134112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 -- 172.21.15.119:0/1882575121 >> 172.21.15.119:0/1882575121 conn(0x7fc1340792f0 msgr2=0x7fc134079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:36.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 -- 172.21.15.119:0/1882575121 shutdown_connections 2024-03-28T18:42:36.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.060+0000 7fc13bc00700 1 -- 172.21.15.119:0/1882575121 wait complete. 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.061+0000 7fc13bc00700 1 Processor -- start 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.061+0000 7fc13bc00700 1 -- start start 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.061+0000 7fc13bc00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc13bc00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 0x7fc13419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc13bc00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 0x7fc1341a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc13bc00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134118bb0 con 0x7fc134074ce0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc13bc00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134118a30 con 0x7fc134113360 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc13bc00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134118d30 con 0x7fc1340743c0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48772/0 (socket says 172.21.15.119:48772) 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 -- 172.21.15.119:0/4246238487 learned_addr learned my addr 172.21.15.119:0/4246238487 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139c00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 0x7fc1341a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc133e00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 0x7fc13419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 -- 172.21.15.119:0/4246238487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 msgr2=0x7fc1341a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 0x7fc1341a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 -- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 msgr2=0x7fc13419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 0x7fc13419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139200700 1 -- 172.21.15.119:0/4246238487 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1341a3fc0 con 0x7fc1340743c0 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc133e00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 0x7fc13419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.062+0000 7fc139c00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 0x7fc1341a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc139200700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc128003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280050c0 con 0x7fc1340743c0 2024-03-28T18:42:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc12800f460 con 0x7fc1340743c0 2024-03-28T18:42:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128017650 con 0x7fc1340743c0 2024-03-28T18:42:36.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1341a4250 con 0x7fc1340743c0 2024-03-28T18:42:36.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.063+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1341abb30 con 0x7fc1340743c0 2024-03-28T18:42:36.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.064+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc134075af0 con 0x7fc1340743c0 2024-03-28T18:42:36.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.066+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc12800f610 con 0x7fc1340743c0 2024-03-28T18:42:36.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.067+0000 7fc131600700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 0x7fc11c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:36.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.068+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fc1280cfa50 con 0x7fc1340743c0 2024-03-28T18:42:36.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.068+0000 7fc133e00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 0x7fc11c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.068+0000 7fc133e00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 0x7fc11c0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1341151d0 tx=0x7fc120009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.071+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc128013070 con 0x7fc1340743c0 2024-03-28T18:42:36.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.305+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fc1341a0310 con 0x7fc1340743c0 2024-03-28T18:42:36.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.305+0000 7fc131600700 1 -- 172.21.15.119:0/4246238487 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc128020370 con 0x7fc1340743c0 2024-03-28T18:42:36.305 INFO:teuthology.orchestra.run.smithi119.stdout:313532612641 2024-03-28T18:42:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 msgr2=0x7fc11c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 0x7fc11c0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1341151d0 tx=0x7fc120009260 comp rx=0 tx=0).stop 2024-03-28T18:42:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 msgr2=0x7fc13419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc128006010 tx=0x7fc128003a90 comp rx=0 tx=0).stop 2024-03-28T18:42:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 shutdown_connections 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc11c0a2920 0x7fc11c0a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc134113360 0x7fc1341a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1340743c0 0x7fc13419f000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 --2- 172.21.15.119:0/4246238487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc134074ce0 0x7fc13419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 >> 172.21.15.119:0/4246238487 conn(0x7fc1340792f0 msgr2=0x7fc134112120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.308+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 shutdown_connections 2024-03-28T18:42:36.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:36.309+0000 7fc13bc00700 1 -- 172.21.15.119:0/4246238487 wait complete. 2024-03-28T18:42:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:36 smithi119 ceph-mon[31813]: pgmap v353: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:42:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1222787400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:42:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:36 smithi196 ceph-mon[36300]: pgmap v353: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:42:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:36 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1222787400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:42:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:36 smithi131 ceph-mon[36065]: pgmap v353: 97 pgs: 2 peering, 95 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-28T18:42:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:36 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1222787400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T18:42:37.492 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:37 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4246238487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:42:37.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:37 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4246238487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:42:37.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:37 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4246238487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T18:42:38.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 -- 172.21.15.119:0/3876335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8074ce0 msgr2=0x7f9bd8112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 --2- 172.21.15.119:0/3876335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8074ce0 0x7f9bd8112e20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009990 tx=0x7f9bc4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 -- 172.21.15.119:0/3876335 shutdown_connections 2024-03-28T18:42:38.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 --2- 172.21.15.119:0/3876335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8113360 0x7f9bd8115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 --2- 172.21.15.119:0/3876335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd80747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 --2- 172.21.15.119:0/3876335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8074ce0 0x7f9bd8112e20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 -- 172.21.15.119:0/3876335 >> 172.21.15.119:0/3876335 conn(0x7f9bd80792f0 msgr2=0x7f9bd8079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 -- 172.21.15.119:0/3876335 shutdown_connections 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.153+0000 7f9be0000700 1 -- 172.21.15.119:0/3876335 wait complete. 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.154+0000 7f9be0000700 1 Processor -- start 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.154+0000 7f9be0000700 1 -- start start 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8074ce0 0x7f9bd819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 0x7f9bd81a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8118ba0 con 0x7f9bd8074ce0 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8118a20 con 0x7f9bd8113360 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9be0000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8118d20 con 0x7f9bd80743c0 2024-03-28T18:42:38.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48780/0 (socket says 172.21.15.119:48780) 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 -- 172.21.15.119:0/3293685224 learned_addr learned my addr 172.21.15.119:0/3293685224 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bde000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 0x7f9bd81a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdcc00700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8074ce0 0x7f9bd819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 -- 172.21.15.119:0/3293685224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 msgr2=0x7f9bd81a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 0x7f9bd81a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 -- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8074ce0 msgr2=0x7f9bd819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.155+0000 7f9bdd600700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8074ce0 0x7f9bd819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bdd600700 1 -- 172.21.15.119:0/3293685224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd81a4020 con 0x7f9bd80743c0 2024-03-28T18:42:38.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bde000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 0x7f9bd81a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bdd600700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00eb30 tx=0x7f9bcc00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc00cd40 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd81a4310 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bcc004500 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc010690 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.156+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81abb90 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.157+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8068e50 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.161+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9bcc003e10 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.163+0000 7f9bd6000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 0x7f9bc00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.163+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f9bcc0cefe0 con 0x7f9bd80743c0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.163+0000 7f9bdcc00700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 0x7f9bc00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.164+0000 7f9bdcc00700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 0x7f9bc00a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400b2e0 tx=0x7f9bc401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:38.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.166+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9bcc09bea0 con 0x7f9bd80743c0 2024-03-28T18:42:38.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:38 smithi119 ceph-mon[31813]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:42:38.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:38 smithi119 ceph-mon[31813]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-28T18:42:38.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:38 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:42:38.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.419+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9bd81a0370 con 0x7f9bd80743c0 2024-03-28T18:42:38.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.420+0000 7f9bd6000700 1 -- 172.21.15.119:0/3293685224 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9bcc09b7a0 con 0x7f9bd80743c0 2024-03-28T18:42:38.421 INFO:teuthology.orchestra.run.smithi119.stdout:223338299442 2024-03-28T18:42:38.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 msgr2=0x7f9bc00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 0x7f9bc00a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400b2e0 tx=0x7f9bc401a040 comp rx=0 tx=0).stop 2024-03-28T18:42:38.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 msgr2=0x7f9bd819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00eb30 tx=0x7f9bcc00eef0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 shutdown_connections 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9bc00a2920 0x7f9bc00a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9bd8113360 0x7f9bd81a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9bd80743c0 0x7f9bd819f010 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 --2- 172.21.15.119:0/3293685224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd8074ce0 0x7f9bd819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 >> 172.21.15.119:0/3293685224 conn(0x7f9bd80792f0 msgr2=0x7f9bd8112120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 shutdown_connections 2024-03-28T18:42:38.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:38.423+0000 7f9be0000700 1 -- 172.21.15.119:0/3293685224 wait complete. 2024-03-28T18:42:38.585 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:38 smithi196 ceph-mon[36300]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:42:38.585 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:38 smithi196 ceph-mon[36300]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-28T18:42:38.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:38 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:42:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:38 smithi131 ceph-mon[36065]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-28T18:42:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:38 smithi131 ceph-mon[36065]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-28T18:42:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:38 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:42:38.696 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351001 got 347892351006 for osd.11 2024-03-28T18:42:38.696 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:38.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 -- 172.21.15.119:0/3944731684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1096c0 msgr2=0x7f559c111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 --2- 172.21.15.119:0/3944731684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1096c0 0x7f559c111290 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f559400b0a0 tx=0x7f559400b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 -- 172.21.15.119:0/3944731684 shutdown_connections 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 --2- 172.21.15.119:0/3944731684 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1096c0 0x7f559c111290 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 --2- 172.21.15.119:0/3944731684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1045e0 0x7f559c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 --2- 172.21.15.119:0/3944731684 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 -- 172.21.15.119:0/3944731684 >> 172.21.15.119:0/3944731684 conn(0x7f559c0ffe90 msgr2=0x7f559c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 -- 172.21.15.119:0/3944731684 shutdown_connections 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.782+0000 7f55a2c00700 1 -- 172.21.15.119:0/3944731684 wait complete. 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 Processor -- start 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 -- start start 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1096c0 0x7f559c19a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c1146b0 con 0x7f559c1096c0 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c114530 con 0x7f559c1045e0 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.784+0000 7f55a2c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c114830 con 0x7f559c104f90 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57580/0 (socket says 172.21.15.119:57580) 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 -- 172.21.15.119:0/2135698552 learned_addr learned my addr 172.21.15.119:0/2135698552 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559b400700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f55a0c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1096c0 0x7f559c19a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 -- 172.21.15.119:0/2135698552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 msgr2=0x7f559c1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 -- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1096c0 msgr2=0x7f559c19a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1096c0 0x7f559c19a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 -- 172.21.15.119:0/2135698552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c19b130 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559b400700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f559be00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e9a0 tx=0x7f558c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.785+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c00ca10 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.786+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f558c00cbb0 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.786+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c19b420 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.786+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c018680 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.786+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c1a7690 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.787+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c068e50 con 0x7f559c1045e0 2024-03-28T18:42:38.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.789+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f558c010430 con 0x7f559c1045e0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.790+0000 7f5598c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 0x7f55800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.790+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f558c0cef10 con 0x7f559c1045e0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.790+0000 7f559b400700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 0x7f55800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.791+0000 7f559b400700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 0x7f55800a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f559c103630 tx=0x7f5588009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:33.798+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f558c09bd70 con 0x7f559c1045e0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.022+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f559c02da30 con 0x7f559c1045e0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.023+0000 7f5598c00700 1 -- 172.21.15.119:0/2135698552 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f558c09b670 con 0x7f559c1045e0 2024-03-28T18:42:38.765 INFO:teuthology.orchestra.run.smithi119.stdout:167503724606 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.025+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 msgr2=0x7f55800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 0x7f55800a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f559c103630 tx=0x7f5588009260 comp rx=0 tx=0).stop 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 msgr2=0x7f559c1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e9a0 tx=0x7f558c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 shutdown_connections 2024-03-28T18:42:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55800a2920 0x7f55800a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f559c1096c0 0x7f559c19a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f559c1045e0 0x7f559c1a07c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 --2- 172.21.15.119:0/2135698552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f559c104f90 0x7f559c1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 >> 172.21.15.119:0/2135698552 conn(0x7f559c0ffe90 msgr2=0x7f559c100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 shutdown_connections 2024-03-28T18:42:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:34.026+0000 7f55a2c00700 1 -- 172.21.15.119:0/2135698552 wait complete. 2024-03-28T18:42:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:39 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3293685224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:42:39.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:39 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3293685224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:42:39.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:39 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3293685224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T18:42:40.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:40 smithi196 ceph-mon[36300]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:42:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:40 smithi119 ceph-mon[31813]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:42:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:40 smithi131 ceph-mon[36065]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-28T18:42:40.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149764 got 111669149768 for osd.3 2024-03-28T18:42:40.885 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:41.566 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037798 got 257698037804 for osd.8 2024-03-28T18:42:41.567 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:41.674 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724601 got 167503724606 for osd.5 2024-03-28T18:42:41.674 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 -- 172.21.15.119:0/2118781137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da8069e40 msgr2=0x7f5da806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 --2- 172.21.15.119:0/2118781137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da8069e40 0x7f5da806a2a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009990 tx=0x7f5d94009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 -- 172.21.15.119:0/2118781137 shutdown_connections 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 --2- 172.21.15.119:0/2118781137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 --2- 172.21.15.119:0/2118781137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da81081e0 0x7f5da81085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 --2- 172.21.15.119:0/2118781137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da8069e40 0x7f5da806a2a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 -- 172.21.15.119:0/2118781137 >> 172.21.15.119:0/2118781137 conn(0x7f5da8077fc0 msgr2=0x7f5da80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 -- 172.21.15.119:0/2118781137 shutdown_connections 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.553+0000 7f5dad000700 1 -- 172.21.15.119:0/2118781137 wait complete. 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.554+0000 7f5dad000700 1 Processor -- start 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 -- start start 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da81081e0 0x7f5da819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8112490 con 0x7f5da806a7e0 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8112310 con 0x7f5da81081e0 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5dad000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da8112610 con 0x7f5da8069e40 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da5e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da5400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da5400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59820/0 (socket says 172.21.15.119:59820) 2024-03-28T18:42:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da5e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:51916/0 (socket says 172.21.15.119:51916) 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da5400700 1 -- 172.21.15.119:0/370297475 learned_addr learned my addr 172.21.15.119:0/370297475 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.555+0000 7f5da6800700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da81081e0 0x7f5da819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 -- 172.21.15.119:0/370297475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da81081e0 msgr2=0x7f5da819a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da81081e0 0x7f5da819a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 -- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 msgr2=0x7f5da81a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 -- 172.21.15.119:0/370297475 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da819b150 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5400700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5da5e00700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800e9f0 tx=0x7f5d9800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98004220 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d98004d10 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da819b440 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.556+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98005020 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.557+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da81a7700 con 0x7f5da8069e40 2024-03-28T18:42:41.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.558+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da8068e50 con 0x7f5da8069e40 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.559+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5d98004750 con 0x7f5da8069e40 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.561+0000 7f5d9ea00700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 0x7f5d8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.561+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f5d980ceb20 con 0x7f5da8069e40 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.561+0000 7f5da5400700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 0x7f5d8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.562+0000 7f5da5400700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 0x7f5d8c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5da81031e0 tx=0x7f5d9401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.564+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d9809b980 con 0x7f5da8069e40 2024-03-28T18:42:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.791+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5da819bf20 con 0x7f5da8069e40 2024-03-28T18:42:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.792+0000 7f5d9ea00700 1 -- 172.21.15.119:0/370297475 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5d9809b280 con 0x7f5da8069e40 2024-03-28T18:42:41.793 INFO:teuthology.orchestra.run.smithi119.stdout:77309411408 2024-03-28T18:42:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 msgr2=0x7f5d8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 0x7f5d8c0a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5da81031e0 tx=0x7f5d9401a040 comp rx=0 tx=0).stop 2024-03-28T18:42:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 msgr2=0x7f5da81a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800e9f0 tx=0x7f5d9800ed00 comp rx=0 tx=0).stop 2024-03-28T18:42:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 shutdown_connections 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5d8c0a2920 0x7f5d8c0a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5da81081e0 0x7f5da819a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5da8069e40 0x7f5da81a0830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 --2- 172.21.15.119:0/370297475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da806a7e0 0x7f5da81a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 >> 172.21.15.119:0/370297475 conn(0x7f5da8077fc0 msgr2=0x7f5da810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.795+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 shutdown_connections 2024-03-28T18:42:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:41.796+0000 7f5dad000700 1 -- 172.21.15.119:0/370297475 wait complete. 2024-03-28T18:42:42.557 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:42 smithi119 ceph-mon[31813]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:42.557 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:42 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/370297475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:42:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:42 smithi196 ceph-mon[36300]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:42 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/370297475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:42:42.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:42 smithi131 ceph-mon[36065]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:42:42.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:42 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/370297475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T18:42:43.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612637 got 313532612641 for osd.10 2024-03-28T18:42:43.106 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 -- 172.21.15.119:0/4226461043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 msgr2=0x7f755c106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 --2- 172.21.15.119:0/4226461043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c106e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7548009990 tx=0x7f7548009ca0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 -- 172.21.15.119:0/4226461043 shutdown_connections 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 --2- 172.21.15.119:0/4226461043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c107510 0x7f755c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 --2- 172.21.15.119:0/4226461043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c102440 0x7f755c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 --2- 172.21.15.119:0/4226461043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c106e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 -- 172.21.15.119:0/4226461043 >> 172.21.15.119:0/4226461043 conn(0x7f755c0792f0 msgr2=0x7f755c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:43.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 -- 172.21.15.119:0/4226461043 shutdown_connections 2024-03-28T18:42:43.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.329+0000 7f7562400700 1 -- 172.21.15.119:0/4226461043 wait complete. 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.330+0000 7f7562400700 1 Processor -- start 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.330+0000 7f7562400700 1 -- start start 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.330+0000 7f7562400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f7562400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f7562400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c107510 0x7f755c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f7562400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c112520 con 0x7f755c107510 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f7562400700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1123a0 con 0x7f755c102df0 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f7562400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1126a0 con 0x7f755c102440 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755aa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:51940/0 (socket says 172.21.15.119:51940) 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 -- 172.21.15.119:0/2467193187 learned_addr learned my addr 172.21.15.119:0/2467193187 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755aa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33538/0 (socket says 172.21.15.119:33538) 2024-03-28T18:42:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755be00700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c107510 0x7f755c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 -- 172.21.15.119:0/2467193187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 msgr2=0x7f755c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 -- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c107510 msgr2=0x7f755c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c107510 0x7f755c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 -- 172.21.15.119:0/2467193187 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c199040 con 0x7f755c102440 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.331+0000 7f755b400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f754c00e9a0 tx=0x7f754c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.332+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c00ca10 con 0x7f755c102440 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.332+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f754c00cbb0 con 0x7f755c102440 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.332+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c018680 con 0x7f755c102440 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.332+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c199330 con 0x7f755c102440 2024-03-28T18:42:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.332+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c1a55a0 con 0x7f755c102440 2024-03-28T18:42:43.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.333+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c068e50 con 0x7f755c102440 2024-03-28T18:42:43.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.335+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f754c010430 con 0x7f755c102440 2024-03-28T18:42:43.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.336+0000 7f7553e00700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 0x7f75400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:43.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.336+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f754c0cee80 con 0x7f755c102440 2024-03-28T18:42:43.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.336+0000 7f755aa00700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 0x7f75400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:43.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.337+0000 7f755aa00700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 0x7f75400a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f755c101d40 tx=0x7f754800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.341+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f754c09bce0 con 0x7f755c102440 2024-03-28T18:42:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.582+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f755c02da30 con 0x7f755c102440 2024-03-28T18:42:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.582+0000 7f7553e00700 1 -- 172.21.15.119:0/2467193187 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f754c09b5e0 con 0x7f755c102440 2024-03-28T18:42:43.584 INFO:teuthology.orchestra.run.smithi119.stdout:146028888134 2024-03-28T18:42:43.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 msgr2=0x7f75400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:43.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 0x7f75400a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f755c101d40 tx=0x7f754800b260 comp rx=0 tx=0).stop 2024-03-28T18:42:43.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 msgr2=0x7f755c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:43.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f754c00e9a0 tx=0x7f754c00ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 shutdown_connections 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f75400a29a0 0x7f75400a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755c107510 0x7f755c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f755c102440 0x7f755c19e6d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 --2- 172.21.15.119:0/2467193187 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f755c102df0 0x7f755c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 >> 172.21.15.119:0/2467193187 conn(0x7f755c0792f0 msgr2=0x7f755c1002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.585+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 shutdown_connections 2024-03-28T18:42:43.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.586+0000 7f7562400700 1 -- 172.21.15.119:0/2467193187 wait complete. 2024-03-28T18:42:44.146 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299436 got 223338299442 for osd.7 2024-03-28T18:42:44.146 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:44.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 -- 172.21.15.119:0/3797646686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 msgr2=0x7f9410111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 --2- 172.21.15.119:0/3797646686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f9410111030 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f940800b0a0 tx=0x7f940800b3b0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 -- 172.21.15.119:0/3797646686 shutdown_connections 2024-03-28T18:42:44.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 --2- 172.21.15.119:0/3797646686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f9410111030 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 --2- 172.21.15.119:0/3797646686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9410069e40 0x7f941006a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 --2- 172.21.15.119:0/3797646686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f941006a760 0x7f941010e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 -- 172.21.15.119:0/3797646686 >> 172.21.15.119:0/3797646686 conn(0x7f9410077fc0 msgr2=0x7f94100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 -- 172.21.15.119:0/3797646686 shutdown_connections 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.943+0000 7f9417200700 1 -- 172.21.15.119:0/3797646686 wait complete. 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.944+0000 7f9417200700 1 Processor -- start 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.944+0000 7f9417200700 1 -- start start 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 0x7f94101a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f941019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9410114690 con 0x7f941006a760 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9410114510 con 0x7f941010ec40 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9417200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9410114810 con 0x7f9410069e40 2024-03-28T18:42:44.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:51970/0 (socket says 172.21.15.119:51970) 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 -- 172.21.15.119:0/3202143406 learned_addr learned my addr 172.21.15.119:0/3202143406 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940f400700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 0x7f94101a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9415200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f941019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 -- 172.21.15.119:0/3202143406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 msgr2=0x7f941019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f941019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 -- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 msgr2=0x7f94101a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 0x7f94101a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940fe00700 1 -- 172.21.15.119:0/3202143406 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941019b0b0 con 0x7f9410069e40 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f9415200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f941019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.945+0000 7f940f400700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 0x7f94101a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f940fe00700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f940000e9a0 tx=0x7f940000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000ca10 con 0x7f9410069e40 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941019b3a0 con 0x7f9410069e40 2024-03-28T18:42:44.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f940000cbb0 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400018680 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.946+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94101a7610 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.947+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9410068e50 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.949+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9400010430 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.951+0000 7f940cc00700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 0x7f93f40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.951+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f94000cf3a0 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.951+0000 7f940f400700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 0x7f93f40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.952+0000 7f940f400700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 0x7f93f40a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9410102540 tx=0x7f93fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:43.953+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f940009c200 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.177+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f941002da30 con 0x7f9410069e40 2024-03-28T18:42:44.241 INFO:teuthology.orchestra.run.smithi119.stdout:292057776169 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.178+0000 7f940cc00700 1 -- 172.21.15.119:0/3202143406 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f940009bb00 con 0x7f9410069e40 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 msgr2=0x7f93f40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 0x7f93f40a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9410102540 tx=0x7f93fc009260 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 msgr2=0x7f94101a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f940000e9a0 tx=0x7f940000ecb0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 shutdown_connections 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f93f40a2920 0x7f93f40a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f941010ec40 0x7f941019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9410069e40 0x7f94101a0740 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 --2- 172.21.15.119:0/3202143406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f941006a760 0x7f94101a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:44.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 >> 172.21.15.119:0/3202143406 conn(0x7f9410077fc0 msgr2=0x7f9410110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:44.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 shutdown_connections 2024-03-28T18:42:44.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:44.181+0000 7f9417200700 1 -- 172.21.15.119:0/3202143406 wait complete. 2024-03-28T18:42:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:44 smithi119 ceph-mon[31813]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T18:42:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:44 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2467193187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:42:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:44 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3202143406' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:42:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:44 smithi196 ceph-mon[36300]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T18:42:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:44 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2467193187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:42:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:44 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3202143406' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:42:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:44 smithi131 ceph-mon[36065]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-03-28T18:42:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:44 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2467193187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T18:42:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:44 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3202143406' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T18:42:45.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411408 for osd.2 2024-03-28T18:42:45.404 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:45.848 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776169 for osd.9 2024-03-28T18:42:45.848 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:46.214 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888134 for osd.4 2024-03-28T18:42:46.214 DEBUG:teuthology.parallel:result is None 2024-03-28T18:42:46.214 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T18:42:46.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:42:46.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:46 smithi119 ceph-mon[31813]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:46.660 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:46 smithi196 ceph-mon[36300]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:46 smithi131 ceph-mon[36065]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:47.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:47 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:47 smithi119 ceph-mon[31813]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:47.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:47 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:47 smithi196 ceph-mon[36300]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:47.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:47 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:47.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:47 smithi131 ceph-mon[36065]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:48.690 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:48.690 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:48.690 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:48 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 -- 172.21.15.119:0/2445296248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101a30 msgr2=0x7f091c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 --2- 172.21.15.119:0/2445296248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101a30 0x7f091c10df10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0914009f90 tx=0x7f091400b330 comp rx=0 tx=0).stop 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 -- 172.21.15.119:0/2445296248 shutdown_connections 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 --2- 172.21.15.119:0/2445296248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101a30 0x7f091c10df10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 --2- 172.21.15.119:0/2445296248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c1070d0 0x7f091c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:48.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 --2- 172.21.15.119:0/2445296248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c101090 0x7f091c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:48.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 -- 172.21.15.119:0/2445296248 >> 172.21.15.119:0/2445296248 conn(0x7f091c077fc0 msgr2=0x7f091c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:48.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 -- 172.21.15.119:0/2445296248 shutdown_connections 2024-03-28T18:42:48.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.857+0000 7f0921e00700 1 -- 172.21.15.119:0/2445296248 wait complete. 2024-03-28T18:42:48.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.858+0000 7f0921e00700 1 Processor -- start 2024-03-28T18:42:48.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 -- start start 2024-03-28T18:42:48.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 0x7f091c1a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:48.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 0x7f091c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:48.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 0x7f091c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c111320 con 0x7f091c101090 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c1111a0 con 0x7f091c101a30 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.859+0000 7f0921e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091c1114a0 con 0x7f091c1070d0 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091aa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 0x7f091c1a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091a000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 0x7f091c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091aa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 0x7f091c1a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59858/0 (socket says 172.21.15.119:59858) 2024-03-28T18:42:48.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091a000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 0x7f091c1a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33568/0 (socket says 172.21.15.119:33568) 2024-03-28T18:42:48.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091aa00700 1 -- 172.21.15.119:0/92575064 learned_addr learned my addr 172.21.15.119:0/92575064 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:48.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 0x7f091c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:48.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 -- 172.21.15.119:0/92575064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 msgr2=0x7f091c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 0x7f091c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 -- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 msgr2=0x7f091c1a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 0x7f091c1a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 -- 172.21.15.119:0/92575064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091c19b030 con 0x7f091c1070d0 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.860+0000 7f091b400700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 0x7f091c19a8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0914015000 tx=0x7f0914004f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.861+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914021070 con 0x7f091c1070d0 2024-03-28T18:42:48.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.861+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0914006020 con 0x7f091c1070d0 2024-03-28T18:42:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.861+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091c19b2c0 con 0x7f091c1070d0 2024-03-28T18:42:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.861+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914005040 con 0x7f091c1070d0 2024-03-28T18:42:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.861+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c1a75e0 con 0x7f091c1070d0 2024-03-28T18:42:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.862+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091c102780 con 0x7f091c1070d0 2024-03-28T18:42:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.864+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0914024070 con 0x7f091c1070d0 2024-03-28T18:42:48.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.865+0000 7f0920a00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 0x7f09000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:48.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.865+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f09140cfb10 con 0x7f091c1070d0 2024-03-28T18:42:48.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.865+0000 7f091aa00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 0x7f09000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:48.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.866+0000 7f091aa00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 0x7f09000a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f091c19ba10 tx=0x7f090c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:48.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:48.870+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f091409ca00 con 0x7f091c1070d0 2024-03-28T18:42:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:48 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:42:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:42:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:48 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:42:49.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.087+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f091c06a570 con 0x7f09000a2920 2024-03-28T18:42:49.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.097+0000 7f0920a00700 1 -- 172.21.15.119:0/92575064 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259008 (secure 0 0 0) 0x7f091c06a570 con 0x7f09000a2920 2024-03-28T18:42:49.099 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 msgr2=0x7f09000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 0x7f09000a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f091c19ba10 tx=0x7f090c009260 comp rx=0 tx=0).stop 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 msgr2=0x7f091c19a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 0x7f091c19a8d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0914015000 tx=0x7f0914004f00 comp rx=0 tx=0).stop 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 shutdown_connections 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09000a2920 0x7f09000a4de0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f091c1070d0 0x7f091c19a8d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f091c101090 0x7f091c1a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:49.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 --2- 172.21.15.119:0/92575064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f091c101a30 0x7f091c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:49.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 >> 172.21.15.119:0/92575064 conn(0x7f091c077fc0 msgr2=0x7f091c069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:49.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 shutdown_connections 2024-03-28T18:42:49.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:49.104+0000 7f0921e00700 1 -- 172.21.15.119:0/92575064 wait complete. 2024-03-28T18:42:49.108 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:42:49.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:49 smithi119 ceph-mon[31813]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:49.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:49 smithi119 ceph-mon[31813]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:49.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:49 smithi196 ceph-mon[36300]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:49.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:49 smithi196 ceph-mon[36300]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:49.943 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-28T18:42:48.997992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659507,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":960,"num_read_kb":604,"num_write":423,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361324,"kb_used_data":12096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580500,"statfs":{"total":1151940427776,"available":1151570432000,"internally_reserved":0,"allocated":12386304,"data_stored":7691061,"data_compressed":246444,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005304"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411432+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:42:31.411432+0000","last_peered":"2024-03-28T18:42:31.411432+0000","last_clean":"2024-03-28T18:42:31.411432+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:42:31.411432+0000","last_undegraded":"2024-03-28T18:42:31.411432+0000","last_fullsized":"2024-03-28T18:42:31.411432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.182960+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:42:31.182960+0000","last_peered":"2024-03-28T18:42:31.182960+0000","last_clean":"2024-03-28T18:42:31.182960+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:42:31.182960+0000","last_undegraded":"2024-03-28T18:42:31.182960+0000","last_fullsized":"2024-03-28T18:42:31.182960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.182143+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:42:31.182143+0000","last_peered":"2024-03-28T18:42:31.182143+0000","last_clean":"2024-03-28T18:42:31.182143+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:42:31.182143+0000","last_undegraded":"2024-03-28T18:42:31.182143+0000","last_fullsized":"2024-03-28T18:42:31.182143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299891+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:42:31.299891+0000","last_peered":"2024-03-28T18:42:31.299891+0000","last_clean":"2024-03-28T18:42:31.299891+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:42:31.299891+0000","last_undegraded":"2024-03-28T18:42:31.299891+0000","last_fullsized":"2024-03-28T18:42:31.299891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239016+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:42:31.239016+0000","last_peered":"2024-03-28T18:42:31.239016+0000","last_clean":"2024-03-28T18:42:31.239016+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:42:31.239016+0000","last_undegraded":"2024-03-28T18:42:31.239016+0000","last_fullsized":"2024-03-28T18:42:31.239016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269632+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:42:31.269632+0000","last_peered":"2024-03-28T18:42:31.269632+0000","last_clean":"2024-03-28T18:42:31.269632+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:42:31.269632+0000","last_undegraded":"2024-03-28T18:42:31.269632+0000","last_fullsized":"2024-03-28T18:42:31.269632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360236+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:42:31.360236+0000","last_peered":"2024-03-28T18:42:31.360236+0000","last_clean":"2024-03-28T18:42:31.360236+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:42:31.360236+0000","last_undegraded":"2024-03-28T18:42:31.360236+0000","last_fullsized":"2024-03-28T18:42:31.360236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403382+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:42:31.403382+0000","last_peered":"2024-03-28T18:42:31.403382+0000","last_clean":"2024-03-28T18:42:31.403382+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:42:31.403382+0000","last_undegraded":"2024-03-28T18:42:31.403382+0000","last_fullsized":"2024-03-28T18:42:31.403382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167948+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:42:31.167948+0000","last_peered":"2024-03-28T18:42:31.167948+0000","last_clean":"2024-03-28T18:42:31.167948+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:42:31.167948+0000","last_undegraded":"2024-03-28T18:42:31.167948+0000","last_fullsized":"2024-03-28T18:42:31.167948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269948+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:42:31.269948+0000","last_peered":"2024-03-28T18:42:31.269948+0000","last_clean":"2024-03-28T18:42:31.269948+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:42:31.269948+0000","last_undegraded":"2024-03-28T18:42:31.269948+0000","last_fullsized":"2024-03-28T18:42:31.269948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360462+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:42:31.360462+0000","last_peered":"2024-03-28T18:42:31.360462+0000","last_clean":"2024-03-28T18:42:31.360462+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:42:31.360462+0000","last_undegraded":"2024-03-28T18:42:31.360462+0000","last_fullsized":"2024-03-28T18:42:31.360462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299890+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:42:31.299890+0000","last_peered":"2024-03-28T18:42:31.299890+0000","last_clean":"2024-03-28T18:42:31.299890+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:42:31.299890+0000","last_undegraded":"2024-03-28T18:42:31.299890+0000","last_fullsized":"2024-03-28T18:42:31.299890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175358+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:42:31.175358+0000","last_peered":"2024-03-28T18:42:31.175358+0000","last_clean":"2024-03-28T18:42:31.175358+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:42:31.175358+0000","last_undegraded":"2024-03-28T18:42:31.175358+0000","last_fullsized":"2024-03-28T18:42:31.175358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.180539+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:42:31.180539+0000","last_peered":"2024-03-28T18:42:31.180539+0000","last_clean":"2024-03-28T18:42:31.180539+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:42:31.180539+0000","last_undegraded":"2024-03-28T18:42:31.180539+0000","last_fullsized":"2024-03-28T18:42:31.180539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.180225+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:42:31.180225+0000","last_peered":"2024-03-28T18:42:31.180225+0000","last_clean":"2024-03-28T18:42:31.180225+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:42:31.180225+0000","last_undegraded":"2024-03-28T18:42:31.180225+0000","last_fullsized":"2024-03-28T18:42:31.180225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167476+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:42:31.167476+0000","last_peered":"2024-03-28T18:42:31.167476+0000","last_clean":"2024-03-28T18:42:31.167476+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:42:31.167476+0000","last_undegraded":"2024-03-28T18:42:31.167476+0000","last_fullsized":"2024-03-28T18:42:31.167476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411470+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:42:31.411470+0000","last_peered":"2024-03-28T18:42:31.411470+0000","last_clean":"2024-03-28T18:42:31.411470+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:42:31.411470+0000","last_undegraded":"2024-03-28T18:42:31.411470+0000","last_fullsized":"2024-03-28T18:42:31.411470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172559+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:42:31.172559+0000","last_peered":"2024-03-28T18:42:31.172559+0000","last_clean":"2024-03-28T18:42:31.172559+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:42:31.172559+0000","last_undegraded":"2024-03-28T18:42:31.172559+0000","last_fullsized":"2024-03-28T18:42:31.172559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317555+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:42:31.317555+0000","last_peered":"2024-03-28T18:42:31.317555+0000","last_clean":"2024-03-28T18:42:31.317555+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:42:31.317555+0000","last_undegraded":"2024-03-28T18:42:31.317555+0000","last_fullsized":"2024-03-28T18:42:31.317555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:43.387035+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:42:43.387035+0000","last_peered":"2024-03-28T18:42:43.387035+0000","last_clean":"2024-03-28T18:42:43.387035+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:42:43.387035+0000","last_undegraded":"2024-03-28T18:42:43.387035+0000","last_fullsized":"2024-03-28T18:42:43.387035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404702+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:42:31.404702+0000","last_peered":"2024-03-28T18:42:31.404702+0000","last_clean":"2024-03-28T18:42:31.404702+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:42:31.404702+0000","last_undegraded":"2024-03-28T18:42:31.404702+0000","last_fullsized":"2024-03-28T18:42:31.404702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404920+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:42:31.404920+0000","last_peered":"2024-03-28T18:42:31.404920+0000","last_clean":"2024-03-28T18:42:31.404920+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:42:31.404920+0000","last_undegraded":"2024-03-28T18:42:31.404920+0000","last_fullsized":"2024-03-28T18:42:31.404920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.183065+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:42:31.183065+0000","last_peered":"2024-03-28T18:42:31.183065+0000","last_clean":"2024-03-28T18:42:31.183065+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:42:31.183065+0000","last_undegraded":"2024-03-28T18:42:31.183065+0000","last_fullsized":"2024-03-28T18:42:31.183065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239229+0000","last_change":"2024-03-28T18:41:30.335571+0000","last_active":"2024-03-28T18:42:31.239229+0000","last_peered":"2024-03-28T18:42:31.239229+0000","last_clean":"2024-03-28T18:42:31.239229+0000","last_became_active":"2024-03-28T18:41:30.311420+0000","last_became_peered":"2024-03-28T18:41:30.311420+0000","last_unstale":"2024-03-28T18:42:31.239229+0000","last_undegraded":"2024-03-28T18:42:31.239229+0000","last_fullsized":"2024-03-28T18:42:31.239229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:39.796667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167624+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:42:31.167624+0000","last_peered":"2024-03-28T18:42:31.167624+0000","last_clean":"2024-03-28T18:42:31.167624+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:42:31.167624+0000","last_undegraded":"2024-03-28T18:42:31.167624+0000","last_fullsized":"2024-03-28T18:42:31.167624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239193+0000","last_change":"2024-03-28T18:41:30.315981+0000","last_active":"2024-03-28T18:42:31.239193+0000","last_peered":"2024-03-28T18:42:31.239193+0000","last_clean":"2024-03-28T18:42:31.239193+0000","last_became_active":"2024-03-28T18:41:30.311208+0000","last_became_peered":"2024-03-28T18:41:30.311208+0000","last_unstale":"2024-03-28T18:42:31.239193+0000","last_undegraded":"2024-03-28T18:42:31.239193+0000","last_fullsized":"2024-03-28T18:42:31.239193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:10.817446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168537+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:42:31.168537+0000","last_peered":"2024-03-28T18:42:31.168537+0000","last_clean":"2024-03-28T18:42:31.168537+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:42:31.168537+0000","last_undegraded":"2024-03-28T18:42:31.168537+0000","last_fullsized":"2024-03-28T18:42:31.168537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":111,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:46.285550+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:42:46.285550+0000","last_peered":"2024-03-28T18:42:46.285550+0000","last_clean":"2024-03-28T18:42:46.285550+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:42:46.285550+0000","last_undegraded":"2024-03-28T18:42:46.285550+0000","last_fullsized":"2024-03-28T18:42:46.285550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238949+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:42:31.238949+0000","last_peered":"2024-03-28T18:42:31.238949+0000","last_clean":"2024-03-28T18:42:31.238949+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:42:31.238949+0000","last_undegraded":"2024-03-28T18:42:31.238949+0000","last_fullsized":"2024-03-28T18:42:31.238949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269565+0000","last_change":"2024-03-28T18:41:30.228521+0000","last_active":"2024-03-28T18:42:31.269565+0000","last_peered":"2024-03-28T18:42:31.269565+0000","last_clean":"2024-03-28T18:42:31.269565+0000","last_became_active":"2024-03-28T18:41:30.221867+0000","last_became_peered":"2024-03-28T18:41:30.221867+0000","last_unstale":"2024-03-28T18:42:31.269565+0000","last_undegraded":"2024-03-28T18:42:31.269565+0000","last_fullsized":"2024-03-28T18:42:31.269565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:45.207508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168453+0000","last_change":"2024-03-28T18:41:30.507663+0000","last_active":"2024-03-28T18:42:31.168453+0000","last_peered":"2024-03-28T18:42:31.168453+0000","last_clean":"2024-03-28T18:42:31.168453+0000","last_became_active":"2024-03-28T18:41:30.507284+0000","last_became_peered":"2024-03-28T18:41:30.507284+0000","last_unstale":"2024-03-28T18:42:31.168453+0000","last_undegraded":"2024-03-28T18:42:31.168453+0000","last_fullsized":"2024-03-28T18:42:31.168453+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:03.045818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175018+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:42:31.175018+0000","last_peered":"2024-03-28T18:42:31.175018+0000","last_clean":"2024-03-28T18:42:31.175018+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:42:31.175018+0000","last_undegraded":"2024-03-28T18:42:31.175018+0000","last_fullsized":"2024-03-28T18:42:31.175018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403246+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:42:31.403246+0000","last_peered":"2024-03-28T18:42:31.403246+0000","last_clean":"2024-03-28T18:42:31.403246+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:42:31.403246+0000","last_undegraded":"2024-03-28T18:42:31.403246+0000","last_fullsized":"2024-03-28T18:42:31.403246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168025+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:42:31.168025+0000","last_peered":"2024-03-28T18:42:31.168025+0000","last_clean":"2024-03-28T18:42:31.168025+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:42:31.168025+0000","last_undegraded":"2024-03-28T18:42:31.168025+0000","last_fullsized":"2024-03-28T18:42:31.168025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172247+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:42:31.172247+0000","last_peered":"2024-03-28T18:42:31.172247+0000","last_clean":"2024-03-28T18:42:31.172247+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:42:31.172247+0000","last_undegraded":"2024-03-28T18:42:31.172247+0000","last_fullsized":"2024-03-28T18:42:31.172247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:47.286252+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:42:47.286252+0000","last_peered":"2024-03-28T18:42:47.286252+0000","last_clean":"2024-03-28T18:42:47.286252+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:42:47.286252+0000","last_undegraded":"2024-03-28T18:42:47.286252+0000","last_fullsized":"2024-03-28T18:42:47.286252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167797+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:42:31.167797+0000","last_peered":"2024-03-28T18:42:31.167797+0000","last_clean":"2024-03-28T18:42:31.167797+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:42:31.167797+0000","last_undegraded":"2024-03-28T18:42:31.167797+0000","last_fullsized":"2024-03-28T18:42:31.167797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360329+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:42:31.360329+0000","last_peered":"2024-03-28T18:42:31.360329+0000","last_clean":"2024-03-28T18:42:31.360329+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:42:31.360329+0000","last_undegraded":"2024-03-28T18:42:31.360329+0000","last_fullsized":"2024-03-28T18:42:31.360329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.183289+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:42:31.183289+0000","last_peered":"2024-03-28T18:42:31.183289+0000","last_clean":"2024-03-28T18:42:31.183289+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:42:31.183289+0000","last_undegraded":"2024-03-28T18:42:31.183289+0000","last_fullsized":"2024-03-28T18:42:31.183289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168630+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:42:31.168630+0000","last_peered":"2024-03-28T18:42:31.168630+0000","last_clean":"2024-03-28T18:42:31.168630+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:42:31.168630+0000","last_undegraded":"2024-03-28T18:42:31.168630+0000","last_fullsized":"2024-03-28T18:42:31.168630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.303127+0000","last_change":"2024-03-28T18:42:31.303127+0000","last_active":"2024-03-28T18:42:31.303127+0000","last_peered":"2024-03-28T18:42:31.303127+0000","last_clean":"2024-03-28T18:42:31.303127+0000","last_became_active":"2024-03-28T18:42:31.301321+0000","last_became_peered":"2024-03-28T18:42:31.301321+0000","last_unstale":"2024-03-28T18:42:31.303127+0000","last_undegraded":"2024-03-28T18:42:31.303127+0000","last_fullsized":"2024-03-28T18:42:31.303127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:08.892803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":223,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:47.286270+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:42:47.286270+0000","last_peered":"2024-03-28T18:42:47.286270+0000","last_clean":"2024-03-28T18:42:47.286270+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:42:47.286270+0000","last_undegraded":"2024-03-28T18:42:47.286270+0000","last_fullsized":"2024-03-28T18:42:47.286270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61701,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":35,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239136+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:42:31.239136+0000","last_peered":"2024-03-28T18:42:31.239136+0000","last_clean":"2024-03-28T18:42:31.239136+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:42:31.239136+0000","last_undegraded":"2024-03-28T18:42:31.239136+0000","last_fullsized":"2024-03-28T18:42:31.239136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317559+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:42:31.317559+0000","last_peered":"2024-03-28T18:42:31.317559+0000","last_clean":"2024-03-28T18:42:31.317559+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:42:31.317559+0000","last_undegraded":"2024-03-28T18:42:31.317559+0000","last_fullsized":"2024-03-28T18:42:31.317559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174870+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:42:31.174870+0000","last_peered":"2024-03-28T18:42:31.174870+0000","last_clean":"2024-03-28T18:42:31.174870+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:42:31.174870+0000","last_undegraded":"2024-03-28T18:42:31.174870+0000","last_fullsized":"2024-03-28T18:42:31.174870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168225+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:42:31.168225+0000","last_peered":"2024-03-28T18:42:31.168225+0000","last_clean":"2024-03-28T18:42:31.168225+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:42:31.168225+0000","last_undegraded":"2024-03-28T18:42:31.168225+0000","last_fullsized":"2024-03-28T18:42:31.168225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360547+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:42:31.360547+0000","last_peered":"2024-03-28T18:42:31.360547+0000","last_clean":"2024-03-28T18:42:31.360547+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:42:31.360547+0000","last_undegraded":"2024-03-28T18:42:31.360547+0000","last_fullsized":"2024-03-28T18:42:31.360547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:43.316978+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:42:43.316978+0000","last_peered":"2024-03-28T18:42:43.316978+0000","last_clean":"2024-03-28T18:42:43.316978+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:42:43.316978+0000","last_undegraded":"2024-03-28T18:42:43.316978+0000","last_fullsized":"2024-03-28T18:42:43.316978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360164+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:42:31.360164+0000","last_peered":"2024-03-28T18:42:31.360164+0000","last_clean":"2024-03-28T18:42:31.360164+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:42:31.360164+0000","last_undegraded":"2024-03-28T18:42:31.360164+0000","last_fullsized":"2024-03-28T18:42:31.360164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238835+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:42:31.238835+0000","last_peered":"2024-03-28T18:42:31.238835+0000","last_clean":"2024-03-28T18:42:31.238835+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:42:31.238835+0000","last_undegraded":"2024-03-28T18:42:31.238835+0000","last_fullsized":"2024-03-28T18:42:31.238835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168124+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:42:31.168124+0000","last_peered":"2024-03-28T18:42:31.168124+0000","last_clean":"2024-03-28T18:42:31.168124+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:42:31.168124+0000","last_undegraded":"2024-03-28T18:42:31.168124+0000","last_fullsized":"2024-03-28T18:42:31.168124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239411+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:42:31.239411+0000","last_peered":"2024-03-28T18:42:31.239411+0000","last_clean":"2024-03-28T18:42:31.239411+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:42:31.239411+0000","last_undegraded":"2024-03-28T18:42:31.239411+0000","last_fullsized":"2024-03-28T18:42:31.239411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411308+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:42:31.411308+0000","last_peered":"2024-03-28T18:42:31.411308+0000","last_clean":"2024-03-28T18:42:31.411308+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:42:31.411308+0000","last_undegraded":"2024-03-28T18:42:31.411308+0000","last_fullsized":"2024-03-28T18:42:31.411308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.173845+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:42:31.173845+0000","last_peered":"2024-03-28T18:42:31.173845+0000","last_clean":"2024-03-28T18:42:31.173845+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:42:31.173845+0000","last_undegraded":"2024-03-28T18:42:31.173845+0000","last_fullsized":"2024-03-28T18:42:31.173845+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269901+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:42:31.269901+0000","last_peered":"2024-03-28T18:42:31.269901+0000","last_clean":"2024-03-28T18:42:31.269901+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:42:31.269901+0000","last_undegraded":"2024-03-28T18:42:31.269901+0000","last_fullsized":"2024-03-28T18:42:31.269901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.392452+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:42:45.392452+0000","last_peered":"2024-03-28T18:42:45.392452+0000","last_clean":"2024-03-28T18:42:45.392452+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:42:45.392452+0000","last_undegraded":"2024-03-28T18:42:45.392452+0000","last_fullsized":"2024-03-28T18:42:45.392452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269813+0000","last_change":"2024-03-28T18:41:30.219905+0000","last_active":"2024-03-28T18:42:31.269813+0000","last_peered":"2024-03-28T18:42:31.269813+0000","last_clean":"2024-03-28T18:42:31.269813+0000","last_became_active":"2024-03-28T18:41:30.219488+0000","last_became_peered":"2024-03-28T18:41:30.219488+0000","last_unstale":"2024-03-28T18:42:31.269813+0000","last_undegraded":"2024-03-28T18:42:31.269813+0000","last_fullsized":"2024-03-28T18:42:31.269813+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:16.108678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.320096+0000","last_change":"2024-03-28T18:42:31.320096+0000","last_active":"2024-03-28T18:42:31.320096+0000","last_peered":"2024-03-28T18:42:31.320096+0000","last_clean":"2024-03-28T18:42:31.320096+0000","last_became_active":"2024-03-28T18:42:31.318988+0000","last_became_peered":"2024-03-28T18:42:31.318988+0000","last_unstale":"2024-03-28T18:42:31.320096+0000","last_undegraded":"2024-03-28T18:42:31.320096+0000","last_fullsized":"2024-03-28T18:42:31.320096+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:50.019195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.319663+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:42:45.319663+0000","last_peered":"2024-03-28T18:42:45.319663+0000","last_clean":"2024-03-28T18:42:45.319663+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:42:45.319663+0000","last_undegraded":"2024-03-28T18:42:45.319663+0000","last_fullsized":"2024-03-28T18:42:45.319663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403866+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:42:31.403866+0000","last_peered":"2024-03-28T18:42:31.403866+0000","last_clean":"2024-03-28T18:42:31.403866+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:42:31.403866+0000","last_undegraded":"2024-03-28T18:42:31.403866+0000","last_fullsized":"2024-03-28T18:42:31.403866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299716+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:42:31.299716+0000","last_peered":"2024-03-28T18:42:31.299716+0000","last_clean":"2024-03-28T18:42:31.299716+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:42:31.299716+0000","last_undegraded":"2024-03-28T18:42:31.299716+0000","last_fullsized":"2024-03-28T18:42:31.299716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360313+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:42:31.360313+0000","last_peered":"2024-03-28T18:42:31.360313+0000","last_clean":"2024-03-28T18:42:31.360313+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:42:31.360313+0000","last_undegraded":"2024-03-28T18:42:31.360313+0000","last_fullsized":"2024-03-28T18:42:31.360313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174559+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:42:31.174559+0000","last_peered":"2024-03-28T18:42:31.174559+0000","last_clean":"2024-03-28T18:42:31.174559+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:42:31.174559+0000","last_undegraded":"2024-03-28T18:42:31.174559+0000","last_fullsized":"2024-03-28T18:42:31.174559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360338+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:42:31.360338+0000","last_peered":"2024-03-28T18:42:31.360338+0000","last_clean":"2024-03-28T18:42:31.360338+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:42:31.360338+0000","last_undegraded":"2024-03-28T18:42:31.360338+0000","last_fullsized":"2024-03-28T18:42:31.360338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411274+0000","last_change":"2024-03-28T18:41:30.227916+0000","last_active":"2024-03-28T18:42:31.411274+0000","last_peered":"2024-03-28T18:42:31.411274+0000","last_clean":"2024-03-28T18:42:31.411274+0000","last_became_active":"2024-03-28T18:41:30.218947+0000","last_became_peered":"2024-03-28T18:41:30.218947+0000","last_unstale":"2024-03-28T18:42:31.411274+0000","last_undegraded":"2024-03-28T18:42:31.411274+0000","last_fullsized":"2024-03-28T18:42:31.411274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:14.922324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175978+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:42:31.175978+0000","last_peered":"2024-03-28T18:42:31.175978+0000","last_clean":"2024-03-28T18:42:31.175978+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:42:31.175978+0000","last_undegraded":"2024-03-28T18:42:31.175978+0000","last_fullsized":"2024-03-28T18:42:31.175978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172882+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:42:31.172882+0000","last_peered":"2024-03-28T18:42:31.172882+0000","last_clean":"2024-03-28T18:42:31.172882+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:42:31.172882+0000","last_undegraded":"2024-03-28T18:42:31.172882+0000","last_fullsized":"2024-03-28T18:42:31.172882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.405004+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:42:31.405004+0000","last_peered":"2024-03-28T18:42:31.405004+0000","last_clean":"2024-03-28T18:42:31.405004+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:42:31.405004+0000","last_undegraded":"2024-03-28T18:42:31.405004+0000","last_fullsized":"2024-03-28T18:42:31.405004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299800+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:42:31.299800+0000","last_peered":"2024-03-28T18:42:31.299800+0000","last_clean":"2024-03-28T18:42:31.299800+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:42:31.299800+0000","last_undegraded":"2024-03-28T18:42:31.299800+0000","last_fullsized":"2024-03-28T18:42:31.299800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.302339+0000","last_change":"2024-03-28T18:42:31.302339+0000","last_active":"2024-03-28T18:42:31.302339+0000","last_peered":"2024-03-28T18:42:31.302339+0000","last_clean":"2024-03-28T18:42:31.302339+0000","last_became_active":"2024-03-28T18:42:31.301324+0000","last_became_peered":"2024-03-28T18:42:31.301324+0000","last_unstale":"2024-03-28T18:42:31.302339+0000","last_undegraded":"2024-03-28T18:42:31.302339+0000","last_fullsized":"2024-03-28T18:42:31.302339+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:56.490969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167476+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:42:31.167476+0000","last_peered":"2024-03-28T18:42:31.167476+0000","last_clean":"2024-03-28T18:42:31.167476+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:42:31.167476+0000","last_undegraded":"2024-03-28T18:42:31.167476+0000","last_fullsized":"2024-03-28T18:42:31.167476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238841+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:42:31.238841+0000","last_peered":"2024-03-28T18:42:31.238841+0000","last_clean":"2024-03-28T18:42:31.238841+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:42:31.238841+0000","last_undegraded":"2024-03-28T18:42:31.238841+0000","last_fullsized":"2024-03-28T18:42:31.238841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403498+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:42:31.403498+0000","last_peered":"2024-03-28T18:42:31.403498+0000","last_clean":"2024-03-28T18:42:31.403498+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:42:31.403498+0000","last_undegraded":"2024-03-28T18:42:31.403498+0000","last_fullsized":"2024-03-28T18:42:31.403498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411114+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:42:31.411114+0000","last_peered":"2024-03-28T18:42:31.411114+0000","last_clean":"2024-03-28T18:42:31.411114+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:42:31.411114+0000","last_undegraded":"2024-03-28T18:42:31.411114+0000","last_fullsized":"2024-03-28T18:42:31.411114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317547+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:42:31.317547+0000","last_peered":"2024-03-28T18:42:31.317547+0000","last_clean":"2024-03-28T18:42:31.317547+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:42:31.317547+0000","last_undegraded":"2024-03-28T18:42:31.317547+0000","last_fullsized":"2024-03-28T18:42:31.317547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404069+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:42:31.404069+0000","last_peered":"2024-03-28T18:42:31.404069+0000","last_clean":"2024-03-28T18:42:31.404069+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:42:31.404069+0000","last_undegraded":"2024-03-28T18:42:31.404069+0000","last_fullsized":"2024-03-28T18:42:31.404069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238985+0000","last_change":"2024-03-28T18:41:30.319270+0000","last_active":"2024-03-28T18:42:31.238985+0000","last_peered":"2024-03-28T18:42:31.238985+0000","last_clean":"2024-03-28T18:42:31.238985+0000","last_became_active":"2024-03-28T18:41:30.311219+0000","last_became_peered":"2024-03-28T18:41:30.311219+0000","last_unstale":"2024-03-28T18:42:31.238985+0000","last_undegraded":"2024-03-28T18:42:31.238985+0000","last_fullsized":"2024-03-28T18:42:31.238985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:23.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360610+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:42:31.360610+0000","last_peered":"2024-03-28T18:42:31.360610+0000","last_clean":"2024-03-28T18:42:31.360610+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:42:31.360610+0000","last_undegraded":"2024-03-28T18:42:31.360610+0000","last_fullsized":"2024-03-28T18:42:31.360610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.340188+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:42:45.340188+0000","last_peered":"2024-03-28T18:42:45.340188+0000","last_clean":"2024-03-28T18:42:45.340188+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:42:45.340188+0000","last_undegraded":"2024-03-28T18:42:45.340188+0000","last_fullsized":"2024-03-28T18:42:45.340188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360460+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:42:31.360460+0000","last_peered":"2024-03-28T18:42:31.360460+0000","last_clean":"2024-03-28T18:42:31.360460+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:42:31.360460+0000","last_undegraded":"2024-03-28T18:42:31.360460+0000","last_fullsized":"2024-03-28T18:42:31.360460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168432+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:42:31.168432+0000","last_peered":"2024-03-28T18:42:31.168432+0000","last_clean":"2024-03-28T18:42:31.168432+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:42:31.168432+0000","last_undegraded":"2024-03-28T18:42:31.168432+0000","last_fullsized":"2024-03-28T18:42:31.168432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360345+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:42:31.360345+0000","last_peered":"2024-03-28T18:42:31.360345+0000","last_clean":"2024-03-28T18:42:31.360345+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:42:31.360345+0000","last_undegraded":"2024-03-28T18:42:31.360345+0000","last_fullsized":"2024-03-28T18:42:31.360345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.176185+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:42:31.176185+0000","last_peered":"2024-03-28T18:42:31.176185+0000","last_clean":"2024-03-28T18:42:31.176185+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:42:31.176185+0000","last_undegraded":"2024-03-28T18:42:31.176185+0000","last_fullsized":"2024-03-28T18:42:31.176185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360591+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:42:31.360591+0000","last_peered":"2024-03-28T18:42:31.360591+0000","last_clean":"2024-03-28T18:42:31.360591+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:42:31.360591+0000","last_undegraded":"2024-03-28T18:42:31.360591+0000","last_fullsized":"2024-03-28T18:42:31.360591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411179+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:42:31.411179+0000","last_peered":"2024-03-28T18:42:31.411179+0000","last_clean":"2024-03-28T18:42:31.411179+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:42:31.411179+0000","last_undegraded":"2024-03-28T18:42:31.411179+0000","last_fullsized":"2024-03-28T18:42:31.411179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317513+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:42:31.317513+0000","last_peered":"2024-03-28T18:42:31.317513+0000","last_clean":"2024-03-28T18:42:31.317513+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:42:31.317513+0000","last_undegraded":"2024-03-28T18:42:31.317513+0000","last_fullsized":"2024-03-28T18:42:31.317513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317445+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:42:31.317445+0000","last_peered":"2024-03-28T18:42:31.317445+0000","last_clean":"2024-03-28T18:42:31.317445+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:42:31.317445+0000","last_undegraded":"2024-03-28T18:42:31.317445+0000","last_fullsized":"2024-03-28T18:42:31.317445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.362520+0000","last_change":"2024-03-28T18:42:31.362520+0000","last_active":"2024-03-28T18:42:31.362520+0000","last_peered":"2024-03-28T18:42:31.362520+0000","last_clean":"2024-03-28T18:42:31.362520+0000","last_became_active":"2024-03-28T18:42:31.361563+0000","last_became_peered":"2024-03-28T18:42:31.361563+0000","last_unstale":"2024-03-28T18:42:31.362520+0000","last_undegraded":"2024-03-28T18:42:31.362520+0000","last_fullsized":"2024-03-28T18:42:31.362520+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:27.960042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175632+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:42:31.175632+0000","last_peered":"2024-03-28T18:42:31.175632+0000","last_clean":"2024-03-28T18:42:31.175632+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:42:31.175632+0000","last_undegraded":"2024-03-28T18:42:31.175632+0000","last_fullsized":"2024-03-28T18:42:31.175632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411422+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:42:31.411422+0000","last_peered":"2024-03-28T18:42:31.411422+0000","last_clean":"2024-03-28T18:42:31.411422+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:42:31.411422+0000","last_undegraded":"2024-03-28T18:42:31.411422+0000","last_fullsized":"2024-03-28T18:42:31.411422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317666+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:42:31.317666+0000","last_peered":"2024-03-28T18:42:31.317666+0000","last_clean":"2024-03-28T18:42:31.317666+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:42:31.317666+0000","last_undegraded":"2024-03-28T18:42:31.317666+0000","last_fullsized":"2024-03-28T18:42:31.317666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404920+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:42:31.404920+0000","last_peered":"2024-03-28T18:42:31.404920+0000","last_clean":"2024-03-28T18:42:31.404920+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:42:31.404920+0000","last_undegraded":"2024-03-28T18:42:31.404920+0000","last_fullsized":"2024-03-28T18:42:31.404920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174053+0000","last_change":"2024-03-28T18:41:30.510267+0000","last_active":"2024-03-28T18:42:31.174053+0000","last_peered":"2024-03-28T18:42:31.174053+0000","last_clean":"2024-03-28T18:42:31.174053+0000","last_became_active":"2024-03-28T18:41:30.508658+0000","last_became_peered":"2024-03-28T18:41:30.508658+0000","last_unstale":"2024-03-28T18:42:31.174053+0000","last_undegraded":"2024-03-28T18:42:31.174053+0000","last_fullsized":"2024-03-28T18:42:31.174053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:30.892433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403310+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:42:31.403310+0000","last_peered":"2024-03-28T18:42:31.403310+0000","last_clean":"2024-03-28T18:42:31.403310+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:42:31.403310+0000","last_undegraded":"2024-03-28T18:42:31.403310+0000","last_fullsized":"2024-03-28T18:42:31.403310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167733+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:42:31.167733+0000","last_peered":"2024-03-28T18:42:31.167733+0000","last_clean":"2024-03-28T18:42:31.167733+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:42:31.167733+0000","last_undegraded":"2024-03-28T18:42:31.167733+0000","last_fullsized":"2024-03-28T18:42:31.167733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175533+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:42:31.175533+0000","last_peered":"2024-03-28T18:42:31.175533+0000","last_clean":"2024-03-28T18:42:31.175533+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:42:31.175533+0000","last_undegraded":"2024-03-28T18:42:31.175533+0000","last_fullsized":"2024-03-28T18:42:31.175533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411254+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:42:31.411254+0000","last_peered":"2024-03-28T18:42:31.411254+0000","last_clean":"2024-03-28T18:42:31.411254+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:42:31.411254+0000","last_undegraded":"2024-03-28T18:42:31.411254+0000","last_fullsized":"2024-03-28T18:42:31.411254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68905,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":824,"num_read_kb":485,"num_write":240,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477349,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1138688,"data_stored":1130358,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776170,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":477167,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477433,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":1070454,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":476884,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":539498,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":478223,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":540109,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":476709,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888135,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":477718,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":1069159,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:42:49.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T18:42:49.948 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:49 smithi131 ceph-mon[36065]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:49.948 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:49 smithi131 ceph-mon[36065]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:50.418 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:52.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:52 smithi119 ceph-mon[31813]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:52.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:52 smithi196 ceph-mon[36300]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:52.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:52 smithi131 ceph-mon[36065]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:42:52.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 -- 172.21.15.119:0/3702564210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43901081e0 msgr2=0x7f43901085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:52.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 --2- 172.21.15.119:0/3702564210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43901081e0 0x7f43901085c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f43800099c0 tx=0x7f4380009cd0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 -- 172.21.15.119:0/3702564210 shutdown_connections 2024-03-28T18:42:52.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 --2- 172.21.15.119:0/3702564210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f439006a7e0 0x7f439010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 --2- 172.21.15.119:0/3702564210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43901081e0 0x7f43901085c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 --2- 172.21.15.119:0/3702564210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4390069e40 0x7f439006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 -- 172.21.15.119:0/3702564210 >> 172.21.15.119:0/3702564210 conn(0x7f4390077fc0 msgr2=0x7f43900783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 -- 172.21.15.119:0/3702564210 shutdown_connections 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.720+0000 7f4396400700 1 -- 172.21.15.119:0/3702564210 wait complete. 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.721+0000 7f4396400700 1 Processor -- start 2024-03-28T18:42:52.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 -- start start 2024-03-28T18:42:52.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:52.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f439006a7e0 0x7f43901a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:52.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43901081e0 0x7f439019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:52.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390112470 con 0x7f439006a7e0 2024-03-28T18:42:52.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901122f0 con 0x7f43901081e0 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f4396400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43901125f0 con 0x7f4390069e40 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f438f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f438f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48728/0 (socket says 172.21.15.119:48728) 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f438f400700 1 -- 172.21.15.119:0/1241315785 learned_addr learned my addr 172.21.15.119:0/1241315785 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f438fe00700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43901081e0 0x7f439019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.722+0000 7f438ea00700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f439006a7e0 0x7f43901a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 -- 172.21.15.119:0/1241315785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43901081e0 msgr2=0x7f439019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43901081e0 0x7f439019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 -- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f439006a7e0 msgr2=0x7f43901a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f439006a7e0 0x7f43901a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 -- 172.21.15.119:0/1241315785 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439019b150 con 0x7f4390069e40 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f438f400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4380005910 tx=0x7f438000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:52.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380027070 con 0x7f4390069e40 2024-03-28T18:42:52.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4380003680 con 0x7f4390069e40 2024-03-28T18:42:52.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439019b3e0 con 0x7f4390069e40 2024-03-28T18:42:52.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43800173f0 con 0x7f4390069e40 2024-03-28T18:42:52.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.723+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43901a7700 con 0x7f4390069e40 2024-03-28T18:42:52.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.724+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4390068e50 con 0x7f4390069e40 2024-03-28T18:42:52.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.726+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f438002a070 con 0x7f4390069e40 2024-03-28T18:42:52.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.727+0000 7f4387e00700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 0x7f43740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:52.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.728+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f43800cf960 con 0x7f4390069e40 2024-03-28T18:42:52.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.728+0000 7f438ea00700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 0x7f43740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:52.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.728+0000 7f438ea00700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 0x7f43740a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4390103220 tx=0x7f437c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:52.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.736+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f438005e740 con 0x7f4390069e40 2024-03-28T18:42:52.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.960+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f439010da80 con 0x7f43740a29a0 2024-03-28T18:42:52.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.982+0000 7f4387e00700 1 -- 172.21.15.119:0/1241315785 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259008 (secure 0 0 0) 0x7f439010da80 con 0x7f43740a29a0 2024-03-28T18:42:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 msgr2=0x7f43740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 0x7f43740a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4390103220 tx=0x7f437c00b410 comp rx=0 tx=0).stop 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 msgr2=0x7f43901a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4380005910 tx=0x7f438000fa00 comp rx=0 tx=0).stop 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 shutdown_connections 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f43740a29a0 0x7f43740a4e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43901081e0 0x7f439019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4390069e40 0x7f43901a0830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 --2- 172.21.15.119:0/1241315785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f439006a7e0 0x7f43901a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 >> 172.21.15.119:0/1241315785 conn(0x7f4390077fc0 msgr2=0x7f439010d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 shutdown_connections 2024-03-28T18:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:52.988+0000 7f4396400700 1 -- 172.21.15.119:0/1241315785 wait complete. 2024-03-28T18:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:42:53.936 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-28T18:42:50.998764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659507,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":970,"num_read_kb":609,"num_write":423,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361324,"kb_used_data":12096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580500,"statfs":{"total":1151940427776,"available":1151570432000,"internally_reserved":0,"allocated":12386304,"data_stored":7691061,"data_compressed":246444,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005096"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411432+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:42:31.411432+0000","last_peered":"2024-03-28T18:42:31.411432+0000","last_clean":"2024-03-28T18:42:31.411432+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:42:31.411432+0000","last_undegraded":"2024-03-28T18:42:31.411432+0000","last_fullsized":"2024-03-28T18:42:31.411432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.182960+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:42:31.182960+0000","last_peered":"2024-03-28T18:42:31.182960+0000","last_clean":"2024-03-28T18:42:31.182960+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:42:31.182960+0000","last_undegraded":"2024-03-28T18:42:31.182960+0000","last_fullsized":"2024-03-28T18:42:31.182960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.182143+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:42:31.182143+0000","last_peered":"2024-03-28T18:42:31.182143+0000","last_clean":"2024-03-28T18:42:31.182143+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:42:31.182143+0000","last_undegraded":"2024-03-28T18:42:31.182143+0000","last_fullsized":"2024-03-28T18:42:31.182143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299891+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:42:31.299891+0000","last_peered":"2024-03-28T18:42:31.299891+0000","last_clean":"2024-03-28T18:42:31.299891+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:42:31.299891+0000","last_undegraded":"2024-03-28T18:42:31.299891+0000","last_fullsized":"2024-03-28T18:42:31.299891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239016+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:42:31.239016+0000","last_peered":"2024-03-28T18:42:31.239016+0000","last_clean":"2024-03-28T18:42:31.239016+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:42:31.239016+0000","last_undegraded":"2024-03-28T18:42:31.239016+0000","last_fullsized":"2024-03-28T18:42:31.239016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269632+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:42:31.269632+0000","last_peered":"2024-03-28T18:42:31.269632+0000","last_clean":"2024-03-28T18:42:31.269632+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:42:31.269632+0000","last_undegraded":"2024-03-28T18:42:31.269632+0000","last_fullsized":"2024-03-28T18:42:31.269632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360236+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:42:31.360236+0000","last_peered":"2024-03-28T18:42:31.360236+0000","last_clean":"2024-03-28T18:42:31.360236+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:42:31.360236+0000","last_undegraded":"2024-03-28T18:42:31.360236+0000","last_fullsized":"2024-03-28T18:42:31.360236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403382+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:42:31.403382+0000","last_peered":"2024-03-28T18:42:31.403382+0000","last_clean":"2024-03-28T18:42:31.403382+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:42:31.403382+0000","last_undegraded":"2024-03-28T18:42:31.403382+0000","last_fullsized":"2024-03-28T18:42:31.403382+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167948+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:42:31.167948+0000","last_peered":"2024-03-28T18:42:31.167948+0000","last_clean":"2024-03-28T18:42:31.167948+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:42:31.167948+0000","last_undegraded":"2024-03-28T18:42:31.167948+0000","last_fullsized":"2024-03-28T18:42:31.167948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269948+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:42:31.269948+0000","last_peered":"2024-03-28T18:42:31.269948+0000","last_clean":"2024-03-28T18:42:31.269948+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:42:31.269948+0000","last_undegraded":"2024-03-28T18:42:31.269948+0000","last_fullsized":"2024-03-28T18:42:31.269948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360462+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:42:31.360462+0000","last_peered":"2024-03-28T18:42:31.360462+0000","last_clean":"2024-03-28T18:42:31.360462+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:42:31.360462+0000","last_undegraded":"2024-03-28T18:42:31.360462+0000","last_fullsized":"2024-03-28T18:42:31.360462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299890+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:42:31.299890+0000","last_peered":"2024-03-28T18:42:31.299890+0000","last_clean":"2024-03-28T18:42:31.299890+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:42:31.299890+0000","last_undegraded":"2024-03-28T18:42:31.299890+0000","last_fullsized":"2024-03-28T18:42:31.299890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175358+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:42:31.175358+0000","last_peered":"2024-03-28T18:42:31.175358+0000","last_clean":"2024-03-28T18:42:31.175358+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:42:31.175358+0000","last_undegraded":"2024-03-28T18:42:31.175358+0000","last_fullsized":"2024-03-28T18:42:31.175358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.180539+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:42:31.180539+0000","last_peered":"2024-03-28T18:42:31.180539+0000","last_clean":"2024-03-28T18:42:31.180539+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:42:31.180539+0000","last_undegraded":"2024-03-28T18:42:31.180539+0000","last_fullsized":"2024-03-28T18:42:31.180539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.180225+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:42:31.180225+0000","last_peered":"2024-03-28T18:42:31.180225+0000","last_clean":"2024-03-28T18:42:31.180225+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:42:31.180225+0000","last_undegraded":"2024-03-28T18:42:31.180225+0000","last_fullsized":"2024-03-28T18:42:31.180225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167476+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:42:31.167476+0000","last_peered":"2024-03-28T18:42:31.167476+0000","last_clean":"2024-03-28T18:42:31.167476+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:42:31.167476+0000","last_undegraded":"2024-03-28T18:42:31.167476+0000","last_fullsized":"2024-03-28T18:42:31.167476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411470+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:42:31.411470+0000","last_peered":"2024-03-28T18:42:31.411470+0000","last_clean":"2024-03-28T18:42:31.411470+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:42:31.411470+0000","last_undegraded":"2024-03-28T18:42:31.411470+0000","last_fullsized":"2024-03-28T18:42:31.411470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172559+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:42:31.172559+0000","last_peered":"2024-03-28T18:42:31.172559+0000","last_clean":"2024-03-28T18:42:31.172559+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:42:31.172559+0000","last_undegraded":"2024-03-28T18:42:31.172559+0000","last_fullsized":"2024-03-28T18:42:31.172559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317555+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:42:31.317555+0000","last_peered":"2024-03-28T18:42:31.317555+0000","last_clean":"2024-03-28T18:42:31.317555+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:42:31.317555+0000","last_undegraded":"2024-03-28T18:42:31.317555+0000","last_fullsized":"2024-03-28T18:42:31.317555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:48.397956+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:42:48.397956+0000","last_peered":"2024-03-28T18:42:48.397956+0000","last_clean":"2024-03-28T18:42:48.397956+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:42:48.397956+0000","last_undegraded":"2024-03-28T18:42:48.397956+0000","last_fullsized":"2024-03-28T18:42:48.397956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404702+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:42:31.404702+0000","last_peered":"2024-03-28T18:42:31.404702+0000","last_clean":"2024-03-28T18:42:31.404702+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:42:31.404702+0000","last_undegraded":"2024-03-28T18:42:31.404702+0000","last_fullsized":"2024-03-28T18:42:31.404702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404920+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:42:31.404920+0000","last_peered":"2024-03-28T18:42:31.404920+0000","last_clean":"2024-03-28T18:42:31.404920+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:42:31.404920+0000","last_undegraded":"2024-03-28T18:42:31.404920+0000","last_fullsized":"2024-03-28T18:42:31.404920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.183065+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:42:31.183065+0000","last_peered":"2024-03-28T18:42:31.183065+0000","last_clean":"2024-03-28T18:42:31.183065+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:42:31.183065+0000","last_undegraded":"2024-03-28T18:42:31.183065+0000","last_fullsized":"2024-03-28T18:42:31.183065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239229+0000","last_change":"2024-03-28T18:41:30.335571+0000","last_active":"2024-03-28T18:42:31.239229+0000","last_peered":"2024-03-28T18:42:31.239229+0000","last_clean":"2024-03-28T18:42:31.239229+0000","last_became_active":"2024-03-28T18:41:30.311420+0000","last_became_peered":"2024-03-28T18:41:30.311420+0000","last_unstale":"2024-03-28T18:42:31.239229+0000","last_undegraded":"2024-03-28T18:42:31.239229+0000","last_fullsized":"2024-03-28T18:42:31.239229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:39.796667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167624+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:42:31.167624+0000","last_peered":"2024-03-28T18:42:31.167624+0000","last_clean":"2024-03-28T18:42:31.167624+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:42:31.167624+0000","last_undegraded":"2024-03-28T18:42:31.167624+0000","last_fullsized":"2024-03-28T18:42:31.167624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239193+0000","last_change":"2024-03-28T18:41:30.315981+0000","last_active":"2024-03-28T18:42:31.239193+0000","last_peered":"2024-03-28T18:42:31.239193+0000","last_clean":"2024-03-28T18:42:31.239193+0000","last_became_active":"2024-03-28T18:41:30.311208+0000","last_became_peered":"2024-03-28T18:41:30.311208+0000","last_unstale":"2024-03-28T18:42:31.239193+0000","last_undegraded":"2024-03-28T18:42:31.239193+0000","last_fullsized":"2024-03-28T18:42:31.239193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:10.817446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168537+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:42:31.168537+0000","last_peered":"2024-03-28T18:42:31.168537+0000","last_clean":"2024-03-28T18:42:31.168537+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:42:31.168537+0000","last_undegraded":"2024-03-28T18:42:31.168537+0000","last_fullsized":"2024-03-28T18:42:31.168537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":111,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:46.285550+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:42:46.285550+0000","last_peered":"2024-03-28T18:42:46.285550+0000","last_clean":"2024-03-28T18:42:46.285550+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:42:46.285550+0000","last_undegraded":"2024-03-28T18:42:46.285550+0000","last_fullsized":"2024-03-28T18:42:46.285550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238949+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:42:31.238949+0000","last_peered":"2024-03-28T18:42:31.238949+0000","last_clean":"2024-03-28T18:42:31.238949+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:42:31.238949+0000","last_undegraded":"2024-03-28T18:42:31.238949+0000","last_fullsized":"2024-03-28T18:42:31.238949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269565+0000","last_change":"2024-03-28T18:41:30.228521+0000","last_active":"2024-03-28T18:42:31.269565+0000","last_peered":"2024-03-28T18:42:31.269565+0000","last_clean":"2024-03-28T18:42:31.269565+0000","last_became_active":"2024-03-28T18:41:30.221867+0000","last_became_peered":"2024-03-28T18:41:30.221867+0000","last_unstale":"2024-03-28T18:42:31.269565+0000","last_undegraded":"2024-03-28T18:42:31.269565+0000","last_fullsized":"2024-03-28T18:42:31.269565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:45.207508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168453+0000","last_change":"2024-03-28T18:41:30.507663+0000","last_active":"2024-03-28T18:42:31.168453+0000","last_peered":"2024-03-28T18:42:31.168453+0000","last_clean":"2024-03-28T18:42:31.168453+0000","last_became_active":"2024-03-28T18:41:30.507284+0000","last_became_peered":"2024-03-28T18:41:30.507284+0000","last_unstale":"2024-03-28T18:42:31.168453+0000","last_undegraded":"2024-03-28T18:42:31.168453+0000","last_fullsized":"2024-03-28T18:42:31.168453+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:03.045818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175018+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:42:31.175018+0000","last_peered":"2024-03-28T18:42:31.175018+0000","last_clean":"2024-03-28T18:42:31.175018+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:42:31.175018+0000","last_undegraded":"2024-03-28T18:42:31.175018+0000","last_fullsized":"2024-03-28T18:42:31.175018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403246+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:42:31.403246+0000","last_peered":"2024-03-28T18:42:31.403246+0000","last_clean":"2024-03-28T18:42:31.403246+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:42:31.403246+0000","last_undegraded":"2024-03-28T18:42:31.403246+0000","last_fullsized":"2024-03-28T18:42:31.403246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168025+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:42:31.168025+0000","last_peered":"2024-03-28T18:42:31.168025+0000","last_clean":"2024-03-28T18:42:31.168025+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:42:31.168025+0000","last_undegraded":"2024-03-28T18:42:31.168025+0000","last_fullsized":"2024-03-28T18:42:31.168025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172247+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:42:31.172247+0000","last_peered":"2024-03-28T18:42:31.172247+0000","last_clean":"2024-03-28T18:42:31.172247+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:42:31.172247+0000","last_undegraded":"2024-03-28T18:42:31.172247+0000","last_fullsized":"2024-03-28T18:42:31.172247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:47.286252+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:42:47.286252+0000","last_peered":"2024-03-28T18:42:47.286252+0000","last_clean":"2024-03-28T18:42:47.286252+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:42:47.286252+0000","last_undegraded":"2024-03-28T18:42:47.286252+0000","last_fullsized":"2024-03-28T18:42:47.286252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167797+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:42:31.167797+0000","last_peered":"2024-03-28T18:42:31.167797+0000","last_clean":"2024-03-28T18:42:31.167797+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:42:31.167797+0000","last_undegraded":"2024-03-28T18:42:31.167797+0000","last_fullsized":"2024-03-28T18:42:31.167797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360329+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:42:31.360329+0000","last_peered":"2024-03-28T18:42:31.360329+0000","last_clean":"2024-03-28T18:42:31.360329+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:42:31.360329+0000","last_undegraded":"2024-03-28T18:42:31.360329+0000","last_fullsized":"2024-03-28T18:42:31.360329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.183289+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:42:31.183289+0000","last_peered":"2024-03-28T18:42:31.183289+0000","last_clean":"2024-03-28T18:42:31.183289+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:42:31.183289+0000","last_undegraded":"2024-03-28T18:42:31.183289+0000","last_fullsized":"2024-03-28T18:42:31.183289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168630+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:42:31.168630+0000","last_peered":"2024-03-28T18:42:31.168630+0000","last_clean":"2024-03-28T18:42:31.168630+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:42:31.168630+0000","last_undegraded":"2024-03-28T18:42:31.168630+0000","last_fullsized":"2024-03-28T18:42:31.168630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.303127+0000","last_change":"2024-03-28T18:42:31.303127+0000","last_active":"2024-03-28T18:42:31.303127+0000","last_peered":"2024-03-28T18:42:31.303127+0000","last_clean":"2024-03-28T18:42:31.303127+0000","last_became_active":"2024-03-28T18:42:31.301321+0000","last_became_peered":"2024-03-28T18:42:31.301321+0000","last_unstale":"2024-03-28T18:42:31.303127+0000","last_undegraded":"2024-03-28T18:42:31.303127+0000","last_fullsized":"2024-03-28T18:42:31.303127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:08.892803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":223,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:47.286270+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:42:47.286270+0000","last_peered":"2024-03-28T18:42:47.286270+0000","last_clean":"2024-03-28T18:42:47.286270+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:42:47.286270+0000","last_undegraded":"2024-03-28T18:42:47.286270+0000","last_fullsized":"2024-03-28T18:42:47.286270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61701,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":35,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239136+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:42:31.239136+0000","last_peered":"2024-03-28T18:42:31.239136+0000","last_clean":"2024-03-28T18:42:31.239136+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:42:31.239136+0000","last_undegraded":"2024-03-28T18:42:31.239136+0000","last_fullsized":"2024-03-28T18:42:31.239136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317559+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:42:31.317559+0000","last_peered":"2024-03-28T18:42:31.317559+0000","last_clean":"2024-03-28T18:42:31.317559+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:42:31.317559+0000","last_undegraded":"2024-03-28T18:42:31.317559+0000","last_fullsized":"2024-03-28T18:42:31.317559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174870+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:42:31.174870+0000","last_peered":"2024-03-28T18:42:31.174870+0000","last_clean":"2024-03-28T18:42:31.174870+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:42:31.174870+0000","last_undegraded":"2024-03-28T18:42:31.174870+0000","last_fullsized":"2024-03-28T18:42:31.174870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168225+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:42:31.168225+0000","last_peered":"2024-03-28T18:42:31.168225+0000","last_clean":"2024-03-28T18:42:31.168225+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:42:31.168225+0000","last_undegraded":"2024-03-28T18:42:31.168225+0000","last_fullsized":"2024-03-28T18:42:31.168225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360547+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:42:31.360547+0000","last_peered":"2024-03-28T18:42:31.360547+0000","last_clean":"2024-03-28T18:42:31.360547+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:42:31.360547+0000","last_undegraded":"2024-03-28T18:42:31.360547+0000","last_fullsized":"2024-03-28T18:42:31.360547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":113,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:48.325212+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:42:48.325212+0000","last_peered":"2024-03-28T18:42:48.325212+0000","last_clean":"2024-03-28T18:42:48.325212+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:42:48.325212+0000","last_undegraded":"2024-03-28T18:42:48.325212+0000","last_fullsized":"2024-03-28T18:42:48.325212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360164+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:42:31.360164+0000","last_peered":"2024-03-28T18:42:31.360164+0000","last_clean":"2024-03-28T18:42:31.360164+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:42:31.360164+0000","last_undegraded":"2024-03-28T18:42:31.360164+0000","last_fullsized":"2024-03-28T18:42:31.360164+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238835+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:42:31.238835+0000","last_peered":"2024-03-28T18:42:31.238835+0000","last_clean":"2024-03-28T18:42:31.238835+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:42:31.238835+0000","last_undegraded":"2024-03-28T18:42:31.238835+0000","last_fullsized":"2024-03-28T18:42:31.238835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168124+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:42:31.168124+0000","last_peered":"2024-03-28T18:42:31.168124+0000","last_clean":"2024-03-28T18:42:31.168124+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:42:31.168124+0000","last_undegraded":"2024-03-28T18:42:31.168124+0000","last_fullsized":"2024-03-28T18:42:31.168124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239411+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:42:31.239411+0000","last_peered":"2024-03-28T18:42:31.239411+0000","last_clean":"2024-03-28T18:42:31.239411+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:42:31.239411+0000","last_undegraded":"2024-03-28T18:42:31.239411+0000","last_fullsized":"2024-03-28T18:42:31.239411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411308+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:42:31.411308+0000","last_peered":"2024-03-28T18:42:31.411308+0000","last_clean":"2024-03-28T18:42:31.411308+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:42:31.411308+0000","last_undegraded":"2024-03-28T18:42:31.411308+0000","last_fullsized":"2024-03-28T18:42:31.411308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.173845+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:42:31.173845+0000","last_peered":"2024-03-28T18:42:31.173845+0000","last_clean":"2024-03-28T18:42:31.173845+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:42:31.173845+0000","last_undegraded":"2024-03-28T18:42:31.173845+0000","last_fullsized":"2024-03-28T18:42:31.173845+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269901+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:42:31.269901+0000","last_peered":"2024-03-28T18:42:31.269901+0000","last_clean":"2024-03-28T18:42:31.269901+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:42:31.269901+0000","last_undegraded":"2024-03-28T18:42:31.269901+0000","last_fullsized":"2024-03-28T18:42:31.269901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.392452+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:42:45.392452+0000","last_peered":"2024-03-28T18:42:45.392452+0000","last_clean":"2024-03-28T18:42:45.392452+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:42:45.392452+0000","last_undegraded":"2024-03-28T18:42:45.392452+0000","last_fullsized":"2024-03-28T18:42:45.392452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269813+0000","last_change":"2024-03-28T18:41:30.219905+0000","last_active":"2024-03-28T18:42:31.269813+0000","last_peered":"2024-03-28T18:42:31.269813+0000","last_clean":"2024-03-28T18:42:31.269813+0000","last_became_active":"2024-03-28T18:41:30.219488+0000","last_became_peered":"2024-03-28T18:41:30.219488+0000","last_unstale":"2024-03-28T18:42:31.269813+0000","last_undegraded":"2024-03-28T18:42:31.269813+0000","last_fullsized":"2024-03-28T18:42:31.269813+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:16.108678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.320096+0000","last_change":"2024-03-28T18:42:31.320096+0000","last_active":"2024-03-28T18:42:31.320096+0000","last_peered":"2024-03-28T18:42:31.320096+0000","last_clean":"2024-03-28T18:42:31.320096+0000","last_became_active":"2024-03-28T18:42:31.318988+0000","last_became_peered":"2024-03-28T18:42:31.318988+0000","last_unstale":"2024-03-28T18:42:31.320096+0000","last_undegraded":"2024-03-28T18:42:31.320096+0000","last_fullsized":"2024-03-28T18:42:31.320096+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:50.019195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":108,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.319663+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:42:45.319663+0000","last_peered":"2024-03-28T18:42:45.319663+0000","last_clean":"2024-03-28T18:42:45.319663+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:42:45.319663+0000","last_undegraded":"2024-03-28T18:42:45.319663+0000","last_fullsized":"2024-03-28T18:42:45.319663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403866+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:42:31.403866+0000","last_peered":"2024-03-28T18:42:31.403866+0000","last_clean":"2024-03-28T18:42:31.403866+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:42:31.403866+0000","last_undegraded":"2024-03-28T18:42:31.403866+0000","last_fullsized":"2024-03-28T18:42:31.403866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299716+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:42:31.299716+0000","last_peered":"2024-03-28T18:42:31.299716+0000","last_clean":"2024-03-28T18:42:31.299716+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:42:31.299716+0000","last_undegraded":"2024-03-28T18:42:31.299716+0000","last_fullsized":"2024-03-28T18:42:31.299716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360313+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:42:31.360313+0000","last_peered":"2024-03-28T18:42:31.360313+0000","last_clean":"2024-03-28T18:42:31.360313+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:42:31.360313+0000","last_undegraded":"2024-03-28T18:42:31.360313+0000","last_fullsized":"2024-03-28T18:42:31.360313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174559+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:42:31.174559+0000","last_peered":"2024-03-28T18:42:31.174559+0000","last_clean":"2024-03-28T18:42:31.174559+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:42:31.174559+0000","last_undegraded":"2024-03-28T18:42:31.174559+0000","last_fullsized":"2024-03-28T18:42:31.174559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360338+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:42:31.360338+0000","last_peered":"2024-03-28T18:42:31.360338+0000","last_clean":"2024-03-28T18:42:31.360338+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:42:31.360338+0000","last_undegraded":"2024-03-28T18:42:31.360338+0000","last_fullsized":"2024-03-28T18:42:31.360338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411274+0000","last_change":"2024-03-28T18:41:30.227916+0000","last_active":"2024-03-28T18:42:31.411274+0000","last_peered":"2024-03-28T18:42:31.411274+0000","last_clean":"2024-03-28T18:42:31.411274+0000","last_became_active":"2024-03-28T18:41:30.218947+0000","last_became_peered":"2024-03-28T18:41:30.218947+0000","last_unstale":"2024-03-28T18:42:31.411274+0000","last_undegraded":"2024-03-28T18:42:31.411274+0000","last_fullsized":"2024-03-28T18:42:31.411274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:14.922324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175978+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:42:31.175978+0000","last_peered":"2024-03-28T18:42:31.175978+0000","last_clean":"2024-03-28T18:42:31.175978+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:42:31.175978+0000","last_undegraded":"2024-03-28T18:42:31.175978+0000","last_fullsized":"2024-03-28T18:42:31.175978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.172882+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:42:31.172882+0000","last_peered":"2024-03-28T18:42:31.172882+0000","last_clean":"2024-03-28T18:42:31.172882+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:42:31.172882+0000","last_undegraded":"2024-03-28T18:42:31.172882+0000","last_fullsized":"2024-03-28T18:42:31.172882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.405004+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:42:31.405004+0000","last_peered":"2024-03-28T18:42:31.405004+0000","last_clean":"2024-03-28T18:42:31.405004+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:42:31.405004+0000","last_undegraded":"2024-03-28T18:42:31.405004+0000","last_fullsized":"2024-03-28T18:42:31.405004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299800+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:42:31.299800+0000","last_peered":"2024-03-28T18:42:31.299800+0000","last_clean":"2024-03-28T18:42:31.299800+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:42:31.299800+0000","last_undegraded":"2024-03-28T18:42:31.299800+0000","last_fullsized":"2024-03-28T18:42:31.299800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.302339+0000","last_change":"2024-03-28T18:42:31.302339+0000","last_active":"2024-03-28T18:42:31.302339+0000","last_peered":"2024-03-28T18:42:31.302339+0000","last_clean":"2024-03-28T18:42:31.302339+0000","last_became_active":"2024-03-28T18:42:31.301324+0000","last_became_peered":"2024-03-28T18:42:31.301324+0000","last_unstale":"2024-03-28T18:42:31.302339+0000","last_undegraded":"2024-03-28T18:42:31.302339+0000","last_fullsized":"2024-03-28T18:42:31.302339+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:56.490969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167476+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:42:31.167476+0000","last_peered":"2024-03-28T18:42:31.167476+0000","last_clean":"2024-03-28T18:42:31.167476+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:42:31.167476+0000","last_undegraded":"2024-03-28T18:42:31.167476+0000","last_fullsized":"2024-03-28T18:42:31.167476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238841+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:42:31.238841+0000","last_peered":"2024-03-28T18:42:31.238841+0000","last_clean":"2024-03-28T18:42:31.238841+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:42:31.238841+0000","last_undegraded":"2024-03-28T18:42:31.238841+0000","last_fullsized":"2024-03-28T18:42:31.238841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403498+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:42:31.403498+0000","last_peered":"2024-03-28T18:42:31.403498+0000","last_clean":"2024-03-28T18:42:31.403498+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:42:31.403498+0000","last_undegraded":"2024-03-28T18:42:31.403498+0000","last_fullsized":"2024-03-28T18:42:31.403498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411114+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:42:31.411114+0000","last_peered":"2024-03-28T18:42:31.411114+0000","last_clean":"2024-03-28T18:42:31.411114+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:42:31.411114+0000","last_undegraded":"2024-03-28T18:42:31.411114+0000","last_fullsized":"2024-03-28T18:42:31.411114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317547+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:42:31.317547+0000","last_peered":"2024-03-28T18:42:31.317547+0000","last_clean":"2024-03-28T18:42:31.317547+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:42:31.317547+0000","last_undegraded":"2024-03-28T18:42:31.317547+0000","last_fullsized":"2024-03-28T18:42:31.317547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404069+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:42:31.404069+0000","last_peered":"2024-03-28T18:42:31.404069+0000","last_clean":"2024-03-28T18:42:31.404069+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:42:31.404069+0000","last_undegraded":"2024-03-28T18:42:31.404069+0000","last_fullsized":"2024-03-28T18:42:31.404069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238985+0000","last_change":"2024-03-28T18:41:30.319270+0000","last_active":"2024-03-28T18:42:31.238985+0000","last_peered":"2024-03-28T18:42:31.238985+0000","last_clean":"2024-03-28T18:42:31.238985+0000","last_became_active":"2024-03-28T18:41:30.311219+0000","last_became_peered":"2024-03-28T18:41:30.311219+0000","last_unstale":"2024-03-28T18:42:31.238985+0000","last_undegraded":"2024-03-28T18:42:31.238985+0000","last_fullsized":"2024-03-28T18:42:31.238985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:23.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360610+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:42:31.360610+0000","last_peered":"2024-03-28T18:42:31.360610+0000","last_clean":"2024-03-28T18:42:31.360610+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:42:31.360610+0000","last_undegraded":"2024-03-28T18:42:31.360610+0000","last_fullsized":"2024-03-28T18:42:31.360610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:45.340188+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:42:45.340188+0000","last_peered":"2024-03-28T18:42:45.340188+0000","last_clean":"2024-03-28T18:42:45.340188+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:42:45.340188+0000","last_undegraded":"2024-03-28T18:42:45.340188+0000","last_fullsized":"2024-03-28T18:42:45.340188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360460+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:42:31.360460+0000","last_peered":"2024-03-28T18:42:31.360460+0000","last_clean":"2024-03-28T18:42:31.360460+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:42:31.360460+0000","last_undegraded":"2024-03-28T18:42:31.360460+0000","last_fullsized":"2024-03-28T18:42:31.360460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168432+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:42:31.168432+0000","last_peered":"2024-03-28T18:42:31.168432+0000","last_clean":"2024-03-28T18:42:31.168432+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:42:31.168432+0000","last_undegraded":"2024-03-28T18:42:31.168432+0000","last_fullsized":"2024-03-28T18:42:31.168432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360345+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:42:31.360345+0000","last_peered":"2024-03-28T18:42:31.360345+0000","last_clean":"2024-03-28T18:42:31.360345+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:42:31.360345+0000","last_undegraded":"2024-03-28T18:42:31.360345+0000","last_fullsized":"2024-03-28T18:42:31.360345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.176185+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:42:31.176185+0000","last_peered":"2024-03-28T18:42:31.176185+0000","last_clean":"2024-03-28T18:42:31.176185+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:42:31.176185+0000","last_undegraded":"2024-03-28T18:42:31.176185+0000","last_fullsized":"2024-03-28T18:42:31.176185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360591+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:42:31.360591+0000","last_peered":"2024-03-28T18:42:31.360591+0000","last_clean":"2024-03-28T18:42:31.360591+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:42:31.360591+0000","last_undegraded":"2024-03-28T18:42:31.360591+0000","last_fullsized":"2024-03-28T18:42:31.360591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411179+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:42:31.411179+0000","last_peered":"2024-03-28T18:42:31.411179+0000","last_clean":"2024-03-28T18:42:31.411179+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:42:31.411179+0000","last_undegraded":"2024-03-28T18:42:31.411179+0000","last_fullsized":"2024-03-28T18:42:31.411179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317513+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:42:31.317513+0000","last_peered":"2024-03-28T18:42:31.317513+0000","last_clean":"2024-03-28T18:42:31.317513+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:42:31.317513+0000","last_undegraded":"2024-03-28T18:42:31.317513+0000","last_fullsized":"2024-03-28T18:42:31.317513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317445+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:42:31.317445+0000","last_peered":"2024-03-28T18:42:31.317445+0000","last_clean":"2024-03-28T18:42:31.317445+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:42:31.317445+0000","last_undegraded":"2024-03-28T18:42:31.317445+0000","last_fullsized":"2024-03-28T18:42:31.317445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.362520+0000","last_change":"2024-03-28T18:42:31.362520+0000","last_active":"2024-03-28T18:42:31.362520+0000","last_peered":"2024-03-28T18:42:31.362520+0000","last_clean":"2024-03-28T18:42:31.362520+0000","last_became_active":"2024-03-28T18:42:31.361563+0000","last_became_peered":"2024-03-28T18:42:31.361563+0000","last_unstale":"2024-03-28T18:42:31.362520+0000","last_undegraded":"2024-03-28T18:42:31.362520+0000","last_fullsized":"2024-03-28T18:42:31.362520+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:27.960042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175632+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:42:31.175632+0000","last_peered":"2024-03-28T18:42:31.175632+0000","last_clean":"2024-03-28T18:42:31.175632+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:42:31.175632+0000","last_undegraded":"2024-03-28T18:42:31.175632+0000","last_fullsized":"2024-03-28T18:42:31.175632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411422+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:42:31.411422+0000","last_peered":"2024-03-28T18:42:31.411422+0000","last_clean":"2024-03-28T18:42:31.411422+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:42:31.411422+0000","last_undegraded":"2024-03-28T18:42:31.411422+0000","last_fullsized":"2024-03-28T18:42:31.411422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317666+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:42:31.317666+0000","last_peered":"2024-03-28T18:42:31.317666+0000","last_clean":"2024-03-28T18:42:31.317666+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:42:31.317666+0000","last_undegraded":"2024-03-28T18:42:31.317666+0000","last_fullsized":"2024-03-28T18:42:31.317666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.404920+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:42:31.404920+0000","last_peered":"2024-03-28T18:42:31.404920+0000","last_clean":"2024-03-28T18:42:31.404920+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:42:31.404920+0000","last_undegraded":"2024-03-28T18:42:31.404920+0000","last_fullsized":"2024-03-28T18:42:31.404920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.174053+0000","last_change":"2024-03-28T18:41:30.510267+0000","last_active":"2024-03-28T18:42:31.174053+0000","last_peered":"2024-03-28T18:42:31.174053+0000","last_clean":"2024-03-28T18:42:31.174053+0000","last_became_active":"2024-03-28T18:41:30.508658+0000","last_became_peered":"2024-03-28T18:41:30.508658+0000","last_unstale":"2024-03-28T18:42:31.174053+0000","last_undegraded":"2024-03-28T18:42:31.174053+0000","last_fullsized":"2024-03-28T18:42:31.174053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:30.892433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403310+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:42:31.403310+0000","last_peered":"2024-03-28T18:42:31.403310+0000","last_clean":"2024-03-28T18:42:31.403310+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:42:31.403310+0000","last_undegraded":"2024-03-28T18:42:31.403310+0000","last_fullsized":"2024-03-28T18:42:31.403310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167733+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:42:31.167733+0000","last_peered":"2024-03-28T18:42:31.167733+0000","last_clean":"2024-03-28T18:42:31.167733+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:42:31.167733+0000","last_undegraded":"2024-03-28T18:42:31.167733+0000","last_fullsized":"2024-03-28T18:42:31.167733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175533+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:42:31.175533+0000","last_peered":"2024-03-28T18:42:31.175533+0000","last_clean":"2024-03-28T18:42:31.175533+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:42:31.175533+0000","last_undegraded":"2024-03-28T18:42:31.175533+0000","last_fullsized":"2024-03-28T18:42:31.175533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411254+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:42:31.411254+0000","last_peered":"2024-03-28T18:42:31.411254+0000","last_clean":"2024-03-28T18:42:31.411254+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:42:31.411254+0000","last_undegraded":"2024-03-28T18:42:31.411254+0000","last_fullsized":"2024-03-28T18:42:31.411254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68905,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":834,"num_read_kb":490,"num_write":240,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477349,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1138688,"data_stored":1130358,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":477167,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477433,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":1070454,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":476884,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":539498,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":478223,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":540109,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":476709,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888136,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":477718,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":1069159,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:42:53.940 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T18:42:53.940 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T18:42:53.940 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T18:42:53.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T18:42:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:54 smithi119 ceph-mon[31813]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:54.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:54 smithi119 ceph-mon[31813]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:54 smithi196 ceph-mon[36300]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:54.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:54 smithi196 ceph-mon[36300]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:54.439 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:42:54.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:54 smithi131 ceph-mon[36065]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:42:54.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:54 smithi131 ceph-mon[36065]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:42:56.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:56 smithi119 ceph-mon[31813]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:56.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:56 smithi196 ceph-mon[36300]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:56.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:56 smithi131 ceph-mon[36065]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 -- 172.21.15.119:0/2559650966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741096c0 msgr2=0x7fd274111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 --2- 172.21.15.119:0/2559650966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741096c0 0x7fd274111290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00b0d0 tx=0x7fd26c00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 -- 172.21.15.119:0/2559650966 shutdown_connections 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 --2- 172.21.15.119:0/2559650966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741096c0 0x7fd274111290 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 --2- 172.21.15.119:0/2559650966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2741045e0 0x7fd2741049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 --2- 172.21.15.119:0/2559650966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd274104f90 0x7fd274108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 -- 172.21.15.119:0/2559650966 >> 172.21.15.119:0/2559650966 conn(0x7fd2740ffe70 msgr2=0x7fd274102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:56.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 -- 172.21.15.119:0/2559650966 shutdown_connections 2024-03-28T18:42:56.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.653+0000 7fd27ac00700 1 -- 172.21.15.119:0/2559650966 wait complete. 2024-03-28T18:42:56.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.654+0000 7fd27ac00700 1 Processor -- start 2024-03-28T18:42:56.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.654+0000 7fd27ac00700 1 -- start start 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 0x7fd2741a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd274104f90 0x7fd2741a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 0x7fd27419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274114700 con 0x7fd274104f90 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274114580 con 0x7fd2741096c0 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd27ac00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274114880 con 0x7fd2741045e0 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd273e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 0x7fd2741a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd273e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 0x7fd2741a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48750/0 (socket says 172.21.15.119:48750) 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd278c00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 0x7fd27419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd273400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd274104f90 0x7fd2741a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:56.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd273e00700 1 -- 172.21.15.119:0/1249880123 learned_addr learned my addr 172.21.15.119:0/1249880123 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:56.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd278c00700 1 -- 172.21.15.119:0/1249880123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 msgr2=0x7fd2741a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:56.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd278c00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 0x7fd2741a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd278c00700 1 -- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd274104f90 msgr2=0x7fd2741a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:56.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.655+0000 7fd278c00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd274104f90 0x7fd2741a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd278c00700 1 -- 172.21.15.119:0/1249880123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27419b020 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd278c00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 0x7fd27419a8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00d010 tx=0x7fd26c009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c003a00 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd26c022070 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27419b2b0 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c01b6c0 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.656+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2741a75d0 con 0x7fd2741096c0 2024-03-28T18:42:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.657+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd274105ac0 con 0x7fd2741096c0 2024-03-28T18:42:56.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.659+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd26c003ba0 con 0x7fd2741096c0 2024-03-28T18:42:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.660+0000 7fd270c00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 0x7fd2580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.661+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fd26c0cfa10 con 0x7fd2741096c0 2024-03-28T18:42:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.661+0000 7fd273e00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 0x7fd2580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.661+0000 7fd273e00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 0x7fd2580a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd27419ba00 tx=0x7fd264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:56.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.664+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd26c09c900 con 0x7fd2741096c0 2024-03-28T18:42:56.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.930+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd2741a77a0 con 0x7fd2741096c0 2024-03-28T18:42:56.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.931+0000 7fd270c00700 1 -- 172.21.15.119:0/1249880123 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd26c09c200 con 0x7fd2741096c0 2024-03-28T18:42:56.931 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:42:56.931 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 msgr2=0x7fd2580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 0x7fd2580a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd27419ba00 tx=0x7fd264009260 comp rx=0 tx=0).stop 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 msgr2=0x7fd27419a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 0x7fd27419a8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00d010 tx=0x7fd26c009810 comp rx=0 tx=0).stop 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 shutdown_connections 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd2580a29a0 0x7fd2580a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd2741096c0 0x7fd27419a8c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd2741045e0 0x7fd2741a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 --2- 172.21.15.119:0/1249880123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd274104f90 0x7fd2741a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 >> 172.21.15.119:0/1249880123 conn(0x7fd2740ffe70 msgr2=0x7fd2741007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 shutdown_connections 2024-03-28T18:42:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:56.934+0000 7fd27ac00700 1 -- 172.21.15.119:0/1249880123 wait complete. 2024-03-28T18:42:57.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:57 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1249880123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:42:57.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:57 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1249880123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:42:57.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:57 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1249880123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T18:42:57.844 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T18:42:57.844 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 -- 172.21.15.119:0/1682354244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401005c0 msgr2=0x7f84401009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 --2- 172.21.15.119:0/1682354244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401005c0 0x7f84401009e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8430009900 tx=0x7f843002bc30 comp rx=0 tx=0).stop 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 -- 172.21.15.119:0/1682354244 shutdown_connections 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 --2- 172.21.15.119:0/1682354244 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8440102170 0x7f8440108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 --2- 172.21.15.119:0/1682354244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84401017b0 0x7f8440101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 --2- 172.21.15.119:0/1682354244 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401005c0 0x7f84401009e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 -- 172.21.15.119:0/1682354244 >> 172.21.15.119:0/1682354244 conn(0x7f84400fbb30 msgr2=0x7f84400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 -- 172.21.15.119:0/1682354244 shutdown_connections 2024-03-28T18:42:58.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.064+0000 7f8446e00640 1 -- 172.21.15.119:0/1682354244 wait complete. 2024-03-28T18:42:58.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.065+0000 7f8446e00640 1 Processor -- start 2024-03-28T18:42:58.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 -- start start 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401017b0 0x7f8440197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 0x7f844019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844010c290 con 0x7f8440102170 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844010c110 con 0x7f84401017b0 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8446e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844010c410 con 0x7f84401005c0 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f843fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f843fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48760/0 (socket says 172.21.15.119:48760) 2024-03-28T18:42:58.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f843fe00640 1 -- 172.21.15.119:0/11439725 learned_addr learned my addr 172.21.15.119:0/11439725 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f843f400640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401017b0 0x7f8440197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.066+0000 7f8444e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 0x7f844019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 -- 172.21.15.119:0/11439725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401017b0 msgr2=0x7f8440197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401017b0 0x7f8440197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 -- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 msgr2=0x7f844019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 0x7f844019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 -- 172.21.15.119:0/11439725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844019f640 con 0x7f84401005c0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f8444e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 0x7f844019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843fe00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8430000c00 tx=0x7f843002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430043020 con 0x7f84401005c0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84300054e0 con 0x7f84401005c0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844019f8d0 con 0x7f84401005c0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430003c40 con 0x7f84401005c0 2024-03-28T18:42:58.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.067+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844019fe20 con 0x7f84401005c0 2024-03-28T18:42:58.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.069+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844019fff0 con 0x7f84401005c0 2024-03-28T18:42:58.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.070+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8430046020 con 0x7f84401005c0 2024-03-28T18:42:58.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.072+0000 7f843cc00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 0x7f84240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.072+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f84300f0c10 con 0x7f84401005c0 2024-03-28T18:42:58.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.072+0000 7f843f400640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 0x7f84240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.073+0000 7f843f400640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 0x7f84240a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8440198f30 tx=0x7f8428005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.078+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f843004d020 con 0x7f84401005c0 2024-03-28T18:42:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.280+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84401a01c0 con 0x7f84401005c0 2024-03-28T18:42:58.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.283+0000 7f843cc00640 1 -- 172.21.15.119:0/11439725 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f84300bdb80 con 0x7f84401005c0 2024-03-28T18:42:58.283 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:42:58.283 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.285+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 msgr2=0x7f84240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.285+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 0x7f84240a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8440198f30 tx=0x7f8428005f90 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 msgr2=0x7f8440197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8430000c00 tx=0x7f843002c650 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 shutdown_connections 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84240a2890 0x7f84240a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8440102170 0x7f844019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84401017b0 0x7f8440197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 --2- 172.21.15.119:0/11439725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84401005c0 0x7f8440197a20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 >> 172.21.15.119:0/11439725 conn(0x7f84400fbb30 msgr2=0x7f84400fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 shutdown_connections 2024-03-28T18:42:58.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.286+0000 7f8446e00640 1 -- 172.21.15.119:0/11439725 wait complete. 2024-03-28T18:42:58.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:42:58.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:42:58.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:58 smithi119 ceph-mon[31813]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:58.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:58 smithi196 ceph-mon[36300]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:58.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:58 smithi131 ceph-mon[36065]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.523+0000 7f5f71800640 1 -- 172.21.15.119:0/1310424639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c077c40 msgr2=0x7f5f6c0784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.523+0000 7f5f71800640 1 --2- 172.21.15.119:0/1310424639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c077c40 0x7f5f6c0784e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009ca0 tx=0x7f5f6402c100 comp rx=0 tx=0).stop 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 -- 172.21.15.119:0/1310424639 shutdown_connections 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 --2- 172.21.15.119:0/1310424639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c077c40 0x7f5f6c0784e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 --2- 172.21.15.119:0/1310424639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 --2- 172.21.15.119:0/1310424639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c076060 0x7f5f6c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 -- 172.21.15.119:0/1310424639 >> 172.21.15.119:0/1310424639 conn(0x7f5f6c0fee50 msgr2=0x7f5f6c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 -- 172.21.15.119:0/1310424639 shutdown_connections 2024-03-28T18:42:58.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.524+0000 7f5f71800640 1 -- 172.21.15.119:0/1310424639 wait complete. 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.525+0000 7f5f71800640 1 Processor -- start 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.525+0000 7f5f71800640 1 -- start start 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.525+0000 7f5f71800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.525+0000 7f5f71800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f71800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c077c40 0x7f5f6c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f71800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113970 con 0x7f5f6c077c40 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f71800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1137f0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f71800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113af0 con 0x7f5f6c076060 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f6aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f6aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48788/0 (socket says 172.21.15.119:48788) 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f6aa00640 1 -- 172.21.15.119:0/3883294218 learned_addr learned my addr 172.21.15.119:0/3883294218 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f6b400640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c077c40 0x7f5f6c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 -- 172.21.15.119:0/3883294218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 msgr2=0x7f5f6c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 -- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c077c40 msgr2=0x7f5f6c1a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c077c40 0x7f5f6c1a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 -- 172.21.15.119:0/3883294218 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c1a5bc0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f6aa00640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.526+0000 7f5f62000640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c19e4e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400e9f0 tx=0x7f5f5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.527+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400ccc0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.527+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f54010910 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.527+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c1a5eb0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.527+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54018960 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.527+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1a63c0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.528+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c110250 con 0x7f5f6c0772a0 2024-03-28T18:42:58.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.530+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5f54010ac0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.531+0000 7f5f63e00640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 0x7f5f500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.531+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f5f540cef60 con 0x7f5f6c0772a0 2024-03-28T18:42:58.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.531+0000 7f5f6aa00640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 0x7f5f500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.532+0000 7f5f6aa00640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 0x7f5f500a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c011550 tx=0x7f5f5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:58.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.535+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f5409b7d0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.734+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f6c02d2c0 con 0x7f5f6c0772a0 2024-03-28T18:42:58.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.734+0000 7f5f63e00640 1 -- 172.21.15.119:0/3883294218 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f5409f870 con 0x7f5f6c0772a0 2024-03-28T18:42:58.734 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 msgr2=0x7f5f500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 0x7f5f500a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c011550 tx=0x7f5f5c005f90 comp rx=0 tx=0).stop 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 msgr2=0x7f5f6c19e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c19e4e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400e9f0 tx=0x7f5f5400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 shutdown_connections 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5f500a2910 0x7f5f500a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c077c40 0x7f5f6c1a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c0772a0 0x7f5f6c19e4e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 --2- 172.21.15.119:0/3883294218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c076060 0x7f5f6c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 >> 172.21.15.119:0/3883294218 conn(0x7f5f6c0fee50 msgr2=0x7f5f6c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 shutdown_connections 2024-03-28T18:42:58.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.737+0000 7f5f71800640 1 -- 172.21.15.119:0/3883294218 wait complete. 2024-03-28T18:42:58.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 -- 172.21.15.119:0/1301422650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc1015a0 msgr2=0x7f1adc1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/1301422650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc1015a0 0x7f1adc1019c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009900 tx=0x7f1acc02bc40 comp rx=0 tx=0).stop 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 -- 172.21.15.119:0/1301422650 shutdown_connections 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/1301422650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc103150 0x7f1adc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/1301422650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc102790 0x7f1adc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/1301422650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc1015a0 0x7f1adc1019c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 -- 172.21.15.119:0/1301422650 >> 172.21.15.119:0/1301422650 conn(0x7f1adc0fcb10 msgr2=0x7f1adc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 -- 172.21.15.119:0/1301422650 shutdown_connections 2024-03-28T18:42:58.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.996+0000 7f1ae0e00640 1 -- 172.21.15.119:0/1301422650 wait complete. 2024-03-28T18:42:58.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 Processor -- start 2024-03-28T18:42:58.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 -- start start 2024-03-28T18:42:58.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc102790 0x7f1adc197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 0x7f1adc19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc10d240 con 0x7f1adc102790 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc10d0c0 con 0x7f1adc1015a0 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1ae0e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc10d3c0 con 0x7f1adc103150 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.997+0000 7f1adb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55430/0 (socket says 172.21.15.119:55430) 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adaa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc102790 0x7f1adc197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 -- 172.21.15.119:0/2753692990 learned_addr learned my addr 172.21.15.119:0/2753692990 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 0x7f1adc19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 -- 172.21.15.119:0/2753692990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 msgr2=0x7f1adc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 0x7f1adc19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 -- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc102790 msgr2=0x7f1adc197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:58.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc102790 0x7f1adc197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 -- 172.21.15.119:0/2753692990 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc19f520 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adbe00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 0x7f1adc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1adb400640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1acc02efd0 tx=0x7f1acc02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc041070 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc19f7b0 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1acc004450 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.998+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1acc003d70 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:58.999+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc19fc70 con 0x7f1adc1015a0 2024-03-28T18:42:59.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.000+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc108990 con 0x7f1adc1015a0 2024-03-28T18:42:59.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.002+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1acc044070 con 0x7f1adc1015a0 2024-03-28T18:42:59.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.003+0000 7f1ad3e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 0x7f1ac00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.003+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f1acc0f0730 con 0x7f1adc1015a0 2024-03-28T18:42:59.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.003+0000 7f1adaa00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 0x7f1ac00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.004+0000 7f1adaa00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 0x7f1ac00a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80095f0 tx=0x7f1ac8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.007+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1acc0bd6a0 con 0x7f1adc1015a0 2024-03-28T18:42:59.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.205+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1adc0622a0 con 0x7f1adc1015a0 2024-03-28T18:42:59.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.205+0000 7f1ad3e00640 1 -- 172.21.15.119:0/2753692990 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1acc0bcfa0 con 0x7f1adc1015a0 2024-03-28T18:42:59.205 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 msgr2=0x7f1ac00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 0x7f1ac00a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80095f0 tx=0x7f1ac8005f90 comp rx=0 tx=0).stop 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 msgr2=0x7f1adc197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1acc02efd0 tx=0x7f1acc02c660 comp rx=0 tx=0).stop 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 shutdown_connections 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ac00a2890 0x7f1ac00a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1adc103150 0x7f1adc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1adc102790 0x7f1adc197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 --2- 172.21.15.119:0/2753692990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1adc1015a0 0x7f1adc197900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 >> 172.21.15.119:0/2753692990 conn(0x7f1adc0fcb10 msgr2=0x7f1adc105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 shutdown_connections 2024-03-28T18:42:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.208+0000 7f1ae0e00640 1 -- 172.21.15.119:0/2753692990 wait complete. 2024-03-28T18:42:59.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 -- 172.21.15.119:0/2717395653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78109870 msgr2=0x7fed78110100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 --2- 172.21.15.119:0/2717395653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78109870 0x7fed78110100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fed70009ca0 tx=0x7fed7002c100 comp rx=0 tx=0).stop 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 -- 172.21.15.119:0/2717395653 shutdown_connections 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 --2- 172.21.15.119:0/2717395653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78109870 0x7fed78110100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 --2- 172.21.15.119:0/2717395653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78108eb0 0x7fed78109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 --2- 172.21.15.119:0/2717395653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed781080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 -- 172.21.15.119:0/2717395653 >> 172.21.15.119:0/2717395653 conn(0x7fed78076c60 msgr2=0x7fed780790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 -- 172.21.15.119:0/2717395653 shutdown_connections 2024-03-28T18:42:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.434+0000 7fed7fe00640 1 -- 172.21.15.119:0/2717395653 wait complete. 2024-03-28T18:42:59.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/11439725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:42:59.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:42:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3883294218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:42:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.435+0000 7fed7fe00640 1 Processor -- start 2024-03-28T18:42:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.435+0000 7fed7fe00640 1 -- start start 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed7819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 0x7fed7819e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78109870 0x7fed781a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78113920 con 0x7fed78107cc0 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed781137a0 con 0x7fed78108eb0 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78113aa0 con 0x7fed78109870 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed7819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed7819df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56692/0 (socket says 172.21.15.119:56692) 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 0x7fed7819e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7d400640 1 -- 172.21.15.119:0/3446258505 learned_addr learned my addr 172.21.15.119:0/3446258505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7de00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78109870 0x7fed781a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 -- 172.21.15.119:0/3446258505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78109870 msgr2=0x7fed781a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78109870 0x7fed781a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 -- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 msgr2=0x7fed7819df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed7819df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.436+0000 7fed7ca00640 1 -- 172.21.15.119:0/3446258505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed781a5b80 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed7ca00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 0x7fed7819e4a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d6d0 tx=0x7fed6400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64017070 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed64009820 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64004d00 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed781a5e70 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.437+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed781a62e0 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.438+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7810ef80 con 0x7fed78108eb0 2024-03-28T18:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.441+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fed640099d0 con 0x7fed78108eb0 2024-03-28T18:42:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.442+0000 7fed76000640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 0x7fed600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.442+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fed640ce7c0 con 0x7fed78108eb0 2024-03-28T18:42:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.442+0000 7fed7d400640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 0x7fed600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.443+0000 7fed7d400640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 0x7fed600a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bc80 tx=0x7fed6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/11439725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:42:59.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:42:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3883294218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:42:59.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.446+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed6409afd0 con 0x7fed78108eb0 2024-03-28T18:42:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/11439725' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:42:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:42:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3883294218' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:42:59.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.644+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed7802d2b0 con 0x7fed78108eb0 2024-03-28T18:42:59.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.645+0000 7fed76000640 1 -- 172.21.15.119:0/3446258505 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed6409f070 con 0x7fed78108eb0 2024-03-28T18:42:59.645 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:42:59.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 msgr2=0x7fed600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 0x7fed600a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bc80 tx=0x7fed6c00b410 comp rx=0 tx=0).stop 2024-03-28T18:42:59.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 msgr2=0x7fed7819e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 0x7fed7819e4a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d6d0 tx=0x7fed6400db90 comp rx=0 tx=0).stop 2024-03-28T18:42:59.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 shutdown_connections 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fed600a2890 0x7fed600a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fed78109870 0x7fed781a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fed78108eb0 0x7fed7819e4a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 --2- 172.21.15.119:0/3446258505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed78107cc0 0x7fed7819df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 >> 172.21.15.119:0/3446258505 conn(0x7fed78076c60 msgr2=0x7fed7810c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.647+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 shutdown_connections 2024-03-28T18:42:59.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.648+0000 7fed7fe00640 1 -- 172.21.15.119:0/3446258505 wait complete. 2024-03-28T18:42:59.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.877+0000 7fe89fe00640 1 -- 172.21.15.119:0/1959389048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 msgr2=0x7fe898109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.877+0000 7fe89fe00640 1 --2- 172.21.15.119:0/1959389048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe898109350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe884009900 tx=0x7fe88402bc30 comp rx=0 tx=0).stop 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 -- 172.21.15.119:0/1959389048 shutdown_connections 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 --2- 172.21.15.119:0/1959389048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898109890 0x7fe898110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 --2- 172.21.15.119:0/1959389048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe898109350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 --2- 172.21.15.119:0/1959389048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898107ce0 0x7fe898108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 -- 172.21.15.119:0/1959389048 >> 172.21.15.119:0/1959389048 conn(0x7fe898076cb0 msgr2=0x7fe8980790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 -- 172.21.15.119:0/1959389048 shutdown_connections 2024-03-28T18:42:59.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.878+0000 7fe89fe00640 1 -- 172.21.15.119:0/1959389048 wait complete. 2024-03-28T18:42:59.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 Processor -- start 2024-03-28T18:42:59.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 -- start start 2024-03-28T18:42:59.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898107ce0 0x7fe89819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 0x7fe8981a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981139b0 con 0x7fe898109890 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898113830 con 0x7fe898107ce0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.879+0000 7fe89fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898113b30 con 0x7fe898108ed0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898107ce0 0x7fe89819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41290/0 (socket says 172.21.15.119:41290) 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 -- 172.21.15.119:0/876707595 learned_addr learned my addr 172.21.15.119:0/876707595 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89de00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 0x7fe8981a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 -- 172.21.15.119:0/876707595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898107ce0 msgr2=0x7fe89819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898107ce0 0x7fe89819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 -- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 msgr2=0x7fe8981a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 0x7fe8981a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 -- 172.21.15.119:0/876707595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8981a5be0 con 0x7fe898108ed0 2024-03-28T18:42:59.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89de00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 0x7fe8981a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.880+0000 7fe89ca00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe884005ab0 tx=0x7fe88402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.881+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884043020 con 0x7fe898108ed0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.881+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8840054e0 con 0x7fe898108ed0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.881+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884003ce0 con 0x7fe898108ed0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.881+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8981a5e70 con 0x7fe898108ed0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.881+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8981a6330 con 0x7fe898108ed0 2024-03-28T18:42:59.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.882+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89810efa0 con 0x7fe898108ed0 2024-03-28T18:42:59.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.884+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe884046020 con 0x7fe898108ed0 2024-03-28T18:42:59.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.885+0000 7fe896000640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 0x7fe8800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:42:59.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.885+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe8840f0d60 con 0x7fe898108ed0 2024-03-28T18:42:59.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.885+0000 7fe89d400640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 0x7fe8800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:42:59.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.886+0000 7fe89d400640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 0x7fe8800a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe898108d30 tx=0x7fe88c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:42:59.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:42:59.889+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe88404d020 con 0x7fe898108ed0 2024-03-28T18:43:00.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.108+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe8840b9c40 con 0x7fe898108ed0 2024-03-28T18:43:00.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.135+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89802d070 con 0x7fe898108ed0 2024-03-28T18:43:00.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.136+0000 7fe896000640 1 -- 172.21.15.119:0/876707595 <== mon.1 v2:172.21.15.196:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fe8840bdcd0 con 0x7fe898108ed0 2024-03-28T18:43:00.136 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:00.136 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:43:00.137 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:00.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 msgr2=0x7fe8800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 0x7fe8800a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe898108d30 tx=0x7fe88c00b390 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 msgr2=0x7fe89819e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe884005ab0 tx=0x7fe88402c6c0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 shutdown_connections 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe8800a2890 0x7fe8800a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe898109890 0x7fe8981a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe898108ed0 0x7fe89819e500 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 --2- 172.21.15.119:0/876707595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe898107ce0 0x7fe89819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 >> 172.21.15.119:0/876707595 conn(0x7fe898076cb0 msgr2=0x7fe89810c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 shutdown_connections 2024-03-28T18:43:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.139+0000 7fe89fe00640 1 -- 172.21.15.119:0/876707595 wait complete. 2024-03-28T18:43:00.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:00.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.360+0000 7f509ca00640 1 -- 172.21.15.119:0/3791572047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981036f0 msgr2=0x7f5098103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.360+0000 7f509ca00640 1 --2- 172.21.15.119:0/3791572047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981036f0 0x7f5098103b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5088009900 tx=0x7f508802bc50 comp rx=0 tx=0).stop 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 -- 172.21.15.119:0/3791572047 shutdown_connections 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 --2- 172.21.15.119:0/3791572047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981052a0 0x7f509810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 --2- 172.21.15.119:0/3791572047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 0x7f5098104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 --2- 172.21.15.119:0/3791572047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981036f0 0x7f5098103b10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 -- 172.21.15.119:0/3791572047 >> 172.21.15.119:0/3791572047 conn(0x7f50980fec80 msgr2=0x7f50981010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 -- 172.21.15.119:0/3791572047 shutdown_connections 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.361+0000 7f509ca00640 1 -- 172.21.15.119:0/3791572047 wait complete. 2024-03-28T18:43:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 Processor -- start 2024-03-28T18:43:00.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 -- start start 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 0x7f5098199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981052a0 0x7f50981a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.362+0000 7f509ca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509810f390 con 0x7f50981048e0 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f509ca00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509810f210 con 0x7f50981036f0 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f509ca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509810f510 con 0x7f50981052a0 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5095e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 0x7f5098199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55480/0 (socket says 172.21.15.119:55480) 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 -- 172.21.15.119:0/1953951159 learned_addr learned my addr 172.21.15.119:0/1953951159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5097200640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981052a0 0x7f50981a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 -- 172.21.15.119:0/1953951159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981052a0 msgr2=0x7f50981a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981052a0 0x7f50981a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 -- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 msgr2=0x7f5098199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 0x7f5098199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 -- 172.21.15.119:0/1953951159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50981a15d0 con 0x7f50981036f0 2024-03-28T18:43:00.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.363+0000 7f5096800640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f508802c260 tx=0x7f508802c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.364+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088041070 con 0x7f50981036f0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.364+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50981a1860 con 0x7f50981036f0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.364+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5088004450 con 0x7f50981036f0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.364+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088034440 con 0x7f50981036f0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.364+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50981a1db0 con 0x7f50981036f0 2024-03-28T18:43:00.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.365+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509810a9b0 con 0x7f50981036f0 2024-03-28T18:43:00.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.367+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5088044070 con 0x7f50981036f0 2024-03-28T18:43:00.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.368+0000 7f508f400640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 0x7f507c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.368+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f50880f05c0 con 0x7f50981036f0 2024-03-28T18:43:00.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.368+0000 7f5095e00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 0x7f507c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.369+0000 7f5095e00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 0x7f507c0a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f509819aec0 tx=0x7f5084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:00.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.372+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f50880bcf30 con 0x7f50981036f0 2024-03-28T18:43:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:00 smithi196 ceph-mon[36300]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:00 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2753692990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:00 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3446258505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:00 smithi131 ceph-mon[36065]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:00 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2753692990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:00 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3446258505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:00 smithi119 ceph-mon[31813]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:00.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:00 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2753692990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:00 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3446258505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.574+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50980622d0 con 0x7f50981036f0 2024-03-28T18:43:00.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.575+0000 7f508f400640 1 -- 172.21.15.119:0/1953951159 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f50880c0fd0 con 0x7f50981036f0 2024-03-28T18:43:00.576 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:00.576 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 msgr2=0x7f507c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 0x7f507c0a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f509819aec0 tx=0x7f5084005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 msgr2=0x7f50981999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f508802c260 tx=0x7f508802c670 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 shutdown_connections 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f507c0a2800 0x7f507c0a4cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50981052a0 0x7f50981a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50981048e0 0x7f5098199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 --2- 172.21.15.119:0/1953951159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f50981036f0 0x7f50981999b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.578+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 >> 172.21.15.119:0/1953951159 conn(0x7f50980fec80 msgr2=0x7f5098107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.579+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 shutdown_connections 2024-03-28T18:43:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.579+0000 7f509ca00640 1 -- 172.21.15.119:0/1953951159 wait complete. 2024-03-28T18:43:00.597 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:00.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 -- 172.21.15.119:0/2540349524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c81015a0 msgr2=0x7fd0c81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 --2- 172.21.15.119:0/2540349524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c81015a0 0x7fd0c81019c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009ca0 tx=0x7fd0c002c100 comp rx=0 tx=0).stop 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 -- 172.21.15.119:0/2540349524 shutdown_connections 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 --2- 172.21.15.119:0/2540349524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c8103150 0x7fd0c81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 --2- 172.21.15.119:0/2540349524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8102790 0x7fd0c8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 --2- 172.21.15.119:0/2540349524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c81015a0 0x7fd0c81019c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 -- 172.21.15.119:0/2540349524 >> 172.21.15.119:0/2540349524 conn(0x7fd0c80fcb10 msgr2=0x7fd0c80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:00.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 -- 172.21.15.119:0/2540349524 shutdown_connections 2024-03-28T18:43:00.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.806+0000 7fd0ce200640 1 -- 172.21.15.119:0/2540349524 wait complete. 2024-03-28T18:43:00.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.807+0000 7fd0ce200640 1 Processor -- start 2024-03-28T18:43:00.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 -- start start 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 0x7fd0c819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8103150 0x7fd0c81a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c810d230 con 0x7fd0c8103150 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c810d0b0 con 0x7fd0c81015a0 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0ce200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c810d3b0 con 0x7fd0c8102790 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0cce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0c7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 0x7fd0c819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0cce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55506/0 (socket says 172.21.15.119:55506) 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0cce00640 1 -- 172.21.15.119:0/126290614 learned_addr learned my addr 172.21.15.119:0/126290614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0cd800640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8103150 0x7fd0c81a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.808+0000 7fd0cce00640 1 -- 172.21.15.119:0/126290614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 msgr2=0x7fd0c819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0cce00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 0x7fd0c819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0cce00640 1 -- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8103150 msgr2=0x7fd0c81a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0cce00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8103150 0x7fd0c81a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0cce00640 1 -- 172.21.15.119:0/126290614 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c81a16e0 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0c7e00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 0x7fd0c819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0cce00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0000f80 tx=0x7fd0c0035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0039040 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c81a1970 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0c0044070 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0004430 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.809+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c81a1e30 con 0x7fd0c81015a0 2024-03-28T18:43:00.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.810+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c81a2060 con 0x7fd0c81015a0 2024-03-28T18:43:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.812+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd0c00045d0 con 0x7fd0c81015a0 2024-03-28T18:43:00.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.813+0000 7fd0c5600640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 0x7fd0ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:00.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.813+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fd0c00f0f90 con 0x7fd0c81015a0 2024-03-28T18:43:00.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.813+0000 7fd0c7e00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 0x7fd0ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:00.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.814+0000 7fd0c7e00640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 0x7fd0ac0a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd0c819afd0 tx=0x7fd0b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:00.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:00.817+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0c00bdfb0 con 0x7fd0c81015a0 2024-03-28T18:43:01.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.015+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0c81a2600 con 0x7fd0c81015a0 2024-03-28T18:43:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.015+0000 7fd0c5600640 1 -- 172.21.15.119:0/126290614 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0c00bd8b0 con 0x7fd0c81015a0 2024-03-28T18:43:01.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:01.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 msgr2=0x7fd0ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 0x7fd0ac0a4cc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd0c819afd0 tx=0x7fd0b0005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 msgr2=0x7fd0c8199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0000f80 tx=0x7fd0c0035d10 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 shutdown_connections 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd0ac0a2800 0x7fd0ac0a4cc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0c8103150 0x7fd0c81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd0c8102790 0x7fd0c819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 --2- 172.21.15.119:0/126290614 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd0c81015a0 0x7fd0c8199ac0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 >> 172.21.15.119:0/126290614 conn(0x7fd0c80fcb10 msgr2=0x7fd0c80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 shutdown_connections 2024-03-28T18:43:01.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.018+0000 7fd0ce200640 1 -- 172.21.15.119:0/126290614 wait complete. 2024-03-28T18:43:01.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 -- 172.21.15.119:0/3150185016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25781038c0 msgr2=0x7f2578103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 --2- 172.21.15.119:0/3150185016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25781038c0 0x7f2578103ce0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2568009900 tx=0x7f256802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 -- 172.21.15.119:0/3150185016 shutdown_connections 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 --2- 172.21.15.119:0/3150185016 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578105470 0x7f257810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 --2- 172.21.15.119:0/3150185016 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f2578104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 --2- 172.21.15.119:0/3150185016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25781038c0 0x7f2578103ce0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 -- 172.21.15.119:0/3150185016 >> 172.21.15.119:0/3150185016 conn(0x7f25780fee50 msgr2=0x7f2578101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 -- 172.21.15.119:0/3150185016 shutdown_connections 2024-03-28T18:43:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.239+0000 7f257ee00640 1 -- 172.21.15.119:0/3150185016 wait complete. 2024-03-28T18:43:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.240+0000 7f257ee00640 1 Processor -- start 2024-03-28T18:43:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.240+0000 7f257ee00640 1 -- start start 2024-03-28T18:43:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.240+0000 7f257ee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 0x7f2578199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ee00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810f580 con 0x7f25781038c0 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ee00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810f400 con 0x7f2578104ab0 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810f700 con 0x7f2578105470 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f2577e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 0x7f2578199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41334/0 (socket says 172.21.15.119:41334) 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55526/0 (socket says 172.21.15.119:55526) 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ce00640 1 -- 172.21.15.119:0/3124618712 learned_addr learned my addr 172.21.15.119:0/3124618712 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 -- 172.21.15.119:0/3124618712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 msgr2=0x7f25781a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 -- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 msgr2=0x7f2578199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 0x7f2578199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f256fe00640 1 -- 172.21.15.119:0/3124618712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25781a17e0 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f257ce00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.241+0000 7f2577e00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 0x7f2578199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f256fe00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9f0 tx=0x7f256000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256000ccc0 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2560005020 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600049e0 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25781a1ad0 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.242+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25781a2020 con 0x7f2578104ab0 2024-03-28T18:43:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.243+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257810abe0 con 0x7f2578104ab0 2024-03-28T18:43:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.245+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2560005560 con 0x7f2578104ab0 2024-03-28T18:43:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.246+0000 7f2575600640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 0x7f255c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.246+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f25600cea30 con 0x7f2578104ab0 2024-03-28T18:43:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.246+0000 7f2577e00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 0x7f255c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.247+0000 7f2577e00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 0x7f255c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f256802c170 tx=0x7f256802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:01.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.255+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f25600051d0 con 0x7f2578104ab0 2024-03-28T18:43:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:01 smithi119 ceph-mon[31813]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T18:43:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:01 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/876707595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:01 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1953951159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:01 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/126290614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:01.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:01 smithi196 ceph-mon[36300]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T18:43:01.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:01 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/876707595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:01 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1953951159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:01 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/126290614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:01 smithi131 ceph-mon[36065]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-28T18:43:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:01 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/876707595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:01 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1953951159' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:01.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:01 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/126290614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:01.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.455+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25780622a0 con 0x7f2578104ab0 2024-03-28T18:43:01.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.455+0000 7f2575600640 1 -- 172.21.15.119:0/3124618712 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f256009b9f0 con 0x7f2578104ab0 2024-03-28T18:43:01.455 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:01.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 msgr2=0x7f255c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 0x7f255c0a4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f256802c170 tx=0x7f256802df90 comp rx=0 tx=0).stop 2024-03-28T18:43:01.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 msgr2=0x7f257819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f256000e9f0 tx=0x7f256000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 shutdown_connections 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f255c0a2880 0x7f255c0a4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578105470 0x7f25781a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2578104ab0 0x7f257819a100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 --2- 172.21.15.119:0/3124618712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25781038c0 0x7f2578199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 >> 172.21.15.119:0/3124618712 conn(0x7f25780fee50 msgr2=0x7f2578107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 shutdown_connections 2024-03-28T18:43:01.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.458+0000 7f257ee00640 1 -- 172.21.15.119:0/3124618712 wait complete. 2024-03-28T18:43:01.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:01.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 -- 172.21.15.119:0/233494606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 msgr2=0x7fcdb8104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 --2- 172.21.15.119:0/233494606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb8104f50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009900 tx=0x7fcda002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 -- 172.21.15.119:0/233494606 shutdown_connections 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 --2- 172.21.15.119:0/233494606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb8105490 0x7fcdb810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 --2- 172.21.15.119:0/233494606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb8104f50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 --2- 172.21.15.119:0/233494606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb81038e0 0x7fcdb8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 -- 172.21.15.119:0/233494606 >> 172.21.15.119:0/233494606 conn(0x7fcdb80fee50 msgr2=0x7fcdb8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.684+0000 7fcdbd000640 1 -- 172.21.15.119:0/233494606 shutdown_connections 2024-03-28T18:43:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.685+0000 7fcdbd000640 1 -- 172.21.15.119:0/233494606 wait complete. 2024-03-28T18:43:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.685+0000 7fcdbd000640 1 Processor -- start 2024-03-28T18:43:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 -- start start 2024-03-28T18:43:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 0x7fcdb81a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb810f590 con 0x7fcdb8104ad0 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb810f410 con 0x7fcdb81038e0 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdbd000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb810f710 con 0x7fcdb8105490 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55550/0 (socket says 172.21.15.119:55550) 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 -- 172.21.15.119:0/996003800 learned_addr learned my addr 172.21.15.119:0/996003800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb6800640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 0x7fcdb81a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5400640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 -- 172.21.15.119:0/996003800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 msgr2=0x7fcdb81a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 0x7fcdb81a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 -- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 msgr2=0x7fcdb819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb5e00640 1 -- 172.21.15.119:0/996003800 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb81a17c0 con 0x7fcdb81038e0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.686+0000 7fcdb6800640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 0x7fcdb81a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdb5e00640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcda800ebc0 tx=0x7fcda800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda8003b70 con 0x7fcdb81038e0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb81a1ab0 con 0x7fcdb81038e0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcda8003d10 con 0x7fcdb81038e0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda8019980 con 0x7fcdb81038e0 2024-03-28T18:43:01.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.687+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb81a1fd0 con 0x7fcdb81038e0 2024-03-28T18:43:01.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.688+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80688e0 con 0x7fcdb81038e0 2024-03-28T18:43:01.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.690+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcda8004b60 con 0x7fcdb81038e0 2024-03-28T18:43:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.691+0000 7fcdaea00640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.691+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fcda80cf320 con 0x7fcdb81038e0 2024-03-28T18:43:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.691+0000 7fcdb5400640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.692+0000 7fcdb5400640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcda002c240 tx=0x7fcda002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:01.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.698+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcda809c2e0 con 0x7fcdb81038e0 2024-03-28T18:43:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.880+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcdb802d0a0 con 0x7fcdb81038e0 2024-03-28T18:43:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.881+0000 7fcdaea00640 1 -- 172.21.15.119:0/996003800 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcda809bbe0 con 0x7fcdb81038e0 2024-03-28T18:43:01.881 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.883+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 msgr2=0x7fcd9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.883+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcda002c240 tx=0x7fcda002df90 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.883+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 msgr2=0x7fcdb8199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcda800ebc0 tx=0x7fcda800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 shutdown_connections 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcdb8105490 0x7fcdb81a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdb8104ad0 0x7fcdb819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 --2- 172.21.15.119:0/996003800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcdb81038e0 0x7fcdb8199ba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 >> 172.21.15.119:0/996003800 conn(0x7fcdb80fee50 msgr2=0x7fcdb8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 shutdown_connections 2024-03-28T18:43:01.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:01.884+0000 7fcdbd000640 1 -- 172.21.15.119:0/996003800 wait complete. 2024-03-28T18:43:01.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:02.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 -- 172.21.15.119:0/3884857466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 msgr2=0x7f6f54106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 --2- 172.21.15.119:0/3884857466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f54106e60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009900 tx=0x7f6f4002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:02.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 -- 172.21.15.119:0/3884857466 shutdown_connections 2024-03-28T18:43:02.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 --2- 172.21.15.119:0/3884857466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 0x7f6f541097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 --2- 172.21.15.119:0/3884857466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f54106e60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 --2- 172.21.15.119:0/3884857466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 -- 172.21.15.119:0/3884857466 >> 172.21.15.119:0/3884857466 conn(0x7f6f540fbb30 msgr2=0x7f6f540fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 -- 172.21.15.119:0/3884857466 shutdown_connections 2024-03-28T18:43:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.113+0000 7f6f5c600640 1 -- 172.21.15.119:0/3884857466 wait complete. 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 Processor -- start 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 -- start start 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f541000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 0x7f6f54100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5410d380 con 0x7f6f54104a70 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5410d200 con 0x7f6f54102140 2024-03-28T18:43:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.114+0000 7f6f5c600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5410d500 con 0x7f6f541073a0 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f541000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55566/0 (socket says 172.21.15.119:55566) 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59c00640 1 -- 172.21.15.119:0/550376561 learned_addr learned my addr 172.21.15.119:0/550376561 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f5a600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 0x7f6f54100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 -- 172.21.15.119:0/550376561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 msgr2=0x7f6f54100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 0x7f6f54100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 -- 172.21.15.119:0/550376561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 msgr2=0x7f6f54101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 -- 172.21.15.119:0/550376561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f54100ec0 con 0x7f6f54104a70 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59c00640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.115+0000 7f6f59200640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f541000f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009230 tx=0x7f6f4002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.116+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40043020 con 0x7f6f54104a70 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.116+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f541a1b10 con 0x7f6f54104a70 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.116+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f400054e0 con 0x7f6f54104a70 2024-03-28T18:43:02.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.116+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f40003b60 con 0x7f6f54104a70 2024-03-28T18:43:02.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.116+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f541a2030 con 0x7f6f54104a70 2024-03-28T18:43:02.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.117+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f540fd620 con 0x7f6f54104a70 2024-03-28T18:43:02.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.119+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6f40046020 con 0x7f6f54104a70 2024-03-28T18:43:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.120+0000 7f6f52000640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 0x7f6f3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.120+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f6f400f0ad0 con 0x7f6f54104a70 2024-03-28T18:43:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.120+0000 7f6f59c00640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 0x7f6f3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.121+0000 7f6f59c00640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 0x7f6f3c0a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6f48011550 tx=0x7f6f48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:02.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.125+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6f4004d020 con 0x7f6f54104a70 2024-03-28T18:43:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:02 smithi119 ceph-mon[31813]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:02 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3124618712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:02 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/996003800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.358+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f5402d070 con 0x7f6f54104a70 2024-03-28T18:43:02.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.360+0000 7f6f52000640 1 -- 172.21.15.119:0/550376561 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f6f400bda90 con 0x7f6f54104a70 2024-03-28T18:43:02.360 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:02.360 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:43:02.361 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.362+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 msgr2=0x7f6f3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.362+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 0x7f6f3c0a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6f48011550 tx=0x7f6f48005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 msgr2=0x7f6f541000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f541000f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009230 tx=0x7f6f4002c5d0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 shutdown_connections 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f6f3c0a2800 0x7f6f3c0a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f541073a0 0x7f6f54100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f54104a70 0x7f6f541000f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 --2- 172.21.15.119:0/550376561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6f54102140 0x7f6f54101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 >> 172.21.15.119:0/550376561 conn(0x7f6f540fbb30 msgr2=0x7f6f54102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 shutdown_connections 2024-03-28T18:43:02.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.363+0000 7f6f5c600640 1 -- 172.21.15.119:0/550376561 wait complete. 2024-03-28T18:43:02.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:02 smithi196 ceph-mon[36300]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:02 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3124618712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:02 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/996003800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:02 smithi131 ceph-mon[36065]: pgmap v366: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:02 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3124618712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:02 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/996003800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.584+0000 7f84caa00640 1 -- 172.21.15.119:0/2647828161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41028b0 msgr2=0x7f84c4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.584+0000 7f84caa00640 1 --2- 172.21.15.119:0/2647828161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41028b0 0x7f84c4102d30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f84ac009900 tx=0x7f84ac02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 -- 172.21.15.119:0/2647828161 shutdown_connections 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 --2- 172.21.15.119:0/2647828161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c4103270 0x7f84c4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 --2- 172.21.15.119:0/2647828161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41028b0 0x7f84c4102d30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 --2- 172.21.15.119:0/2647828161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c41016c0 0x7f84c4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 -- 172.21.15.119:0/2647828161 >> 172.21.15.119:0/2647828161 conn(0x7f84c40fcc50 msgr2=0x7f84c40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 -- 172.21.15.119:0/2647828161 shutdown_connections 2024-03-28T18:43:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.585+0000 7f84caa00640 1 -- 172.21.15.119:0/2647828161 wait complete. 2024-03-28T18:43:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.586+0000 7f84caa00640 1 Processor -- start 2024-03-28T18:43:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.586+0000 7f84caa00640 1 -- start start 2024-03-28T18:43:02.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.586+0000 7f84caa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.586+0000 7f84caa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c41028b0 0x7f84c4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84caa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 0x7f84c419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84caa00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c410d370 con 0x7f84c4103270 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84caa00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c410d1f0 con 0x7f84c41016c0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84caa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c410d4f0 con 0x7f84c41028b0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55584/0 (socket says 172.21.15.119:55584) 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 -- 172.21.15.119:0/3098113128 learned_addr learned my addr 172.21.15.119:0/3098113128 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3400640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c41028b0 0x7f84c4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c8a00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 0x7f84c419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 -- 172.21.15.119:0/3098113128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c41028b0 msgr2=0x7f84c4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c41028b0 0x7f84c4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 -- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 msgr2=0x7f84c419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 0x7f84c419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 -- 172.21.15.119:0/3098113128 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c419f5b0 con 0x7f84c41016c0 2024-03-28T18:43:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c8a00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 0x7f84c419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.587+0000 7f84c3e00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009d20 tx=0x7f84b400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.588+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4010630 con 0x7f84c41016c0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.588+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b4010cb0 con 0x7f84c41016c0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.588+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4018de0 con 0x7f84c41016c0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.588+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c419f8a0 con 0x7f84c41016c0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.588+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c419fd70 con 0x7f84c41016c0 2024-03-28T18:43:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.589+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c41089e0 con 0x7f84c41016c0 2024-03-28T18:43:02.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.591+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f84b401f020 con 0x7f84c41016c0 2024-03-28T18:43:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.592+0000 7f84c0c00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 0x7f84a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.592+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f84b40cef40 con 0x7f84c41016c0 2024-03-28T18:43:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.592+0000 7f84c3400640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 0x7f84a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.593+0000 7f84c3400640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 0x7f84a80a4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f84ac02c240 tx=0x7f84ac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:02.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.599+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84b409bf00 con 0x7f84c41016c0 2024-03-28T18:43:02.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.786+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c402d230 con 0x7f84c41016c0 2024-03-28T18:43:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.789+0000 7f84c0c00640 1 -- 172.21.15.119:0/3098113128 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f84b409b800 con 0x7f84c41016c0 2024-03-28T18:43:02.789 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:02.789 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 msgr2=0x7f84a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 0x7f84a80a4d40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f84ac02c240 tx=0x7f84ac02df90 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 msgr2=0x7f84c4197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009d20 tx=0x7f84b400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 shutdown_connections 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f84a80a2880 0x7f84a80a4d40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c4103270 0x7f84c419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c41028b0 0x7f84c4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 --2- 172.21.15.119:0/3098113128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c41016c0 0x7f84c4197990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 >> 172.21.15.119:0/3098113128 conn(0x7f84c40fcc50 msgr2=0x7f84c4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 shutdown_connections 2024-03-28T18:43:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:02.792+0000 7f84caa00640 1 -- 172.21.15.119:0/3098113128 wait complete. 2024-03-28T18:43:02.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 -- 172.21.15.119:0/2193712938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 msgr2=0x7f98e0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 --2- 172.21.15.119:0/2193712938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0103d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009900 tx=0x7f98d002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 -- 172.21.15.119:0/2193712938 shutdown_connections 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 --2- 172.21.15.119:0/2193712938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0105490 0x7f98e010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 --2- 172.21.15.119:0/2193712938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0104ad0 0x7f98e0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 --2- 172.21.15.119:0/2193712938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0103d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 -- 172.21.15.119:0/2193712938 >> 172.21.15.119:0/2193712938 conn(0x7f98e00fee90 msgr2=0x7f98e01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.010+0000 7f98e5800640 1 -- 172.21.15.119:0/2193712938 shutdown_connections 2024-03-28T18:43:03.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.011+0000 7f98e5800640 1 -- 172.21.15.119:0/2193712938 wait complete. 2024-03-28T18:43:03.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 Processor -- start 2024-03-28T18:43:03.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 -- start start 2024-03-28T18:43:03.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0104ad0 0x7f98e019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 0x7f98e01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e010f580 con 0x7f98e0104ad0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e010f400 con 0x7f98e0105490 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98e5800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e010f700 con 0x7f98e01038e0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98dea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98df400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 0x7f98e01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98de000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0104ad0 0x7f98e019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98dea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41382/0 (socket says 172.21.15.119:41382) 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.012+0000 7f98dea00640 1 -- 172.21.15.119:0/3569299712 learned_addr learned my addr 172.21.15.119:0/3569299712 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 -- 172.21.15.119:0/3569299712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 msgr2=0x7f98e01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 0x7f98e01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 -- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0104ad0 msgr2=0x7f98e019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0104ad0 0x7f98e019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 -- 172.21.15.119:0/3569299712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e01a1810 con 0x7f98e01038e0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98df400640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 0x7f98e01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98dea00640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000c00 tx=0x7f98d002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0043020 con 0x7f98e01038e0 2024-03-28T18:43:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98d00054e0 con 0x7f98e01038e0 2024-03-28T18:43:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e01a1aa0 con 0x7f98e01038e0 2024-03-28T18:43:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.013+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0003c40 con 0x7f98e01038e0 2024-03-28T18:43:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.014+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e01a1ff0 con 0x7f98e01038e0 2024-03-28T18:43:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.014+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e010ac00 con 0x7f98e01038e0 2024-03-28T18:43:03.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.017+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f98d0046020 con 0x7f98e01038e0 2024-03-28T18:43:03.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.018+0000 7f98d7400640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 0x7f98c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.018+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f98d00f0f40 con 0x7f98e01038e0 2024-03-28T18:43:03.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.018+0000 7f98de000640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 0x7f98c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.019+0000 7f98de000640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 0x7f98c40a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98c80095f0 tx=0x7f98c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.020+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f98d004d020 con 0x7f98e01038e0 2024-03-28T18:43:03.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.242+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98e002d070 con 0x7f98e01038e0 2024-03-28T18:43:03.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.243+0000 7f98d7400640 1 -- 172.21.15.119:0/3569299712 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f98d00bdf00 con 0x7f98e01038e0 2024-03-28T18:43:03.243 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:03.244 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:43:03.244 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 msgr2=0x7f98c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 0x7f98c40a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98c80095f0 tx=0x7f98c8005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 msgr2=0x7f98e0199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000c00 tx=0x7f98d002c650 comp rx=0 tx=0).stop 2024-03-28T18:43:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 shutdown_connections 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f98c40a2800 0x7f98c40a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f98e0105490 0x7f98e01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e0104ad0 0x7f98e019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 --2- 172.21.15.119:0/3569299712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98e01038e0 0x7f98e0199bf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 >> 172.21.15.119:0/3569299712 conn(0x7f98e00fee90 msgr2=0x7f98e0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 shutdown_connections 2024-03-28T18:43:03.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.246+0000 7f98e5800640 1 -- 172.21.15.119:0/3569299712 wait complete. 2024-03-28T18:43:03.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:41:20.510917+0000', 'modified': '2024-03-28T18:41:28.190103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24563, 'mds_1': 24538, 'mds_2': 24517, 'mds_3': 24515, 'mds_4': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14448': {'gid': 14448, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6839/1834295684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 1834295684}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 1834295684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14460': {'gid': 14460, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6841/683166305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6840', 'nonce': 683166305}, {'type': 'v1', 'addr': '172.21.15.119:6841', 'nonce': 683166305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6833/4160748351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6832', 'nonce': 4160748351}, {'type': 'v1', 'addr': '172.21.15.131:6833', 'nonce': 4160748351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6833/1244574561', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6832', 'nonce': 1244574561}, {'type': 'v1', 'addr': '172.21.15.196:6833', 'nonce': 1244574561}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24485': {'gid': 24485, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6835/1538863538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6834', 'nonce': 1538863538}, {'type': 'v1', 'addr': '172.21.15.131:6835', 'nonce': 1538863538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24505': {'gid': 24505, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.196:6835/480911404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 480911404}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 480911404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.131:6837/2241845025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6836', 'nonce': 2241845025}, {'type': 'v1', 'addr': '172.21.15.131:6837', 'nonce': 2241845025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.196:6837/4226502296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 4226502296}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 4226502296}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.196:6839/2667663000', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2667663000}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2667663000}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24563': {'gid': 24563, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.131:6839/2855046426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6838', 'nonce': 2855046426}, {'type': 'v1', 'addr': '172.21.15.131:6839', 'nonce': 2855046426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24563, 'qdb_cluster': [24538, 24517, 24563, 24515, 24505]} 2024-03-28T18:43:03.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T18:43:03.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T18:43:03.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:03 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/550376561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:03 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3098113128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:03 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/550376561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:03 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3098113128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 -- 172.21.15.119:0/404406010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 msgr2=0x7f9ea0104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 --2- 172.21.15.119:0/404406010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea0104f50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009900 tx=0x7f9e8c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 -- 172.21.15.119:0/404406010 shutdown_connections 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 --2- 172.21.15.119:0/404406010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 --2- 172.21.15.119:0/404406010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea0104f50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.480+0000 7f9ea8800640 1 --2- 172.21.15.119:0/404406010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.481+0000 7f9ea8800640 1 -- 172.21.15.119:0/404406010 >> 172.21.15.119:0/404406010 conn(0x7f9ea00fee50 msgr2=0x7f9ea0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.481+0000 7f9ea8800640 1 -- 172.21.15.119:0/404406010 shutdown_connections 2024-03-28T18:43:03.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.481+0000 7f9ea8800640 1 -- 172.21.15.119:0/404406010 wait complete. 2024-03-28T18:43:03.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 Processor -- start 2024-03-28T18:43:03.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 -- start start 2024-03-28T18:43:03.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea010f5b0 con 0x7f9ea01038e0 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea010f430 con 0x7f9ea0105490 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea8800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea010f730 con 0x7f9ea0104ad0 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56802/0 (socket says 172.21.15.119:56802) 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea5e00640 1 -- 172.21.15.119:0/4139499376 learned_addr learned my addr 172.21.15.119:0/4139499376 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.482+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 -- 172.21.15.119:0/4139499376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 msgr2=0x7f9ea019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 -- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 msgr2=0x7f9ea0199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 -- 172.21.15.119:0/4139499376 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea01a1810 con 0x7f9ea0105490 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea6800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea01a11a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009fd0 tx=0x7f9e9c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9e9aa00640 1 -- 172.21.15.119:0/4139499376 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c00ce20 con 0x7f9ea0105490 2024-03-28T18:43:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea01a1b00 con 0x7f9ea0105490 2024-03-28T18:43:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9e9aa00640 1 -- 172.21.15.119:0/4139499376 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e9c005020 con 0x7f9ea0105490 2024-03-28T18:43:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9e9aa00640 1 -- 172.21.15.119:0/4139499376 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c010720 con 0x7f9ea0105490 2024-03-28T18:43:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.483+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea01a2070 con 0x7f9ea0105490 2024-03-28T18:43:03.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.487+0000 7f9eab42b740 1 Processor -- start 2024-03-28T18:43:03.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9e9aa00640 1 -- 172.21.15.119:0/4139499376 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9e9c005540 con 0x7f9ea0105490 2024-03-28T18:43:03.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 -- start start 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 0x564cda022fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda028700 0x564cda02ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cd9f87c90 con 0x564cda022ba0 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cd9f30980 con 0x564cda028700 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9eab42b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cd9f83d30 con 0x564cda023590 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41406/0 (socket says 172.21.15.119:41406) 2024-03-28T18:43:03.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9ea5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 0x564cda022fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.488+0000 7f9ea5e00640 1 -- 172.21.15.119:0/634410777 learned_addr learned my addr 172.21.15.119:0/634410777 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5400640 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda028700 0x564cda02ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 -- 172.21.15.119:0/634410777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda028700 msgr2=0x564cda02ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda028700 0x564cda02ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 -- 172.21.15.119:0/634410777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 msgr2=0x564cda022fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 0x564cda022fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 -- 172.21.15.119:0/634410777 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cd9f48510 con 0x564cda023590 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5400640 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 0x564cda022fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e940115f0 tx=0x7f9e94011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2b70da3d4e8a882 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9e93000640 1 -- 172.21.15.119:0/634410777 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400f6f0 con 0x564cda023590 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9e93000640 1 -- 172.21.15.119:0/634410777 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e9400fd70 con 0x564cda023590 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9e93000640 1 -- 172.21.15.119:0/634410777 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94019e60 con 0x564cda023590 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.489+0000 7f9e9aa00640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 0x7f9e880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9e9aa00640 1 -- 172.21.15.119:0/4139499376 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f9e9c0cf060 con 0x7f9ea0105490 2024-03-28T18:43:03.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 0x7f9e880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 -- 172.21.15.119:0/634410777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 msgr2=0x564cda028090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e940115f0 tx=0x7f9e94011ab0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 -- 172.21.15.119:0/634410777 shutdown_connections 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda028700 0x564cda02ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda028090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 --2- 172.21.15.119:0/634410777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda022ba0 0x564cda022fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 -- 172.21.15.119:0/634410777 >> 172.21.15.119:0/634410777 conn(0x564cda01e110 msgr2=0x564cda01e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 -- 172.21.15.119:0/634410777 shutdown_connections 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9eab42b740 1 -- 172.21.15.119:0/634410777 wait complete. 2024-03-28T18:43:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.490+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 0x7f9e880a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c02c240 tx=0x7f9e8c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 -- start start 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 Processor -- start 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda0345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda02b6b0 con 0x564cda028700 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cd9f43dd0 con 0x564cda022ba0 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda02b830 con 0x564cda023590 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.491+0000 7f9eab42b740 10 client.0 _ll_register_callbacks cb 0x564cd9c9bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41412/0 (socket says 172.21.15.119:41412) 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 -- 172.21.15.119:0/4253331606 learned_addr learned my addr 172.21.15.119:0/4253331606 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea6800640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda0345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 -- 172.21.15.119:0/4253331606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 msgr2=0x564cda02d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 -- 172.21.15.119:0/4253331606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 msgr2=0x564cda0345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda0345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 -- 172.21.15.119:0/4253331606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cd9c9c2d0 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9ea5400640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c000c00 tx=0x7f9e8c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=186d3f35f6ed4251 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9e87400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c053020 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f9e8c004380 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.492+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e9400fd70 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e8c004ed0 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 10 client.24994 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564cd9d666b0 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e9400fd70 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9e8c003bc0 con 0x564cda023590 2024-03-28T18:43:03.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 10 client.24994 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 10 client.24994 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 20 client.24994 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda0398e0 0x564cda038910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 4 client.24994 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564cd9f29520 con 0x564cda0398e0 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.493+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda0398e0 0x564cda038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.494+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda0398e0 0x564cda038910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=11c30dd872781df4 server_cookie=32c6a130d1e7628c in_seq=0 out_seq=0 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.494+0000 7f9e87400640 10 client.24994 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9e87400640 1 -- 172.21.15.119:0/4253331606 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564cd9f29520 con 0x564cda0398e0 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9e87400640 10 client.24994 handle_command_reply: tid=0 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 client.24994 shutdown 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 msgr2=0x564cda02d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c000c00 tx=0x7f9e8c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 shutdown_connections 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda0398e0 0x564cda038910 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda0345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda02d590 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 --2- 172.21.15.119:0/4253331606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9e87400640 0 client.24994 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 >> 172.21.15.119:0/4253331606 conn(0x564cda01e110 msgr2=0x564cda01d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 shutdown_connections 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 1 -- 172.21.15.119:0/4253331606 wait complete. 2024-03-28T18:43:03.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.495+0000 7f9eab42b740 20 client.24994 trim_cache size 0 max 16384 2024-03-28T18:43:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 Processor -- start 2024-03-28T18:43:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 -- start start 2024-03-28T18:43:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda034590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda02bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda039100 con 0x564cda028700 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda039760 con 0x564cda022ba0 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.513+0000 7f9eab42b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda039dc0 con 0x564cda023590 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda02bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda034590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55626/0 (socket says 172.21.15.119:55626) 2024-03-28T18:43:03.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 -- 172.21.15.119:0/3144480778 learned_addr learned my addr 172.21.15.119:0/3144480778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 -- 172.21.15.119:0/3144480778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 msgr2=0x564cda034590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda034590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 -- 172.21.15.119:0/3144480778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 msgr2=0x564cda02bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda02bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 -- 172.21.15.119:0/3144480778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cda01ee80 con 0x564cda022ba0 2024-03-28T18:43:03.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea6800640 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda02bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:03.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9e940115f0 tx=0x7f9e940094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d273a0f600d0620 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.514+0000 7f9e92600640 1 -- 172.21.15.119:0/3144480778 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94032070 con 0x564cda022ba0 2024-03-28T18:43:03.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9e92600640 1 -- 172.21.15.119:0/3144480778 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e94005380 con 0x564cda022ba0 2024-03-28T18:43:03.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9e92600640 1 -- 172.21.15.119:0/3144480778 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400f6c0 con 0x564cda022ba0 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 -- 172.21.15.119:0/3144480778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 msgr2=0x564cda02aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9e940115f0 tx=0x7f9e940094b0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 -- 172.21.15.119:0/3144480778 shutdown_connections 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda028700 0x564cda02bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda023590 0x564cda034590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 --2- 172.21.15.119:0/3144480778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda02aaf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 -- 172.21.15.119:0/3144480778 >> 172.21.15.119:0/3144480778 conn(0x564cda01e110 msgr2=0x564cda01e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 -- 172.21.15.119:0/3144480778 shutdown_connections 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.515+0000 7f9eab42b740 1 -- 172.21.15.119:0/3144480778 wait complete. 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 -- start start 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 Processor -- start 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 0x564cda030d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 0x564cda0337e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda03aa80 con 0x564cda023590 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda03a420 con 0x564cda022ba0 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cda03b0e0 con 0x564cda028700 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9eab42b740 10 client.0 _ll_register_callbacks cb 0x7f9e9c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9ea6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 0x564cda0337e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9ea5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 0x564cda030d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9ea5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55642/0 (socket says 172.21.15.119:55642) 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.516+0000 7f9ea5e00640 1 -- 172.21.15.119:0/2426576613 learned_addr learned my addr 172.21.15.119:0/2426576613 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 -- 172.21.15.119:0/2426576613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 msgr2=0x564cda0307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 -- 172.21.15.119:0/2426576613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 msgr2=0x564cda030d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 0x564cda030d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 -- 172.21.15.119:0/2426576613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cda033dd0 con 0x564cda028700 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:03.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea5400640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 0x564cda030d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9ea6800640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 0x564cda0337e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00c760 tx=0x7f9e9c09ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0838250b1b58e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9e87400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c0b8070 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f9e9c0a3c90 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e94005380 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e9c0a3e90 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9eab42b740 10 client.25000 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.517+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564cda094fe0 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e94005380 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9e9c09af60 con 0x564cda028700 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 10 client.25000 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 10 client.25000 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 20 client.25000 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda099690 0x564cda095320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 4 client.25000 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564cd9f951d0 con 0x564cda099690 2024-03-28T18:43:03.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.518+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda099690 0x564cda095320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9ea5e00640 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda099690 0x564cda095320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d584581071776d1 server_cookie=3b7d7dba699fd6a in_seq=0 out_seq=0 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9e87400640 10 client.25000 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9e87400640 1 -- 172.21.15.119:0/2426576613 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x564cd9f951d0 con 0x564cda099690 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9e87400640 10 client.25000 handle_command_reply: tid=0 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 client.25000 shutdown 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 msgr2=0x564cda0337e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 0x564cda0337e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00c760 tx=0x7f9e9c09ba40 comp rx=0 tx=0).stop 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 shutdown_connections 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x564cda099690 0x564cda095320 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564cda028700 0x564cda0337e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564cda023590 0x564cda030d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 --2- 172.21.15.119:0/2426576613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564cda022ba0 0x564cda0307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9e87400640 0 client.25000 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 >> 172.21.15.119:0/2426576613 conn(0x564cda01e110 msgr2=0x564cda038910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 shutdown_connections 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 1 -- 172.21.15.119:0/2426576613 wait complete. 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.519+0000 7f9eab42b740 20 client.25000 trim_cache size 0 max 16384 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 4, 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24505, 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 96.339059745, 2024-03-28T18:43:03.528 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 129.19474364499999 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 msgr2=0x7f9e880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 0x7f9e880a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c02c240 tx=0x7f9e8c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 msgr2=0x7f9ea01a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea01a11a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009fd0 tx=0x7f9e9c00ef10 comp rx=0 tx=0).stop 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 shutdown_connections 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f9e880a2880 0x7f9e880a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9ea0105490 0x7f9ea01a11a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ea0104ad0 0x7f9ea019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 --2- 172.21.15.119:0/4139499376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea01038e0 0x7f9ea0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 >> 172.21.15.119:0/4139499376 conn(0x7f9ea00fee50 msgr2=0x7f9ea0107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 shutdown_connections 2024-03-28T18:43:03.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.520+0000 7f9ea8800640 1 -- 172.21.15.119:0/4139499376 wait complete. 2024-03-28T18:43:03.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:03 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/550376561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.534 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:03 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3098113128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:03.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T18:43:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 -- 172.21.15.119:0/1681278663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084076060 msgr2=0x7f8084076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 --2- 172.21.15.119:0/1681278663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084076060 0x7f8084076480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8074009900 tx=0x7f807402bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 -- 172.21.15.119:0/1681278663 shutdown_connections 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 --2- 172.21.15.119:0/1681278663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8084077c40 0x7f80840784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 --2- 172.21.15.119:0/1681278663 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80840772a0 0x7f8084077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 --2- 172.21.15.119:0/1681278663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084076060 0x7f8084076480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 -- 172.21.15.119:0/1681278663 >> 172.21.15.119:0/1681278663 conn(0x7f80840fee50 msgr2=0x7f8084101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 -- 172.21.15.119:0/1681278663 shutdown_connections 2024-03-28T18:43:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.748+0000 7f808ae00640 1 -- 172.21.15.119:0/1681278663 wait complete. 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.749+0000 7f808ae00640 1 Processor -- start 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.749+0000 7f808ae00640 1 -- start start 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80840772a0 0x7f808419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084077c40 0x7f80841a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084113990 con 0x7f80840772a0 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084113810 con 0x7f8084076060 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f808ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8084113b10 con 0x7f8084077c40 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55656/0 (socket says 172.21.15.119:55656) 2024-03-28T18:43:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 -- 172.21.15.119:0/1602228609 learned_addr learned my addr 172.21.15.119:0/1602228609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083400640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80840772a0 0x7f808419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8088e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084077c40 0x7f80841a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 -- 172.21.15.119:0/1602228609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084077c40 msgr2=0x7f80841a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084077c40 0x7f80841a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 -- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80840772a0 msgr2=0x7f808419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80840772a0 0x7f808419e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.750+0000 7f8083e00640 1 -- 172.21.15.119:0/1602228609 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80841a5c40 con 0x7f8084076060 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f8083e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f807402c8e0 tx=0x7f807402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f8080c00640 1 -- 172.21.15.119:0/1602228609 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074043020 con 0x7f8084076060 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f8080c00640 1 -- 172.21.15.119:0/1602228609 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80740054e0 con 0x7f8084076060 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80841a5ed0 con 0x7f8084076060 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f8080c00640 1 -- 172.21.15.119:0/1602228609 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074003c70 con 0x7f8084076060 2024-03-28T18:43:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.751+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80841a6330 con 0x7f8084076060 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f8080c00640 1 -- 172.21.15.119:0/1602228609 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8074046020 con 0x7f8084076060 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 Processor -- start 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 -- start start 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb94f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb951b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb8aec90 con 0x55b5cb94f700 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb857980 con 0x55b5cb94a590 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.754+0000 7f808db91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb8aad30 con 0x55b5cb949ba0 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8083e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb94f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41454/0 (socket says 172.21.15.119:41454) 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8083400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb951b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 -- 172.21.15.119:0/2965297640 learned_addr learned my addr 172.21.15.119:0/2965297640 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 -- 172.21.15.119:0/2965297640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 msgr2=0x55b5cb94f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb94f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 -- 172.21.15.119:0/2965297640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 msgr2=0x55b5cb951b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb951b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 -- 172.21.15.119:0/2965297640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5cb86f510 con 0x55b5cb949ba0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8088e00640 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f807c009fd0 tx=0x7f807c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5a001bc81275131 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8080c00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 0x7f80680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.755+0000 7f8079600640 1 -- 172.21.15.119:0/2965297640 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c004d20 con 0x55b5cb949ba0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f8079600640 1 -- 172.21.15.119:0/2965297640 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f807c004050 con 0x55b5cb949ba0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f8079600640 1 -- 172.21.15.119:0/2965297640 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c005260 con 0x55b5cb949ba0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f8080c00640 1 -- 172.21.15.119:0/1602228609 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f80740f0be0 con 0x7f8084076060 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f8083e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 0x7f80680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 -- 172.21.15.119:0/2965297640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 msgr2=0x55b5cb949fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f807c009fd0 tx=0x7f807c0118d0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f8083e00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 0x7f80680a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80740bd430 tx=0x7f80740dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 -- 172.21.15.119:0/2965297640 shutdown_connections 2024-03-28T18:43:03.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb951b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb94f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 --2- 172.21.15.119:0/2965297640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb949fc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 -- 172.21.15.119:0/2965297640 >> 172.21.15.119:0/2965297640 conn(0x55b5cb945110 msgr2=0x55b5cb945510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 -- 172.21.15.119:0/2965297640 shutdown_connections 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.756+0000 7f808db91740 1 -- 172.21.15.119:0/2965297640 wait complete. 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 -- start start 2024-03-28T18:43:03.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 Processor -- start 2024-03-28T18:43:03.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 0x55b5cb95b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb9526b0 con 0x55b5cb94a590 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb86add0 con 0x55b5cb94f700 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb952830 con 0x55b5cb949ba0 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f808db91740 10 client.0 _ll_register_callbacks cb 0x55b5cb5c2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f8083400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.757+0000 7f8088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56884/0 (socket says 172.21.15.119:56884) 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 -- 172.21.15.119:0/3814181219 learned_addr learned my addr 172.21.15.119:0/3814181219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8088e00640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 0x55b5cb95b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 -- 172.21.15.119:0/3814181219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 msgr2=0x55b5cb9540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 -- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 msgr2=0x55b5cb95b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 0x55b5cb95b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 -- 172.21.15.119:0/3814181219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5cb5c32d0 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8088e00640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 0x55b5cb95b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f8083400640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8070012700 tx=0x7f8070012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d382e3691e37e0cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f806e000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807000f8c0 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f807000fa60 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.758+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f807c004050 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8070021020 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 10 client.14892 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b5cb68d6b0 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f807c004050 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8070026430 con 0x55b5cb94a590 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 10 client.14892 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 10 client.14892 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 20 client.14892 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb960970 0x55b5cb95f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 4 client.14892 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:03.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.759+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5cb850520 con 0x55b5cb960970 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.760+0000 7f8088e00640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb960970 0x55b5cb95f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.760+0000 7f8088e00640 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb960970 0x55b5cb95f9d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e122c57ddf436b6 server_cookie=82b252d407f0bee3 in_seq=0 out_seq=0 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.760+0000 7f806e000640 10 client.14892 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f806e000640 1 -- 172.21.15.119:0/3814181219 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b5cb850520 con 0x55b5cb960970 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f806e000640 10 client.14892 handle_command_reply: tid=0 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 client.14892 shutdown 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 msgr2=0x55b5cb9545f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8070012700 tx=0x7f8070012bc0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 shutdown_connections 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94f700 0x55b5cb95b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb960970 0x55b5cb95f9d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.761+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb9545f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f808db91740 1 --2- 172.21.15.119:0/3814181219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f806e000640 0 client.14892 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 >> 172.21.15.119:0/3814181219 conn(0x55b5cb945110 msgr2=0x55b5cb944310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 shutdown_connections 2024-03-28T18:43:03.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f808db91740 1 -- 172.21.15.119:0/3814181219 wait complete. 2024-03-28T18:43:03.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.762+0000 7f808db91740 20 client.14892 trim_cache size 0 max 16384 2024-03-28T18:43:03.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 Processor -- start 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 -- start start 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb95b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb953380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb960190 con 0x55b5cb94f700 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb9607f0 con 0x55b5cb94a590 2024-03-28T18:43:03.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f808db91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb960e50 con 0x55b5cb949ba0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f8088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f8088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41464/0 (socket says 172.21.15.119:41464) 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f8088e00640 1 -- 172.21.15.119:0/1344160747 learned_addr learned my addr 172.21.15.119:0/1344160747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f8083400640 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb95b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.780+0000 7f8088e00640 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb953380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 -- 172.21.15.119:0/1344160747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 msgr2=0x55b5cb95b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb95b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 -- 172.21.15.119:0/1344160747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 msgr2=0x55b5cb953380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb953380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 -- 172.21.15.119:0/1344160747 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5cb93f420 con 0x55b5cb949ba0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8088e00640 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f807c014040 tx=0x7f807c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90e1b2a55faab47f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8078c00640 1 -- 172.21.15.119:0/1344160747 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c004050 con 0x55b5cb949ba0 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8078c00640 1 -- 172.21.15.119:0/1344160747 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f807c0046d0 con 0x55b5cb949ba0 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.781+0000 7f8078c00640 1 -- 172.21.15.119:0/1344160747 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c02fb10 con 0x55b5cb949ba0 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- 172.21.15.119:0/1344160747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 msgr2=0x55b5cb9454f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f807c014040 tx=0x7f807c009500 comp rx=0 tx=0).stop 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- 172.21.15.119:0/1344160747 shutdown_connections 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94f700 0x55b5cb953380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb94a590 0x55b5cb95b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 --2- 172.21.15.119:0/1344160747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb949ba0 0x55b5cb9454f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- 172.21.15.119:0/1344160747 >> 172.21.15.119:0/1344160747 conn(0x55b5cb945110 msgr2=0x55b5cb93fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- 172.21.15.119:0/1344160747 shutdown_connections 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- 172.21.15.119:0/1344160747 wait complete. 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 -- start start 2024-03-28T18:43:03.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.782+0000 7f808db91740 1 Processor -- start 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb949ba0 0x55b5cb95b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 0x55b5cb958360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb961b10 con 0x55b5cb94a590 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb9614b0 con 0x55b5cb949ba0 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5cb962170 con 0x55b5cb94f700 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f808db91740 10 client.0 _ll_register_callbacks cb 0x7f8070009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56906/0 (socket says 172.21.15.119:56906) 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 -- 172.21.15.119:0/1209562061 learned_addr learned my addr 172.21.15.119:0/1209562061 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8088e00640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 0x55b5cb958360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8088e00640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb949ba0 0x55b5cb95b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 -- 172.21.15.119:0/1209562061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 msgr2=0x55b5cb958360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 0x55b5cb958360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 -- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb949ba0 msgr2=0x55b5cb95b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb949ba0 0x55b5cb95b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8083400640 1 -- 172.21.15.119:0/1209562061 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5cb945e80 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.783+0000 7f8088e00640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 0x55b5cb958360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f8083400640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80700267d0 tx=0x7f8070027d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81a8126e7c3a805 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f806e000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80700052c0 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f807c0046d0 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f8070005460 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8070037050 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f808db91740 10 client.14898 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b5cb9bc650 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.784+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f807c0046d0 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8070003820 con 0x55b5cb94a590 2024-03-28T18:43:03.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 10 client.14898 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 10 client.14898 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 20 client.14898 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb9c0c40 0x55b5cb9bc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 4 client.14898 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5cb8bc1d0 con 0x55b5cb9c0c40 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f8088e00640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb9c0c40 0x55b5cb9bc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f8088e00640 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb9c0c40 0x55b5cb9bc8e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=37a22991902634ec server_cookie=f6a71d8dd0720430 in_seq=0 out_seq=0 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.785+0000 7f806e000640 10 client.14898 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f806e000640 1 -- 172.21.15.119:0/1209562061 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b5cb8bc1d0 con 0x55b5cb9c0c40 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f806e000640 10 client.14898 handle_command_reply: tid=0 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 client.14898 shutdown 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 msgr2=0x55b5cb957e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f80700267d0 tx=0x7f8070027d00 comp rx=0 tx=0).stop 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 shutdown_connections 2024-03-28T18:43:03.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5cb94f700 0x55b5cb958360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55b5cb9c0c40 0x55b5cb9bc8e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b5cb94a590 0x55b5cb957e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 --2- 172.21.15.119:0/1209562061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b5cb949ba0 0x55b5cb95b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f806e000640 0 client.14898 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 >> 172.21.15.119:0/1209562061 conn(0x55b5cb945110 msgr2=0x55b5cb9454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 shutdown_connections 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 1 -- 172.21.15.119:0/1209562061 wait complete. 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.786+0000 7f808db91740 20 client.14898 trim_cache size 0 max 16384 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:03.790 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 3, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24515, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 97.626857576000006, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 123.966133386 2024-03-28T18:43:03.791 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 msgr2=0x7f80680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 0x7f80680a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80740bd430 tx=0x7f80740dd040 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 msgr2=0x7f808419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f807402c8e0 tx=0x7f807402c650 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 shutdown_connections 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80680a2800 0x7f80680a4cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8084077c40 0x7f80841a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80840772a0 0x7f808419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 --2- 172.21.15.119:0/1602228609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8084076060 0x7f808419e020 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 >> 172.21.15.119:0/1602228609 conn(0x7f80840fee50 msgr2=0x7f8084100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 shutdown_connections 2024-03-28T18:43:03.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:03.787+0000 7f808ae00640 1 -- 172.21.15.119:0/1602228609 wait complete. 2024-03-28T18:43:03.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T18:43:04.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 -- 172.21.15.119:0/3902673363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c301038e0 msgr2=0x7f0c30103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 --2- 172.21.15.119:0/3902673363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c301038e0 0x7f0c30103d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009900 tx=0x7f0c2002bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 -- 172.21.15.119:0/3902673363 shutdown_connections 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 --2- 172.21.15.119:0/3902673363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c3010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 --2- 172.21.15.119:0/3902673363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c30104ad0 0x7f0c30104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 --2- 172.21.15.119:0/3902673363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c301038e0 0x7f0c30103d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 -- 172.21.15.119:0/3902673363 >> 172.21.15.119:0/3902673363 conn(0x7f0c300fee50 msgr2=0x7f0c30101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 -- 172.21.15.119:0/3902673363 shutdown_connections 2024-03-28T18:43:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.007+0000 7f0c35000640 1 -- 172.21.15.119:0/3902673363 wait complete. 2024-03-28T18:43:04.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 Processor -- start 2024-03-28T18:43:04.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 -- start start 2024-03-28T18:43:04.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 0x7f0c3019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c301a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3010f5a0 con 0x7f0c30105490 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3010f420 con 0x7f0c301038e0 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c35000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3010f720 con 0x7f0c30104ad0 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c2de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 0x7f0c3019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c2de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55716/0 (socket says 172.21.15.119:55716) 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c2de00640 1 -- 172.21.15.119:0/3301775522 learned_addr learned my addr 172.21.15.119:0/3301775522 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.008+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c301a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 -- 172.21.15.119:0/3301775522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 msgr2=0x7f0c3019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 0x7f0c3019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 -- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 msgr2=0x7f0c301a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c301a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 -- 172.21.15.119:0/3301775522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c301a1860 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c25400640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 0x7f0c3019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c301a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c2de00640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c2002efd0 tx=0x7f0c2002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c27400640 1 -- 172.21.15.119:0/3301775522 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20041070 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c301a1af0 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c27400640 1 -- 172.21.15.119:0/3301775522 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c20004450 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c27400640 1 -- 172.21.15.119:0/3301775522 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20003e50 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.009+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c301a2000 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.012+0000 7f0c37c73740 1 Processor -- start 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.012+0000 7f0c27400640 1 -- 172.21.15.119:0/3301775522 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0c20044070 con 0x7f0c301038e0 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.012+0000 7f0c37c73740 1 -- start start 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 0x5578b70b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b7013c90 con 0x5578b70af590 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b6fbc980 con 0x5578b70b4700 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c37c73740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b700fd30 con 0x5578b70aeba0 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 0x5578b70b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 0x5578b70b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56920/0 (socket says 172.21.15.119:56920) 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41482/0 (socket says 172.21.15.119:41482) 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2de00640 1 -- 172.21.15.119:0/2661796221 learned_addr learned my addr 172.21.15.119:0/2661796221 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c25400640 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 -- 172.21.15.119:0/2661796221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 msgr2=0x5578b70b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 -- 172.21.15.119:0/2661796221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 msgr2=0x5578b70b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 0x5578b70b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.013+0000 7f0c2e800640 1 -- 172.21.15.119:0/2661796221 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578b6fd4510 con 0x5578b70aeba0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c2e800640 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005fd0 tx=0x7f0c28011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7f924df0be9161 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c27400640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 0x7f0c140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c1f400640 1 -- 172.21.15.119:0/2661796221 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28004d20 con 0x5578b70aeba0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c1f400640 1 -- 172.21.15.119:0/2661796221 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c2800f9a0 con 0x5578b70aeba0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c1f400640 1 -- 172.21.15.119:0/2661796221 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28004510 con 0x5578b70aeba0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c2de00640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 0x7f0c140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.014+0000 7f0c27400640 1 -- 172.21.15.119:0/3301775522 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0c200f0730 con 0x7f0c301038e0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- 172.21.15.119:0/2661796221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 msgr2=0x5578b70aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0c28005fd0 tx=0x7f0c28011bf0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c2de00640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 0x7f0c140a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0c200ee150 tx=0x7f0c200f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- 172.21.15.119:0/2661796221 shutdown_connections 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70af590 0x5578b70b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 --2- 172.21.15.119:0/2661796221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70aeba0 0x5578b70aefc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- 172.21.15.119:0/2661796221 >> 172.21.15.119:0/2661796221 conn(0x5578b70aa110 msgr2=0x5578b70aa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- 172.21.15.119:0/2661796221 shutdown_connections 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- 172.21.15.119:0/2661796221 wait complete. 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 -- start start 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.015+0000 7f0c37c73740 1 Processor -- start 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70b9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70c0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70b76b0 con 0x5578b70aeba0 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b6fcfdd0 con 0x5578b70b4700 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70b7830 con 0x5578b70af590 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c37c73740 10 client.0 _ll_register_callbacks cb 0x5578b6d27ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41494/0 (socket says 172.21.15.119:41494) 2024-03-28T18:43:04.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 -- 172.21.15.119:0/3977455488 learned_addr learned my addr 172.21.15.119:0/3977455488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c2e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 -- 172.21.15.119:0/3977455488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 msgr2=0x5578b70c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 -- 172.21.15.119:0/3977455488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 msgr2=0x5578b70b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c25400640 1 -- 172.21.15.119:0/3977455488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578b6d282d0 con 0x5578b70af590 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.016+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c25400640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0c18012d20 tx=0x7f0c1800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b04a10b2dc0d3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c13e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18012110 con 0x5578b70af590 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0c18004030 con 0x5578b70af590 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c2800f9a0 con 0x5578b70af590 2024-03-28T18:43:04.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c18021030 con 0x5578b70af590 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c37c73740 10 client.25030 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5578b6df26b0 con 0x5578b70af590 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.017+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c2800f9a0 con 0x5578b70af590 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0c18026430 con 0x5578b70af590 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 10 client.25030 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 10 client.25030 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 20 client.25030 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b70c5a30 0x5578b70c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 4 client.25030 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578b6fb5520 con 0x5578b70c5a30 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.018+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b70c5a30 0x5578b70c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.019+0000 7f0c13e00640 10 client.25030 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.019+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b70c5a30 0x5578b70c4ab0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4692f66e9a97166e server_cookie=1b72a3d998fc48d3 in_seq=0 out_seq=0 2024-03-28T18:43:04.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.019+0000 7f0c13e00640 1 -- 172.21.15.119:0/3977455488 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578b6fb5520 con 0x5578b70c5a30 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.019+0000 7f0c13e00640 10 client.25030 handle_command_reply: tid=0 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.019+0000 7f0c37c73740 1 client.25030 shutdown 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 msgr2=0x5578b70b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0c18012d20 tx=0x7f0c1800f790 comp rx=0 tx=0).stop 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 shutdown_connections 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b70c5a30 0x5578b70c4ab0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70b9650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c13e00640 0 client.25030 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3977455488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 >> 172.21.15.119:0/3977455488 conn(0x5578b70aa110 msgr2=0x5578b70a9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 shutdown_connections 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 1 -- 172.21.15.119:0/3977455488 wait complete. 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.020+0000 7f0c37c73740 20 client.25030 trim_cache size 0 max 16384 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.037+0000 7f0c37c73740 1 Processor -- start 2024-03-28T18:43:04.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.037+0000 7f0c37c73740 1 -- start start 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70aeba0 0x5578b70b95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 0x5578b70c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c5250 con 0x5578b70b4700 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c58b0 con 0x5578b70aeba0 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c37c73740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c5f10 con 0x5578b70af590 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41506/0 (socket says 172.21.15.119:41506) 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 -- 172.21.15.119:0/938360082 learned_addr learned my addr 172.21.15.119:0/938360082 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c2e800640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70aeba0 0x5578b70b95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c2e800640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 0x5578b70c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 -- 172.21.15.119:0/938360082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70aeba0 msgr2=0x5578b70b95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70aeba0 0x5578b70b95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 -- 172.21.15.119:0/938360082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 msgr2=0x5578b70c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 0x5578b70c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 -- 172.21.15.119:0/938360082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578b70ba100 con 0x5578b70af590 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c2e800640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 0x5578b70c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.038+0000 7f0c25400640 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0c18001eb0 tx=0x7f0c1800fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a102348bab89452b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c1ea00640 1 -- 172.21.15.119:0/938360082 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1802a030 con 0x5578b70af590 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c1ea00640 1 -- 172.21.15.119:0/938360082 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c18005590 con 0x5578b70af590 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c1ea00640 1 -- 172.21.15.119:0/938360082 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c180035f0 con 0x5578b70af590 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 -- 172.21.15.119:0/938360082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 msgr2=0x5578b70c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0c18001eb0 tx=0x7f0c1800fc90 comp rx=0 tx=0).stop 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 -- 172.21.15.119:0/938360082 shutdown_connections 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70b4700 0x5578b70c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70c0650 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 --2- 172.21.15.119:0/938360082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70aeba0 0x5578b70b95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 -- 172.21.15.119:0/938360082 >> 172.21.15.119:0/938360082 conn(0x5578b70aa110 msgr2=0x5578b70b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 -- 172.21.15.119:0/938360082 shutdown_connections 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.039+0000 7f0c37c73740 1 -- 172.21.15.119:0/938360082 wait complete. 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 -- start start 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 Processor -- start 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70bcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c6bd0 con 0x5578b70aeba0 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c6570 con 0x5578b70b4700 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578b70c7230 con 0x5578b70af590 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c37c73740 10 client.0 _ll_register_callbacks cb 0x7f0c18009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c2e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70bcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c2e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56930/0 (socket says 172.21.15.119:56930) 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.040+0000 7f0c2e800640 1 -- 172.21.15.119:0/3984926320 learned_addr learned my addr 172.21.15.119:0/3984926320 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 -- 172.21.15.119:0/3984926320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 msgr2=0x5578b70bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70bfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 -- 172.21.15.119:0/3984926320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 msgr2=0x5578b70bc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 -- 172.21.15.119:0/3984926320 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578b70c0170 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c25400640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70bcb60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0c18026470 tx=0x7f0c18012a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb195de410639d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c13e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1802a030 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c18005590 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0c18018030 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.041+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c18005110 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 10 client.25036 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5578b7129b80 con 0x5578b70af590 2024-03-28T18:43:04.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c18005590 con 0x5578b70af590 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0c18010430 con 0x5578b70af590 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 10 client.25036 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 10 client.25036 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 20 client.25036 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b712e010 0x5578b7129e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 4 client.25036 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5578b70211d0 con 0x5578b712e010 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.042+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b712e010 0x5578b7129e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c2e800640 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b712e010 0x5578b7129e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b6f798f796ffcf6 server_cookie=18ca76bcc54964b7 in_seq=0 out_seq=0 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c13e00640 10 client.25036 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c13e00640 1 -- 172.21.15.119:0/3984926320 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5578b70211d0 con 0x5578b712e010 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c13e00640 10 client.25036 handle_command_reply: tid=0 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 client.25036 shutdown 2024-03-28T18:43:04.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 msgr2=0x5578b70bcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70bcb60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0c18026470 tx=0x7f0c18012a00 comp rx=0 tx=0).stop 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 shutdown_connections 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5578b712e010 0x5578b7129e10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578b70b4700 0x5578b70bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578b70af590 0x5578b70bcb60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 --2- 172.21.15.119:0/3984926320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578b70aeba0 0x5578b70bc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c13e00640 0 client.25036 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 >> 172.21.15.119:0/3984926320 conn(0x5578b70aa110 msgr2=0x5578b70c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 shutdown_connections 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 1 -- 172.21.15.119:0/3984926320 wait complete. 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.043+0000 7f0c37c73740 20 client.25036 trim_cache size 0 max 16384 2024-03-28T18:43:04.075 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 2, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24517, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 98.897954077999998, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 126.880591236 2024-03-28T18:43:04.076 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:04.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 msgr2=0x7f0c140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 0x7f0c140a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0c200ee150 tx=0x7f0c200f6530 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 msgr2=0x7f0c30199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0c2002efd0 tx=0x7f0c2002c660 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 shutdown_connections 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0c140a29a0 0x7f0c140a4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c30105490 0x7f0c301a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0c30104ad0 0x7f0c3019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 --2- 172.21.15.119:0/3301775522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c301038e0 0x7f0c30199cf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 >> 172.21.15.119:0/3301775522 conn(0x7f0c300fee50 msgr2=0x7f0c30107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 shutdown_connections 2024-03-28T18:43:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.044+0000 7f0c35000640 1 -- 172.21.15.119:0/3301775522 wait complete. 2024-03-28T18:43:04.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 -- 172.21.15.119:0/532164733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0108eb0 msgr2=0x7fcfe0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 --2- 172.21.15.119:0/532164733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0108eb0 0x7fcfe0109330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 -- 172.21.15.119:0/532164733 shutdown_connections 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 --2- 172.21.15.119:0/532164733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 0x7fcfe0110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 --2- 172.21.15.119:0/532164733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0108eb0 0x7fcfe0109330 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 --2- 172.21.15.119:0/532164733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0107cc0 0x7fcfe01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 -- 172.21.15.119:0/532164733 >> 172.21.15.119:0/532164733 conn(0x7fcfe0076c60 msgr2=0x7fcfe00790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 -- 172.21.15.119:0/532164733 shutdown_connections 2024-03-28T18:43:04.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.281+0000 7fcfe6200640 1 -- 172.21.15.119:0/532164733 wait complete. 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.282+0000 7fcfe6200640 1 Processor -- start 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.282+0000 7fcfe6200640 1 -- start start 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.282+0000 7fcfe6200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfe6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0108eb0 0x7fcfe019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfe6200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 0x7fcfe01a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfe6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0113970 con 0x7fcfe0108eb0 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfe6200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe01137f0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfe6200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0113af0 con 0x7fcfe0109870 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55776/0 (socket says 172.21.15.119:55776) 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 -- 172.21.15.119:0/1551815842 learned_addr learned my addr 172.21.15.119:0/1551815842 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 0x7fcfe01a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdea00640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0108eb0 0x7fcfe019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 -- 172.21.15.119:0/1551815842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 msgr2=0x7fcfe01a5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 0x7fcfe01a5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 -- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0108eb0 msgr2=0x7fcfe019e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0108eb0 0x7fcfe019e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 -- 172.21.15.119:0/1551815842 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe01a5bb0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.283+0000 7fcfdf400640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9c0 tx=0x7fcfd000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.284+0000 7fcfd7e00640 1 -- 172.21.15.119:0/1551815842 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000cd80 con 0x7fcfe0107cc0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.284+0000 7fcfd7e00640 1 -- 172.21.15.119:0/1551815842 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd00035f0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.284+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe01a5ea0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.284+0000 7fcfd7e00640 1 -- 172.21.15.119:0/1551815842 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0010690 con 0x7fcfe0107cc0 2024-03-28T18:43:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.284+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe01a6310 con 0x7fcfe0107cc0 2024-03-28T18:43:04.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 Processor -- start 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfd7e00640 1 -- 172.21.15.119:0/1551815842 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcfd00037a0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 -- start start 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb0590 0x55b88deb5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88de14c90 con 0x55b88deb5700 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88ddbd980 con 0x55b88deafba0 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88de10d30 con 0x55b88deb0590 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.287+0000 7fcfdfe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb0590 0x55b88deb5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55790/0 (socket says 172.21.15.119:55790) 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 -- 172.21.15.119:0/2838180003 learned_addr learned my addr 172.21.15.119:0/2838180003 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdea00640 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 -- 172.21.15.119:0/2838180003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb0590 msgr2=0x55b88deb5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb0590 0x55b88deb5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 -- 172.21.15.119:0/2838180003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 msgr2=0x55b88deb7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 -- 172.21.15.119:0/2838180003 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b88ddd5510 con 0x55b88deafba0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009fd0 tx=0x7fcfd80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf36c5d4ed387a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfd7e00640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfcbe00640 1 -- 172.21.15.119:0/2838180003 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8004d20 con 0x55b88deafba0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfcbe00640 1 -- 172.21.15.119:0/2838180003 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd8004050 con 0x55b88deafba0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.288+0000 7fcfcbe00640 1 -- 172.21.15.119:0/2838180003 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8005260 con 0x55b88deafba0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfd7e00640 1 -- 172.21.15.119:0/1551815842 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fcfd00cedf0 con 0x7fcfe0107cc0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfdf400640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/2838180003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 msgr2=0x55b88deaffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009fd0 tx=0x7fcfd80118d0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfdf400640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 0x7fcfc40a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00ccbe0 tx=0x7fcfd00d28d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/2838180003 shutdown_connections 2024-03-28T18:43:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb0590 0x55b88deb5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/2838180003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deafba0 0x55b88deaffc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/2838180003 >> 172.21.15.119:0/2838180003 conn(0x55b88deab110 msgr2=0x55b88deab510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/2838180003 shutdown_connections 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.289+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/2838180003 wait complete. 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 -- start start 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 Processor -- start 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deafba0 0x55b88deba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 0x55b88dec1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88deb86b0 con 0x55b88deafba0 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88ddd0dd0 con 0x55b88deb0590 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88deb8830 con 0x55b88deb5700 2024-03-28T18:43:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.290+0000 7fcfe8f7e740 10 client.0 _ll_register_callbacks cb 0x55b88db28ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55794/0 (socket says 172.21.15.119:55794) 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 -- 172.21.15.119:0/607941537 learned_addr learned my addr 172.21.15.119:0/607941537 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deafba0 0x55b88deba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 0x55b88dec1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 -- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 msgr2=0x55b88dec1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 0x55b88dec1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 -- 172.21.15.119:0/607941537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deafba0 msgr2=0x55b88deba000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deafba0 0x55b88deba000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 -- 172.21.15.119:0/607941537 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b88db292d0 con 0x55b88deb0590 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 0x55b88dec1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfdea00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000c00 tx=0x7fcfcc004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88c8617c13d0f9b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.291+0000 7fcfc8c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005d50 con 0x55b88deb0590 2024-03-28T18:43:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fcfcc0046c0 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfd8004050 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfcc050020 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 10 client.25028 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b88dbf36b0 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd8004050 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcfcc0035f0 con 0x55b88deb0590 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 10 client.25028 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 10 client.25028 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 20 client.25028 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88dec6760 0x55b88dec5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 4 client.25028 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.292+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b88ddb6520 con 0x55b88dec6760 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.293+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88dec6760 0x55b88dec5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.293+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88dec6760 0x55b88dec5810 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ad9a11cc05a7483b server_cookie=7e3dcb2f9cd8159d in_seq=0 out_seq=0 2024-03-28T18:43:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.293+0000 7fcfc8c00640 10 client.25028 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:04.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfc8c00640 1 -- 172.21.15.119:0/607941537 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b88ddb6520 con 0x55b88dec6760 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfc8c00640 10 client.25028 handle_command_reply: tid=0 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 client.25028 shutdown 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 msgr2=0x55b88deba540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000c00 tx=0x7fcfcc004250 comp rx=0 tx=0).stop 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 shutdown_connections 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88dec6760 0x55b88dec5810 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deb5700 0x55b88dec1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88deba540 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/607941537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deafba0 0x55b88deba000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfc8c00640 0 client.25028 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.294+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 >> 172.21.15.119:0/607941537 conn(0x55b88deab110 msgr2=0x55b88deaa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.295+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 shutdown_connections 2024-03-28T18:43:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.295+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/607941537 wait complete. 2024-03-28T18:43:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.295+0000 7fcfe8f7e740 20 client.25028 trim_cache size 0 max 16384 2024-03-28T18:43:04.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 Processor -- start 2024-03-28T18:43:04.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 -- start start 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88deba4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec5f80 con 0x55b88deb5700 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec65e0 con 0x55b88deb0590 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.313+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec6c40 con 0x55b88deafba0 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55800/0 (socket says 172.21.15.119:55800) 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 -- 172.21.15.119:0/3527952335 learned_addr learned my addr 172.21.15.119:0/3527952335 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88deba4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 -- 172.21.15.119:0/3527952335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 msgr2=0x55b88deba4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88deba4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 -- 172.21.15.119:0/3527952335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 msgr2=0x55b88deb7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 -- 172.21.15.119:0/3527952335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b88deb0e20 con 0x55b88deb0590 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88deba4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0039c0 tx=0x7fcfcc009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd2e150dc435fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfcb400640 1 -- 172.21.15.119:0/3527952335 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005d50 con 0x55b88deb0590 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfcb400640 1 -- 172.21.15.119:0/3527952335 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc004ba0 con 0x55b88deb0590 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.314+0000 7fcfcb400640 1 -- 172.21.15.119:0/3527952335 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005110 con 0x55b88deb0590 2024-03-28T18:43:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3527952335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 msgr2=0x55b88dec1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0039c0 tx=0x7fcfcc009620 comp rx=0 tx=0).stop 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3527952335 shutdown_connections 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb5700 0x55b88deb7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb0590 0x55b88dec1540 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3527952335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88deba4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3527952335 >> 172.21.15.119:0/3527952335 conn(0x55b88deab110 msgr2=0x55b88dec5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3527952335 shutdown_connections 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.315+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3527952335 wait complete. 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 -- start start 2024-03-28T18:43:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 Processor -- start 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88dec1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb5700 0x55b88debe230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec7900 con 0x55b88deb0590 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec72a0 con 0x55b88deb5700 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b88dec7f60 con 0x55b88deafba0 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfe8f7e740 10 client.0 _ll_register_callbacks cb 0x7fcfcc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfdfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88dec1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfdea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57008/0 (socket says 172.21.15.119:57008) 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.316+0000 7fcfdea00640 1 -- 172.21.15.119:0/3670330737 learned_addr learned my addr 172.21.15.119:0/3670330737 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb5700 0x55b88debe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 -- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 msgr2=0x55b88dec1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88dec1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 -- 172.21.15.119:0/3670330737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb5700 msgr2=0x55b88debe230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb5700 0x55b88debe230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 -- 172.21.15.119:0/3670330737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b88debe7a0 con 0x55b88deb0590 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88dec1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfdea00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc004650 tx=0x7fcfcc004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6aec0f4bb7a0a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfc8c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:43:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc062020 con 0x55b88deb0590 2024-03-28T18:43:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fcfcc0052c0 con 0x55b88deb0590 2024-03-28T18:43:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfcc004ba0 con 0x55b88deb0590 2024-03-28T18:43:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfcc06d050 con 0x55b88deb0590 2024-03-28T18:43:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.317+0000 7fcfe8f7e740 10 client.14934 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b88df227b0 con 0x55b88deb0590 2024-03-28T18:43:04.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:04 smithi119 ceph-mon[31813]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T18:43:04.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:04 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3569299712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc004ba0 con 0x55b88deb0590 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcfcc04b960 con 0x55b88deb0590 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 10 client.14934 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 10 client.14934 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 20 client.14934 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88df26a40 0x55b88df229e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 4 client.14934 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.318+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b88de221d0 con 0x55b88df26a40 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88df26a40 0x55b88df229e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfdfe00640 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88df26a40 0x55b88df229e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a6d835ccde203a6 server_cookie=d7642b0004f8db6d in_seq=0 out_seq=0 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfc8c00640 10 client.14934 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfc8c00640 1 -- 172.21.15.119:0/3670330737 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b88de221d0 con 0x55b88df26a40 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfc8c00640 10 client.14934 handle_command_reply: tid=0 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 client.14934 shutdown 2024-03-28T18:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 msgr2=0x55b88debdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc004650 tx=0x7fcfcc004950 comp rx=0 tx=0).stop 2024-03-28T18:43:04.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 shutdown_connections 2024-03-28T18:43:04.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b88df26a40 0x55b88df229e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b88deb5700 0x55b88debe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b88deb0590 0x55b88debdcf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 --2- 172.21.15.119:0/3670330737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b88deafba0 0x55b88dec1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfc8c00640 0 client.14934 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.319+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 >> 172.21.15.119:0/3670330737 conn(0x55b88deab110 msgr2=0x55b88dea5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 shutdown_connections 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe8f7e740 1 -- 172.21.15.119:0/3670330737 wait complete. 2024-03-28T18:43:04.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe8f7e740 20 client.14934 trim_cache size 0 max 16384 2024-03-28T18:43:04.331 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:04.331 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:04.331 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 1, 2024-03-28T18:43:04.331 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24538, 2024-03-28T18:43:04.331 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 100.18475951400001, 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 121.594648014 2024-03-28T18:43:04.332 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:04.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 msgr2=0x7fcfc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 0x7fcfc40a4d40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00ccbe0 tx=0x7fcfd00d28d0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 msgr2=0x7fcfe019df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000e9c0 tx=0x7fcfd000ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 shutdown_connections 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fcfc40a2880 0x7fcfc40a4d40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcfe0109870 0x7fcfe01a5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfe0108eb0 0x7fcfe019e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 --2- 172.21.15.119:0/1551815842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcfe0107cc0 0x7fcfe019df90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.320+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 >> 172.21.15.119:0/1551815842 conn(0x7fcfe0076c60 msgr2=0x7fcfe010c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.321+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 shutdown_connections 2024-03-28T18:43:04.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.321+0000 7fcfe6200640 1 -- 172.21.15.119:0/1551815842 wait complete. 2024-03-28T18:43:04.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:43:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:04 smithi196 ceph-mon[36300]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T18:43:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:04 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3569299712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:04 smithi131 ceph-mon[36065]: pgmap v367: 97 pgs: 97 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T18:43:04.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:04 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3569299712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.538+0000 7ff6f2a00640 1 -- 172.21.15.119:0/1162818237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec108eb0 msgr2=0x7ff6ec109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.538+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/1162818237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec109330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009900 tx=0x7ff6d802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.538+0000 7ff6f2a00640 1 -- 172.21.15.119:0/1162818237 shutdown_connections 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.538+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/1162818237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec109870 0x7ff6ec110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.539+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/1162818237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec109330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.539+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/1162818237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.539+0000 7ff6f2a00640 1 -- 172.21.15.119:0/1162818237 >> 172.21.15.119:0/1162818237 conn(0x7ff6ec076c60 msgr2=0x7ff6ec0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.539+0000 7ff6f2a00640 1 -- 172.21.15.119:0/1162818237 shutdown_connections 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.539+0000 7ff6f2a00640 1 -- 172.21.15.119:0/1162818237 wait complete. 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 Processor -- start 2024-03-28T18:43:04.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 -- start start 2024-03-28T18:43:04.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec113970 con 0x7ff6ec108eb0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec1137f0 con 0x7ff6ec109870 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.540+0000 7ff6f2a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec113af0 con 0x7ff6ec107cc0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55832/0 (socket says 172.21.15.119:55832) 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 -- 172.21.15.119:0/2290489505 learned_addr learned my addr 172.21.15.119:0/2290489505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec19e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 -- 172.21.15.119:0/2290489505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 msgr2=0x7ff6ec19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 -- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec108eb0 msgr2=0x7ff6ec19e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec19e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 -- 172.21.15.119:0/2290489505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec1a5bb0 con 0x7ff6ec109870 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.541+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400eb10 tx=0x7ff6e400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.542+0000 7ff6e8c00640 1 -- 172.21.15.119:0/2290489505 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4004b20 con 0x7ff6ec109870 2024-03-28T18:43:04.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.542+0000 7ff6e8c00640 1 -- 172.21.15.119:0/2290489505 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e4004cc0 con 0x7ff6ec109870 2024-03-28T18:43:04.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.542+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec1a5ea0 con 0x7ff6ec109870 2024-03-28T18:43:04.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.542+0000 7ff6e8c00640 1 -- 172.21.15.119:0/2290489505 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4010760 con 0x7ff6ec109870 2024-03-28T18:43:04.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.542+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec1a6360 con 0x7ff6ec109870 2024-03-28T18:43:04.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6e8c00640 1 -- 172.21.15.119:0/2290489505 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff6e4005020 con 0x7ff6ec109870 2024-03-28T18:43:04.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 Processor -- start 2024-03-28T18:43:04.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 -- start start 2024-03-28T18:43:04.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0822cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c08234b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08191c90 con 0x557c0822cba0 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c0813a980 con 0x557c08232700 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.545+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c0818dd30 con 0x557c0822d590 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41582/0 (socket says 172.21.15.119:41582) 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 -- 172.21.15.119:0/4074522269 learned_addr learned my addr 172.21.15.119:0/4074522269 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 -- 172.21.15.119:0/4074522269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 msgr2=0x557c08234b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c08234b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6eb400640 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0822cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 -- 172.21.15.119:0/4074522269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 msgr2=0x557c0822cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0822cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6e8c00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 0x7ff6d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 -- 172.21.15.119:0/4074522269 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c08152510 con 0x557c0822d590 2024-03-28T18:43:04.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6eb400640 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0822cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6e8c00640 1 -- 172.21.15.119:0/2290489505 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ff6e40cef80 con 0x7ff6ec109870 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc010550 tx=0x7ff6dc010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3b04552e470ec9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.546+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 0x7ff6d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6e1200640 1 -- 172.21.15.119:0/4074522269 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc011760 con 0x557c0822d590 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6e1200640 1 -- 172.21.15.119:0/4074522269 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc011de0 con 0x557c0822d590 2024-03-28T18:43:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6e1200640 1 -- 172.21.15.119:0/4074522269 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00e430 con 0x557c0822d590 2024-03-28T18:43:04.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 -- 172.21.15.119:0/4074522269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 msgr2=0x557c08232090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc010550 tx=0x7ff6dc010a10 comp rx=0 tx=0).stop 2024-03-28T18:43:04.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 0x7ff6d00a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6d802c240 tx=0x7ff6d802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 -- 172.21.15.119:0/4074522269 shutdown_connections 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c08234b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c08232090 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/4074522269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0822cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 -- 172.21.15.119:0/4074522269 >> 172.21.15.119:0/4074522269 conn(0x557c08228110 msgr2=0x557c08228510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 -- 172.21.15.119:0/4074522269 shutdown_connections 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.547+0000 7ff6f57c2740 1 -- 172.21.15.119:0/4074522269 wait complete. 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 -- start start 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 Processor -- start 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08237080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c082375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.548+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c082356b0 con 0x557c0822cba0 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c0814ddd0 con 0x557c0822d590 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08235830 con 0x557c08232700 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f57c2740 10 client.0 _ll_register_callbacks cb 0x557c07ea5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c082375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08237080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c082375c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55842/0 (socket says 172.21.15.119:55842) 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08237080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57058/0 (socket says 172.21.15.119:57058) 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6ebe00640 1 -- 172.21.15.119:0/363040292 learned_addr learned my addr 172.21.15.119:0/363040292 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41594/0 (socket says 172.21.15.119:41594) 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 -- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 msgr2=0x557c082375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c082375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 -- 172.21.15.119:0/363040292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 msgr2=0x557c08237080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08237080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 -- 172.21.15.119:0/363040292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c07ea62d0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40cca50 tx=0x7ff6e409b200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=415c0ed2803349fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.549+0000 7ff6d5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e40b4060 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ff6e409cbd0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6dc011de0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e40ba170 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 10 client.25066 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557c07f706b0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc011de0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff6e40a3bb0 con 0x557c08232700 2024-03-28T18:43:04.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 10 client.25066 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 10 client.25066 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 20 client.25066 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082437f0 0x557c082428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 4 client.25066 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.550+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c08133520 con 0x557c082437f0 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.551+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082437f0 0x557c082428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.551+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082437f0 0x557c082428b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b2d01fe8309853f server_cookie=1586549bf620a0d3 in_seq=0 out_seq=0 2024-03-28T18:43:04.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.551+0000 7ff6d5600640 10 client.25066 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:04.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.552+0000 7ff6d5600640 1 -- 172.21.15.119:0/363040292 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c08133520 con 0x557c082437f0 2024-03-28T18:43:04.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.552+0000 7ff6d5600640 10 client.25066 handle_command_reply: tid=0 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.552+0000 7ff6f57c2740 1 client.25066 shutdown 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.552+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 msgr2=0x557c0823e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.552+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40cca50 tx=0x7ff6e409b200 comp rx=0 tx=0).stop 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 shutdown_connections 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082437f0 0x557c082428b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c0823e600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c082375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/363040292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08237080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6d5600640 0 client.25066 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 >> 172.21.15.119:0/363040292 conn(0x557c08228110 msgr2=0x557c08227310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 shutdown_connections 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 1 -- 172.21.15.119:0/363040292 wait complete. 2024-03-28T18:43:04.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.553+0000 7ff6f57c2740 20 client.25066 trim_cache size 0 max 16384 2024-03-28T18:43:04.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 Processor -- start 2024-03-28T18:43:04.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 -- start start 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08226a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c08226fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08243010 con 0x557c0822cba0 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.571+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08243670 con 0x557c0822d590 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08243cd0 con 0x557c08232700 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08226a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55848/0 (socket says 172.21.15.119:55848) 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 -- 172.21.15.119:0/2256648020 learned_addr learned my addr 172.21.15.119:0/2256648020 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c08226fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 -- 172.21.15.119:0/2256648020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 msgr2=0x557c08226fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c08226fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 -- 172.21.15.119:0/2256648020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 msgr2=0x557c08226a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08226a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6eb400640 1 -- 172.21.15.119:0/2256648020 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c0822d970 con 0x557c0822d590 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.572+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08226a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6eb400640 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80096c0 tx=0x7ff6d8040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72daf7fac032dd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6d7e00640 1 -- 172.21.15.119:0/2256648020 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d80041c0 con 0x557c0822d590 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6d7e00640 1 -- 172.21.15.119:0/2256648020 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6d8004ba0 con 0x557c0822d590 2024-03-28T18:43:04.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6d7e00640 1 -- 172.21.15.119:0/2256648020 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d80035f0 con 0x557c0822d590 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 -- 172.21.15.119:0/2256648020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 msgr2=0x557c0823e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80096c0 tx=0x7ff6d8040000 comp rx=0 tx=0).stop 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 -- 172.21.15.119:0/2256648020 shutdown_connections 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c08232700 0x557c08226fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c0822d590 0x557c0823e520 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/2256648020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c08226a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.573+0000 7ff6f57c2740 1 -- 172.21.15.119:0/2256648020 >> 172.21.15.119:0/2256648020 conn(0x557c08228110 msgr2=0x557c082284f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- 172.21.15.119:0/2256648020 shutdown_connections 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- 172.21.15.119:0/2256648020 wait complete. 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- start start 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 Processor -- start 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0823df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c0823a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08244990 con 0x557c0822cba0 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08244330 con 0x557c08232700 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.574+0000 7ff6f57c2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c08244ff0 con 0x557c0822d590 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6f57c2740 10 client.0 _ll_register_callbacks cb 0x7ff6e409b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41620/0 (socket says 172.21.15.119:41620) 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0823df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 -- 172.21.15.119:0/1549781284 learned_addr learned my addr 172.21.15.119:0/1549781284 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:04.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6f0a00640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c0823a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 -- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 msgr2=0x557c0823a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c0823a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 -- 172.21.15.119:0/1549781284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 msgr2=0x557c0823df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0823df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 -- 172.21.15.119:0/1549781284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c0823b220 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6eb400640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005ec0 tx=0x7ff6d8004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9556a9db03fa8f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.575+0000 7ff6d5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d805a040 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6d8004ba0 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ff6d8005450 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6d8005650 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 10 client.25072 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557c0829f5c0 con 0x557c0822d590 2024-03-28T18:43:04.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6d8004ba0 con 0x557c0822d590 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff6d805d070 con 0x557c0822d590 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 10 client.25072 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 10 client.25072 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 20 client.25072 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082a3780 0x557c0829f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 4 client.25072 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.576+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557c0819f1d0 con 0x557c082a3780 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082a3780 0x557c0829f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6ebe00640 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082a3780 0x557c0829f790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=442d5e6e6c70e253 server_cookie=841f1030164590f1 in_seq=0 out_seq=0 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6d5600640 10 client.25072 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6d5600640 1 -- 172.21.15.119:0/1549781284 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x557c0819f1d0 con 0x557c082a3780 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6d5600640 10 client.25072 handle_command_reply: tid=0 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 client.25072 shutdown 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 msgr2=0x557c0823e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005ec0 tx=0x7ff6d8004d50 comp rx=0 tx=0).stop 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 shutdown_connections 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c082a3780 0x557c0829f790 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c08232700 0x557c0823a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c0822d590 0x557c0823e490 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 --2- 172.21.15.119:0/1549781284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c0822cba0 0x557c0823df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6d5600640 0 client.25072 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.577+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 >> 172.21.15.119:0/1549781284 conn(0x557c08228110 msgr2=0x557c0822b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 shutdown_connections 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f57c2740 1 -- 172.21.15.119:0/1549781284 wait complete. 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f57c2740 20 client.25072 trim_cache size 0 max 16384 2024-03-28T18:43:04.582 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24563, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 103.476681418, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 111.268597298 2024-03-28T18:43:04.583 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 msgr2=0x7ff6d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 0x7ff6d00a4d40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6d802c240 tx=0x7ff6d802df90 comp rx=0 tx=0).stop 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 msgr2=0x7ff6ec1a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.578+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400eb10 tx=0x7ff6e400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 shutdown_connections 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff6d00a2880 0x7ff6d00a4d40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff6ec109870 0x7ff6ec1a5540 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6ec108eb0 0x7ff6ec19e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 --2- 172.21.15.119:0/2290489505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6ec107cc0 0x7ff6ec19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:04.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 >> 172.21.15.119:0/2290489505 conn(0x7ff6ec076c60 msgr2=0x7ff6ec10c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:04.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 shutdown_connections 2024-03-28T18:43:04.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:04.579+0000 7ff6f2a00640 1 -- 172.21.15.119:0/2290489505 wait complete. 2024-03-28T18:43:04.595 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T18:43:04.596 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:43:04.659 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:43:04.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T18:43:05.119 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:06.314 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:06 smithi119 ceph-mon[31813]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:06.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:06 smithi196 ceph-mon[36300]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:06.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:06 smithi131 ceph-mon[36065]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:07.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 -- 172.21.15.119:0/1839846925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0204102cb0 msgr2=0x7f02041050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:07.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 --2- 172.21.15.119:0/1839846925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0204102cb0 0x7f02041050a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009990 tx=0x7f01f4009ca0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 -- 172.21.15.119:0/1839846925 shutdown_connections 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 --2- 172.21.15.119:0/1839846925 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204107f10 0x7f020410a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 --2- 172.21.15.119:0/1839846925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0204102cb0 0x7f02041050a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 --2- 172.21.15.119:0/1839846925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02041055e0 0x7f02041079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 -- 172.21.15.119:0/1839846925 >> 172.21.15.119:0/1839846925 conn(0x7f02040fc660 msgr2=0x7f02040feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 -- 172.21.15.119:0/1839846925 shutdown_connections 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.260+0000 7f020a800700 1 -- 172.21.15.119:0/1839846925 wait complete. 2024-03-28T18:43:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 Processor -- start 2024-03-28T18:43:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 -- start start 2024-03-28T18:43:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 0x7f0204198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 0x7f020419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020410de20 con 0x7f0204107f10 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020410dca0 con 0x7f0204102cb0 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.262+0000 7f020a800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020410dfa0 con 0x7f02041055e0 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55866/0 (socket says 172.21.15.119:55866) 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0202a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 0x7f0204198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 -- 172.21.15.119:0/4242765466 learned_addr learned my addr 172.21.15.119:0/4242765466 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203e00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 0x7f020419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 -- 172.21.15.119:0/4242765466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 msgr2=0x7f0204198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 0x7f0204198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 -- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 msgr2=0x7f020419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 0x7f020419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 -- 172.21.15.119:0/4242765466 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02041a00b0 con 0x7f0204102cb0 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203e00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 0x7f020419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0202a00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 0x7f0204198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:07.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.263+0000 7f0203400700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005910 tx=0x7f01f400bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.264+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4026070 con 0x7f0204102cb0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.264+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01f400f460 con 0x7f0204102cb0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.264+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02041a0340 con 0x7f0204102cb0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.264+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f4021650 con 0x7f0204102cb0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.264+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02041a07f0 con 0x7f0204102cb0 2024-03-28T18:43:07.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.265+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0204068e50 con 0x7f0204102cb0 2024-03-28T18:43:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.267+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f01f4029070 con 0x7f0204102cb0 2024-03-28T18:43:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.268+0000 7f01fbe00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 0x7f01e80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.268+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f01f40cf860 con 0x7f0204102cb0 2024-03-28T18:43:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.268+0000 7f0202a00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 0x7f01e80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.269+0000 7f0202a00700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 0x7f01e80a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f01f0010420 tx=0x7f01f00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.276+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01f4013070 con 0x7f0204102cb0 2024-03-28T18:43:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.517+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02040625d0 con 0x7f01e80a2910 2024-03-28T18:43:07.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.519+0000 7f01fbe00700 1 -- 172.21.15.119:0/4242765466 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f02040625d0 con 0x7f01e80a2910 2024-03-28T18:43:07.519 INFO:teuthology.orchestra.run.smithi119.stdout:Backend: cephadm 2024-03-28T18:43:07.519 INFO:teuthology.orchestra.run.smithi119.stdout:Available: Yes 2024-03-28T18:43:07.519 INFO:teuthology.orchestra.run.smithi119.stdout:Paused: No 2024-03-28T18:43:07.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 msgr2=0x7f01e80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:07.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 0x7f01e80a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f01f0010420 tx=0x7f01f00073d0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 msgr2=0x7f0204198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:07.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005910 tx=0x7f01f400bbb0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 shutdown_connections 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f01e80a2910 0x7f01e80a4dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0204107f10 0x7f020419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0204102cb0 0x7f0204198450 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 --2- 172.21.15.119:0/4242765466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02041055e0 0x7f0204198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 >> 172.21.15.119:0/4242765466 conn(0x7f02040fc660 msgr2=0x7f02041036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.522+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 shutdown_connections 2024-03-28T18:43:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:07.523+0000 7f020a800700 1 -- 172.21.15.119:0/4242765466 wait complete. 2024-03-28T18:43:08.261 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:08 smithi119 ceph-mon[31813]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T18:43:08.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T18:43:08.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:08 smithi196 ceph-mon[36300]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T18:43:08.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:08 smithi131 ceph-mon[36065]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T18:43:08.762 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:09.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:09 smithi119 ceph-mon[31813]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:09.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:09 smithi196 ceph-mon[36300]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:09.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:09 smithi131 ceph-mon[36065]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:10.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:10 smithi196 ceph-mon[36300]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:43:10.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:10 smithi131 ceph-mon[36065]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:43:10.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:10 smithi119 ceph-mon[31813]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-28T18:43:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 -- 172.21.15.119:0/2902619200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238105fb0 msgr2=0x7fe23810c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 --2- 172.21.15.119:0/2902619200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238105fb0 0x7fe23810c860 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe23000b0d0 tx=0x7fe23000b3e0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 -- 172.21.15.119:0/2902619200 shutdown_connections 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 --2- 172.21.15.119:0/2902619200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238105fb0 0x7fe23810c860 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 --2- 172.21.15.119:0/2902619200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238104410 0x7fe238104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 --2- 172.21.15.119:0/2902619200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 0x7fe238105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 -- 172.21.15.119:0/2902619200 >> 172.21.15.119:0/2902619200 conn(0x7fe2380ff9b0 msgr2=0x7fe238101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 -- 172.21.15.119:0/2902619200 shutdown_connections 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.050+0000 7fe23de00700 1 -- 172.21.15.119:0/2902619200 wait complete. 2024-03-28T18:43:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.051+0000 7fe23de00700 1 Processor -- start 2024-03-28T18:43:11.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 -- start start 2024-03-28T18:43:11.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:11.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 0x7fe23819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe236a00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe236a00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51214/0 (socket says 172.21.15.119:51214) 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 0x7fe2381a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe238110050 con 0x7fe238105610 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe236a00700 1 -- 172.21.15.119:0/3823910351 learned_addr learned my addr 172.21.15.119:0/3823910351 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23810fed0 con 0x7fe238104410 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe22fe00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 0x7fe23819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.052+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2381101d0 con 0x7fe238105fb0 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe237400700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 0x7fe2381a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 -- 172.21.15.119:0/3823910351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 msgr2=0x7fe2381a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 0x7fe2381a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 -- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 msgr2=0x7fe23819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 0x7fe23819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2381a2210 con 0x7fe238104410 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe237400700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 0x7fe2381a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe236a00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe22800e950 tx=0x7fe22800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22800eea0 con 0x7fe238104410 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe228004510 con 0x7fe238104410 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2381a2500 con 0x7fe238104410 2024-03-28T18:43:11.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe228010430 con 0x7fe238104410 2024-03-28T18:43:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.053+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2381a2a40 con 0x7fe238104410 2024-03-28T18:43:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.054+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe238068e50 con 0x7fe238104410 2024-03-28T18:43:11.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.056+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe228003eb0 con 0x7fe238104410 2024-03-28T18:43:11.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.057+0000 7fe23ca00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 0x7fe21c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:11.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.058+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe2280cedf0 con 0x7fe238104410 2024-03-28T18:43:11.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.058+0000 7fe22fe00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 0x7fe21c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:11.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.058+0000 7fe22fe00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 0x7fe21c0a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe220009c60 tx=0x7fe220009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:11.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.061+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe22809bdb0 con 0x7fe238104410 2024-03-28T18:43:11.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.294+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2380625d0 con 0x7fe21c0a2910 2024-03-28T18:43:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.334+0000 7fe23ca00700 1 -- 172.21.15.119:0/3823910351 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fe2380625d0 con 0x7fe21c0a2910 2024-03-28T18:43:11.334 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T18:43:11.334 INFO:teuthology.orchestra.run.smithi119.stdout:mds.a smithi119 running (2m) 24s ago 2m 15.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 943dcc038444 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.b smithi131 running (118s) 49s ago 117s 26.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 db7dc93fa3eb 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.c smithi196 running (2m) 49s ago 2m 19.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4890878252c1 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.d smithi119 running (2m) 24s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9e4486e6ac16 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.e smithi131 running (2m) 49s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 85797476721b 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.f smithi196 running (2m) 49s ago 2m 17.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 dfc3be8469c9 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.g smithi119 running (2m) 24s ago 2m 15.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9831f119a165 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.h smithi131 running (2m) 49s ago 2m 17.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 c7c6084b52b7 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.i smithi196 running (2m) 49s ago 2m 19.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 0d9bbf43cdc0 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.j smithi119 running (2m) 24s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 94cbb2f839de 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.k smithi131 running (2m) 49s ago 2m 19.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bee28058a7d4 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mds.l smithi196 running (2m) 49s ago 2m 18.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d9354d6b3c93 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi119 *:9283,8765 running (10m) 24s ago 10m 515M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 fc993220303c 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi131 *:8443,8765 running (8m) 49s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 776175699c69 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.z smithi196 *:8443,8765 running (8m) 49s ago 8m 452M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 47eb4f473621 2024-03-28T18:43:11.335 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (10m) 24s ago 10m 48.0M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 38d4767e084a 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi131 running (8m) 49s ago 8m 47.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1527ac86c6d6 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi196 running (8m) 49s ago 8m 44.7M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e852c2239cf9 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (7m) 24s ago 7m 54.6M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 77b07b7c6e33 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi131 running (7m) 49s ago 7m 58.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d1627e40f90b 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi196 running (7m) 49s ago 7m 47.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a8ef1c8b5993 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (6m) 24s ago 6m 42.2M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 10b7de6ed13a 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi131 running (6m) 49s ago 6m 42.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a1ad531d9c53 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi196 running (5m) 49s ago 5m 55.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ac4fc0153863 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi119 running (5m) 24s ago 5m 52.3M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 a686821ef936 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi131 running (4m) 49s ago 4m 54.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 95c27047623f 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.8 smithi196 running (4m) 49s ago 4m 45.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b7ef2b3fc513 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.9 smithi119 running (3m) 24s ago 3m 40.8M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 d2b45de65bf9 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.10 smithi131 running (3m) 49s ago 3m 42.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 98e9cc69b8fb 2024-03-28T18:43:11.336 INFO:teuthology.orchestra.run.smithi119.stdout:osd.11 smithi196 running (2m) 49s ago 2m 39.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 30e5c6193ef0 2024-03-28T18:43:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 msgr2=0x7fe21c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 0x7fe21c0a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe220009c60 tx=0x7fe220009260 comp rx=0 tx=0).stop 2024-03-28T18:43:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 msgr2=0x7fe23819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe22800e950 tx=0x7fe22800ec60 comp rx=0 tx=0).stop 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 shutdown_connections 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe21c0a2910 0x7fe21c0a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe238105fb0 0x7fe2381a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe238104410 0x7fe23819a5b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 --2- 172.21.15.119:0/3823910351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe238105610 0x7fe23819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 >> 172.21.15.119:0/3823910351 conn(0x7fe2380ff9b0 msgr2=0x7fe238108840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 shutdown_connections 2024-03-28T18:43:11.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:11.337+0000 7fe23de00700 1 -- 172.21.15.119:0/3823910351 wait complete. 2024-03-28T18:43:12.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T18:43:12.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:12 smithi196 ceph-mon[36300]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:12.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:12 smithi131 ceph-mon[36065]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:12 smithi119 ceph-mon[31813]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-28T18:43:12.591 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:13.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:13 smithi119 ceph-mon[31813]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:13.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:13 smithi196 ceph-mon[36300]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:13.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:13 smithi131 ceph-mon[36065]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:14.407 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:14 smithi119 ceph-mon[31813]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T18:43:14.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:14 smithi196 ceph-mon[36300]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T18:43:14.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:14 smithi131 ceph-mon[36065]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-28T18:43:14.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 -- 172.21.15.119:0/192974376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 msgr2=0x7f89bc105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:14.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 --2- 172.21.15.119:0/192974376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc105a70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009990 tx=0x7f89a8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 -- 172.21.15.119:0/192974376 shutdown_connections 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 --2- 172.21.15.119:0/192974376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc105fb0 0x7f89bc10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 --2- 172.21.15.119:0/192974376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc104410 0x7f89bc104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 --2- 172.21.15.119:0/192974376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc105a70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 -- 172.21.15.119:0/192974376 >> 172.21.15.119:0/192974376 conn(0x7f89bc0ff9b0 msgr2=0x7f89bc101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 -- 172.21.15.119:0/192974376 shutdown_connections 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.740+0000 7f89c1600700 1 -- 172.21.15.119:0/192974376 wait complete. 2024-03-28T18:43:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 Processor -- start 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 -- start start 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 0x7f89bc19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 0x7f89bc1a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc10ffe0 con 0x7f89bc105fb0 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc10fe60 con 0x7f89bc104410 2024-03-28T18:43:14.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89c1600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc110160 con 0x7f89bc105610 2024-03-28T18:43:14.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89ba000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89baa00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 0x7f89bc19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89ba000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:33604/0 (socket says 172.21.15.119:33604) 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89ba000700 1 -- 172.21.15.119:0/3859586830 learned_addr learned my addr 172.21.15.119:0/3859586830 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.742+0000 7f89bb400700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 0x7f89bc1a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 -- 172.21.15.119:0/3859586830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 msgr2=0x7f89bc19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 0x7f89bc19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 -- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 msgr2=0x7f89bc1a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 0x7f89bc1a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 -- 172.21.15.119:0/3859586830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc1a2220 con 0x7f89bc105610 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89baa00700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 0x7f89bc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89bb400700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 0x7f89bc1a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89ba000700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f89a8000c00 tx=0x7f89a800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8027070 con 0x7f89bc105610 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a800fe20 con 0x7f89bc105610 2024-03-28T18:43:14.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.743+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc1a24b0 con 0x7f89bc105610 2024-03-28T18:43:14.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.744+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8017650 con 0x7f89bc105610 2024-03-28T18:43:14.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.744+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc1a2900 con 0x7f89bc105610 2024-03-28T18:43:14.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.745+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc068e50 con 0x7f89bc105610 2024-03-28T18:43:14.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.747+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f89a802a070 con 0x7f89bc105610 2024-03-28T18:43:14.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.748+0000 7f89b3400700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 0x7f89a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:14.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.748+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f89a80cfcc0 con 0x7f89bc105610 2024-03-28T18:43:14.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.748+0000 7f89baa00700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 0x7f89a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:14.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.749+0000 7f89baa00700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 0x7f89a00a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89bc105470 tx=0x7f89ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:14.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.755+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f89a8031080 con 0x7f89bc105610 2024-03-28T18:43:14.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:14.986+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f89bc0625d0 con 0x7f89a00a2890 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.004+0000 7f89b3400700 1 -- 172.21.15.119:0/3859586830 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f89bc0625d0 con 0x7f89a00a2890 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stdout:mds.all 12/12 53s ago 2m smithi119=a;smithi119=d;smithi119=g;smithi119=j;smithi131=b;smithi131=e;smithi131=h;smithi131=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stdout:mgr 3/3 53s ago 8m smithi119=x;smithi131=y;smithi196=z;count:3 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stdout:mon 3/3 53s ago 8m smithi119:172.21.15.119=a;smithi131:172.21.15.131=b;smithi196:172.21.15.196=c;count:3 2024-03-28T18:43:15.004 INFO:teuthology.orchestra.run.smithi119.stdout:osd 12 53s ago - 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 msgr2=0x7f89a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 0x7f89a00a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89bc105470 tx=0x7f89ac009260 comp rx=0 tx=0).stop 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 msgr2=0x7f89bc19ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f89a8000c00 tx=0x7f89a800f690 comp rx=0 tx=0).stop 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 shutdown_connections 2024-03-28T18:43:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89a00a2890 0x7f89a00a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89bc105fb0 0x7f89bc1a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89bc104410 0x7f89bc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 --2- 172.21.15.119:0/3859586830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89bc105610 0x7f89bc19ab00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.007+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 >> 172.21.15.119:0/3859586830 conn(0x7f89bc0ff9b0 msgr2=0x7f89bc108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.008+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 shutdown_connections 2024-03-28T18:43:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:15.008+0000 7f89c1600700 1 -- 172.21.15.119:0/3859586830 wait complete. 2024-03-28T18:43:15.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T18:43:16.280 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:16.431 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:16 smithi119 ceph-mon[31813]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:16.431 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:16 smithi119 ceph-mon[31813]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:16 smithi196 ceph-mon[36300]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:16.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:16 smithi196 ceph-mon[36300]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:16 smithi131 ceph-mon[36065]: from='client.25087 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:16.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:16 smithi131 ceph-mon[36065]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-28T18:43:18.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:18 smithi119 ceph-mon[31813]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 -- 172.21.15.119:0/3624332188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e4810a0c0 msgr2=0x7f0e4810c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 --2- 172.21.15.119:0/3624332188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e4810a0c0 0x7f0e4810c530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000b0a0 tx=0x7f0e4000b3b0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 -- 172.21.15.119:0/3624332188 shutdown_connections 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 --2- 172.21.15.119:0/3624332188 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e4810a0c0 0x7f0e4810c530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 --2- 172.21.15.119:0/3624332188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e480745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 --2- 172.21.15.119:0/3624332188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e48074b70 0x7f0e48074ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 -- 172.21.15.119:0/3624332188 >> 172.21.15.119:0/3624332188 conn(0x7f0e480fd7a0 msgr2=0x7f0e480ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 -- 172.21.15.119:0/3624332188 shutdown_connections 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.420+0000 7f0e4f400700 1 -- 172.21.15.119:0/3624332188 wait complete. 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 Processor -- start 2024-03-28T18:43:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 -- start start 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e4819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 0x7f0e481a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4810ff60 con 0x7f0e4810a0c0 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4810fde0 con 0x7f0e48074b70 2024-03-28T18:43:18.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4f400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481100e0 con 0x7f0e48076140 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4ca00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 0x7f0e481a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e47e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e4819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e4d400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 0x7f0e481a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40342/0 (socket says 172.21.15.119:40342) 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e4d400700 1 -- 172.21.15.119:0/310008796 learned_addr learned my addr 172.21.15.119:0/310008796 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.422+0000 7f0e4ca00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51264/0 (socket says 172.21.15.119:51264) 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 -- 172.21.15.119:0/310008796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 msgr2=0x7f0e4819a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 -- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 msgr2=0x7f0e481a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 0x7f0e481a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 -- 172.21.15.119:0/310008796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e481a2180 con 0x7f0e48076140 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e4ca00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e47e00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e4819aa60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ea20 tx=0x7f0e3400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.423+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34004220 con 0x7f0e48076140 2024-03-28T18:43:18.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.424+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e3400ce50 con 0x7f0e48076140 2024-03-28T18:43:18.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.424+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e481a2470 con 0x7f0e48076140 2024-03-28T18:43:18.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.424+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34003dc0 con 0x7f0e48076140 2024-03-28T18:43:18.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.424+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e481a2a00 con 0x7f0e48076140 2024-03-28T18:43:18.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.425+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e48068e50 con 0x7f0e48076140 2024-03-28T18:43:18.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.426+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0e3400c970 con 0x7f0e48076140 2024-03-28T18:43:18.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.428+0000 7f0e45600700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 0x7f0e300a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:18.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.428+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0e340cec90 con 0x7f0e48076140 2024-03-28T18:43:18.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.428+0000 7f0e4ca00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 0x7f0e300a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:18.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.429+0000 7f0e4ca00700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 0x7f0e300a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c011490 tx=0x7f0e3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:18.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.432+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0e3409bc50 con 0x7f0e48076140 2024-03-28T18:43:18.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:18 smithi196 ceph-mon[36300]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:18.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:18 smithi131 ceph-mon[36065]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:18.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.664+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e480625d0 con 0x7f0e300a2910 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.668+0000 7f0e45600700 1 -- 172.21.15.119:0/310008796 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f0e480625d0 con 0x7f0e300a2910 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:HOST ADDR LABELS STATUS 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 172.21.15.119 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:smithi131 172.21.15.131 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:smithi196 172.21.15.196 2024-03-28T18:43:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:3 hosts in cluster 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.670+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 msgr2=0x7f0e300a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.670+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 0x7f0e300a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c011490 tx=0x7f0e3c009260 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.670+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 msgr2=0x7f0e4819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.670+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e4819aa60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ea20 tx=0x7f0e3400ed30 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 shutdown_connections 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f0e300a2910 0x7f0e300a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e4810a0c0 0x7f0e481a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0e48074b70 0x7f0e4819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 --2- 172.21.15.119:0/310008796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e48076140 0x7f0e4819aa60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 >> 172.21.15.119:0/310008796 conn(0x7f0e480fd7a0 msgr2=0x7f0e48106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 shutdown_connections 2024-03-28T18:43:18.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:18.671+0000 7f0e4f400700 1 -- 172.21.15.119:0/310008796 wait complete. 2024-03-28T18:43:19.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T18:43:19.873 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:20 smithi119 ceph-mon[31813]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:20 smithi119 ceph-mon[31813]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:20 smithi196 ceph-mon[36300]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:20.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:20 smithi196 ceph-mon[36300]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:20 smithi131 ceph-mon[36065]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:20.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:20 smithi131 ceph-mon[36065]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:22.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 -- 172.21.15.119:0/1065241237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8054103d60 msgr2=0x7f805410a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 --2- 172.21.15.119:0/1065241237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8054103d60 0x7f805410a610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f804c00b0d0 tx=0x7f804c00b3e0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 -- 172.21.15.119:0/1065241237 shutdown_connections 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 --2- 172.21.15.119:0/1065241237 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8054103d60 0x7f805410a610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 --2- 172.21.15.119:0/1065241237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80541021c0 0x7f80541025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 --2- 172.21.15.119:0/1065241237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 0x7f8054103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 -- 172.21.15.119:0/1065241237 >> 172.21.15.119:0/1065241237 conn(0x7f80540fd7a0 msgr2=0x7f80540ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 -- 172.21.15.119:0/1065241237 shutdown_connections 2024-03-28T18:43:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.054+0000 7f8059000700 1 -- 172.21.15.119:0/1065241237 wait complete. 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.055+0000 7f8059000700 1 Processor -- start 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 -- start start 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 0x7f805419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 0x7f80541a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805410ddf0 con 0x7f8054103d60 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805410dc70 con 0x7f80541021c0 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8059000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805410df70 con 0x7f80541033c0 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8051e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8051e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45600/0 (socket says 172.21.15.119:45600) 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8051e00700 1 -- 172.21.15.119:0/688900181 learned_addr learned my addr 172.21.15.119:0/688900181 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f804be00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 0x7f805419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.056+0000 7f8052800700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 0x7f80541a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 -- 172.21.15.119:0/688900181 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 msgr2=0x7f805419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 0x7f805419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 -- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 msgr2=0x7f80541a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 0x7f80541a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 -- 172.21.15.119:0/688900181 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80541a2210 con 0x7f80541021c0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8052800700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 0x7f80541a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8051e00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f804400e970 tx=0x7f804400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400cbd0 con 0x7f80541021c0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8044004500 con 0x7f80541021c0 2024-03-28T18:43:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044003e30 con 0x7f80541021c0 2024-03-28T18:43:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80541a2500 con 0x7f80541021c0 2024-03-28T18:43:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.057+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80541a29f0 con 0x7f80541021c0 2024-03-28T18:43:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.059+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8054068e50 con 0x7f80541021c0 2024-03-28T18:43:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.060+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8044004000 con 0x7f80541021c0 2024-03-28T18:43:22.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.061+0000 7f804aa00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 0x7f80380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:22.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.062+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f80440cee90 con 0x7f80541021c0 2024-03-28T18:43:22.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.062+0000 7f804be00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 0x7f80380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:22.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.062+0000 7f804be00700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 0x7f80380a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f803c007be0 tx=0x7f803c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.066+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f804400b2c0 con 0x7f80541021c0 2024-03-28T18:43:22.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.299+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8054075ae0 con 0x7f80380a2890 2024-03-28T18:43:22.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.301+0000 7f804aa00700 1 -- 172.21.15.119:0/688900181 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f8054075ae0 con 0x7f80380a2890 2024-03-28T18:43:22.302 INFO:teuthology.orchestra.run.smithi119.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T18:43:22.302 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y1400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:43:22.302 INFO:teuthology.orchestra.run.smithi119.stdout:smithi131 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300N4400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T18:43:22.303 INFO:teuthology.orchestra.run.smithi119.stdout:smithi196 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T18:43:22.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 msgr2=0x7f80380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:22.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 0x7f80380a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f803c007be0 tx=0x7f803c0073d0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 msgr2=0x7f805419a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:22.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f804400e970 tx=0x7f804400ec80 comp rx=0 tx=0).stop 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 shutdown_connections 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f80380a2890 0x7f80380a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8054103d60 0x7f80541a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80541021c0 0x7f805419a5b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 --2- 172.21.15.119:0/688900181 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80541033c0 0x7f805419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 >> 172.21.15.119:0/688900181 conn(0x7f80540fd7a0 msgr2=0x7f80540743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 shutdown_connections 2024-03-28T18:43:22.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:22.304+0000 7f8059000700 1 -- 172.21.15.119:0/688900181 wait complete. 2024-03-28T18:43:22.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:22 smithi196 ceph-mon[36300]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:22.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:22 smithi131 ceph-mon[36065]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:22 smithi119 ceph-mon[31813]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:23.099 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T18:43:23.110 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:43:23.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T18:43:23.572 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:24 smithi196 ceph-mon[36300]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:24.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:24 smithi196 ceph-mon[36300]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:24 smithi131 ceph-mon[36065]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:24.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:24 smithi131 ceph-mon[36065]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:24.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:24 smithi119 ceph-mon[31813]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:43:24.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:24 smithi119 ceph-mon[31813]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:25.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 -- 172.21.15.119:0/1238565148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9150109a10 msgr2=0x7f9150109e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:25.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 --2- 172.21.15.119:0/1238565148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9150109a10 0x7f9150109e70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f913c0099c0 tx=0x7f913c009cd0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 -- 172.21.15.119:0/1238565148 shutdown_connections 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 --2- 172.21.15.119:0/1238565148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f915010a3b0 0x7f9150110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 --2- 172.21.15.119:0/1238565148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f9150108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 --2- 172.21.15.119:0/1238565148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9150109a10 0x7f9150109e70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 -- 172.21.15.119:0/1238565148 >> 172.21.15.119:0/1238565148 conn(0x7f91500777e0 msgr2=0x7f9150079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 -- 172.21.15.119:0/1238565148 shutdown_connections 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.914+0000 7f9157200700 1 -- 172.21.15.119:0/1238565148 wait complete. 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.915+0000 7f9157200700 1 Processor -- start 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 -- start start 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 0x7f915019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 0x7f91501a5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9150114420 con 0x7f9150109a10 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91501142a0 con 0x7f915010a3b0 2024-03-28T18:43:25.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f9157200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91501145a0 con 0x7f9150108810 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f914fe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f914fe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:57498/0 (socket says 172.21.15.119:57498) 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.916+0000 7f914fe00700 1 -- 172.21.15.119:0/1962007883 learned_addr learned my addr 172.21.15.119:0/1962007883 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914f400700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 0x7f915019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f9155200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 0x7f91501a5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 -- 172.21.15.119:0/1962007883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 msgr2=0x7f91501a5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 0x7f91501a5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 -- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 msgr2=0x7f915019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 0x7f915019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 -- 172.21.15.119:0/1962007883 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91501a6660 con 0x7f9150108810 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f9155200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 0x7f91501a5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914f400700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 0x7f915019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.917+0000 7f914fe00700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f914000e9a0 tx=0x7f914000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.918+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914000cbd0 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.918+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9140004500 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.918+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91501a6950 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.918+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9140010430 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.918+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91501a6ee0 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.919+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9150068e50 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.921+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9140003f40 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.922+0000 7f914cc00700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 0x7f91340a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.923+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f91400cedd0 con 0x7f9150108810 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.923+0000 7f914f400700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 0x7f91340a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.923+0000 7f914f400700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 0x7f91340a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:25.926+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f914009bd90 con 0x7f9150108810 2024-03-28T18:43:26.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.194+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9150062da0 con 0x7f9150108810 2024-03-28T18:43:26.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:26 smithi119 ceph-mon[31813]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:26.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.195+0000 7f914cc00700 1 -- 172.21.15.119:0/1962007883 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+3047 (secure 0 0 0) 0x7f914009b690 con 0x7f9150108810 2024-03-28T18:43:26.196 INFO:teuthology.orchestra.run.smithi119.stdout:e27 2024-03-28T18:43:26.196 INFO:teuthology.orchestra.run.smithi119.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T18:43:26.196 INFO:teuthology.orchestra.run.smithi119.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:legacy client fscid: 1 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:Filesystem 'cephfs' (1) 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:fs_name cephfs 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 27 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:created 2024-03-28T18:41:20.510917+0000 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:modified 2024-03-28T18:41:28.190103+0000 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:tableserver 0 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:root 0 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:session_timeout 300 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:session_autoclose 300 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:max_file_size 1099511627776 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:max_xattr_size 65536 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:required_client_features {} 2024-03-28T18:43:26.197 INFO:teuthology.orchestra.run.smithi119.stdout:last_failure 0 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:last_failure_osd_epoch 0 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:max_mds 5 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:in 0,1,2,3,4 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:up {0=24563,1=24538,2=24517,3=24515,4=24505} 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:failed 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:damaged 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:stopped 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:data_pools [3] 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:metadata_pool 2 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:inline_data disabled 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:balancer 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:bal_rank_mask -1 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:standby_count_wanted 0 2024-03-28T18:43:26.198 INFO:teuthology.orchestra.run.smithi119.stdout:qdb_cluster leader: 24563 members: 24538,24517,24563,24515,24505 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b{0:24563} state up:active seq 3 addr [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.e{0:24485} state up:standby-replay seq 1 addr [v2:172.21.15.131:6834/1538863538,v1:172.21.15.131:6835/1538863538] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.i{1:24538} state up:active seq 7 addr [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.l{1:24442} state up:standby-replay seq 1 addr [v2:172.21.15.196:6832/1244574561,v1:172.21.15.196:6833/1244574561] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c{2:24517} state up:active seq 8 addr [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.k{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.131:6832/4160748351,v1:172.21.15.131:6833/4160748351] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.h{3:24515} state up:active seq 8 addr [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a{3:14460} state up:standby-replay seq 1 addr [v2:172.21.15.119:6840/683166305,v1:172.21.15.119:6841/683166305] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.f{4:24505} state up:active seq 10 addr [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.g{4:14448} state up:standby-replay seq 1 addr [v2:172.21.15.119:6838/1834295684,v1:172.21.15.119:6839/1834295684] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout:Standby daemons: 2024-03-28T18:43:26.199 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:26.200 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.119:6834/2486653544,v1:172.21.15.119:6835/2486653544] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.201 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d{-1:14400} state up:standby seq 1 addr [v2:172.21.15.119:6836/2841831538,v1:172.21.15.119:6837/2841831538] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 msgr2=0x7f91340a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 0x7f91340a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f913c000c00 tx=0x7f913c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 msgr2=0x7f915019ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f914000e9a0 tx=0x7f914000ed60 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 shutdown_connections 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f91340a2910 0x7f91340a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f915010a3b0 0x7f91501a5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9150108810 0x7f915019ea00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 --2- 172.21.15.119:0/1962007883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9150109a10 0x7f915019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 >> 172.21.15.119:0/1962007883 conn(0x7f91500777e0 msgr2=0x7f915010ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:26.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 shutdown_connections 2024-03-28T18:43:26.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:26.199+0000 7f9157200700 1 -- 172.21.15.119:0/1962007883 wait complete. 2024-03-28T18:43:26.203 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:26.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:26 smithi196 ceph-mon[36300]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:26.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:26 smithi131 ceph-mon[36065]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:27.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T18:43:27.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:27 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1962007883' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:43:27.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:27 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1962007883' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:43:27.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:27 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1962007883' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T18:43:27.492 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:28 smithi196 ceph-mon[36300]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:28.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:28 smithi131 ceph-mon[36065]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:28 smithi119 ceph-mon[31813]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:29.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.776+0000 7f96eb200700 1 -- 172.21.15.119:0/734198410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec109900 msgr2=0x7f96ec109d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:29.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 --2- 172.21.15.119:0/734198410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec109900 0x7f96ec109d60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f96d8009990 tx=0x7f96d8009ca0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 -- 172.21.15.119:0/734198410 shutdown_connections 2024-03-28T18:43:29.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 --2- 172.21.15.119:0/734198410 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec10a2a0 0x7f96ec110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 --2- 172.21.15.119:0/734198410 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96ec108700 0x7f96ec108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 --2- 172.21.15.119:0/734198410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec109900 0x7f96ec109d60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 -- 172.21.15.119:0/734198410 >> 172.21.15.119:0/734198410 conn(0x7f96ec0776d0 msgr2=0x7f96ec079b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 -- 172.21.15.119:0/734198410 shutdown_connections 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.777+0000 7f96eb200700 1 -- 172.21.15.119:0/734198410 wait complete. 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.778+0000 7f96eb200700 1 Processor -- start 2024-03-28T18:43:29.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 -- start start 2024-03-28T18:43:29.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:29.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96ec109900 0x7f96ec19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:29.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec10a2a0 0x7f96ec1a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec1142f0 con 0x7f96ec109900 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec114170 con 0x7f96ec108700 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96eb200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec114470 con 0x7f96ec10a2a0 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96e9e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96e9e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57314/0 (socket says 172.21.15.119:57314) 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96e9e00700 1 -- 172.21.15.119:0/1260740160 learned_addr learned my addr 172.21.15.119:0/1260740160 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.779+0000 7f96ea800700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec10a2a0 0x7f96ec1a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 -- 172.21.15.119:0/1260740160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec10a2a0 msgr2=0x7f96ec1a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec10a2a0 0x7f96ec1a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 -- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96ec109900 msgr2=0x7f96ec19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96ec109900 0x7f96ec19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 -- 172.21.15.119:0/1260740160 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec1a6510 con 0x7f96ec108700 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e9e00700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00d980 tx=0x7f96dc00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:29.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc016070 con 0x7f96ec108700 2024-03-28T18:43:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96dc00be30 con 0x7f96ec108700 2024-03-28T18:43:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc00f3c0 con 0x7f96ec108700 2024-03-28T18:43:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec1a6800 con 0x7f96ec108700 2024-03-28T18:43:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.780+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec1a6d40 con 0x7f96ec108700 2024-03-28T18:43:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.781+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec068e50 con 0x7f96ec108700 2024-03-28T18:43:29.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.783+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96dc019070 con 0x7f96ec108700 2024-03-28T18:43:29.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.785+0000 7f96e2a00700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 0x7f96d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:29.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.785+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f96dc0cef60 con 0x7f96ec108700 2024-03-28T18:43:29.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.785+0000 7f96e9400700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 0x7f96d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:29.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.786+0000 7f96e9400700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 0x7f96d00a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96d80098e0 tx=0x7f96d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:29.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:29.787+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96dc020030 con 0x7f96ec108700 2024-03-28T18:43:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.008+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f96ec062da0 con 0x7f96ec108700 2024-03-28T18:43:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.009+0000 7f96e2a00700 1 -- 172.21.15.119:0/1260740160 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) v1 ==== 56+0+5442 (secure 0 0 0) 0x7f96dc09bf20 con 0x7f96ec108700 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 90 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:created 2024-03-28T18:32:22.573529+0000 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:modified 2024-03-28T18:42:31.162785+0000 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:crush_version 26 2024-03-28T18:43:30.012 INFO:teuthology.orchestra.run.smithi119.stdout:full_ratio 0.95 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:backfillfull_ratio 0.9 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:nearfull_ratio 0.85 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:require_min_compat_client luminous 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:min_compat_client luminous 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:require_osd_release squid 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:stretch_mode_enabled false 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:max_osd 12 2024-03-28T18:43:30.013 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6802/532063015,v1:172.21.15.119:6803/532063015] [v2:172.21.15.119:6804/532063015,v1:172.21.15.119:6805/532063015] exists,up 6f9791ee-9ecd-4e9e-b59e-90533047d529 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6800/1240079334,v1:172.21.15.131:6801/1240079334] [v2:172.21.15.131:6802/1240079334,v1:172.21.15.131:6803/1240079334] exists,up e56d8077-84c3-494e-8b08-9f5ea7b7cfa4 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/1413796404,v1:172.21.15.196:6801/1413796404] [v2:172.21.15.196:6802/1413796404,v1:172.21.15.196:6803/1413796404] exists,up 4c080cdf-65ee-41c9-be0f-18988515b0f0 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6810/837858906,v1:172.21.15.119:6811/837858906] [v2:172.21.15.119:6812/837858906,v1:172.21.15.119:6813/837858906] exists,up 15e6e92d-b038-4f17-8cb8-9a0c6db99e93 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6808/1834627175,v1:172.21.15.131:6809/1834627175] [v2:172.21.15.131:6810/1834627175,v1:172.21.15.131:6811/1834627175] exists,up ce922e09-9a15-418f-a01b-b70921dfbe71 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/3397729850,v1:172.21.15.196:6809/3397729850] [v2:172.21.15.196:6810/3397729850,v1:172.21.15.196:6811/3397729850] exists,up 6b57956f-6e7d-4bc7-8b89-5390058b7ba8 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 up in weight 1 up_from 47 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6818/2797182795,v1:172.21.15.119:6819/2797182795] [v2:172.21.15.119:6820/2797182795,v1:172.21.15.119:6821/2797182795] exists,up 7b527cff-096d-4e00-bfda-8e3e77bc70e1 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6816/417830389,v1:172.21.15.131:6817/417830389] [v2:172.21.15.131:6818/417830389,v1:172.21.15.131:6819/417830389] exists,up 044ba726-17cb-4d14-9040-5efd8e892821 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.8 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/1701924552,v1:172.21.15.196:6817/1701924552] [v2:172.21.15.196:6818/1701924552,v1:172.21.15.196:6819/1701924552] exists,up 4c9fcc00-0b32-4fc0-9ade-3f152779462e 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.9 up in weight 1 up_from 68 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6826/1454392710,v1:172.21.15.119:6827/1454392710] [v2:172.21.15.119:6828/1454392710,v1:172.21.15.119:6829/1454392710] exists,up db64ebe3-7820-4944-9d3d-9efdad1576bd 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.131:6824/2982886262,v1:172.21.15.131:6825/2982886262] [v2:172.21.15.131:6826/2982886262,v1:172.21.15.131:6827/2982886262] exists,up 46b91f6b-8ffe-4d4e-9c36-d3faafc12d12 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/2114329937,v1:172.21.15.196:6825/2114329937] [v2:172.21.15.196:6826/2114329937,v1:172.21.15.196:6827/2114329937] exists,up 5eb4209c-3199-4138-b104-8f07d626633d 2024-03-28T18:43:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.4 [2,11] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.5 [8,11] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.b [2,4] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.1a [8,1] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.1c [5,4] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 2.1d [2,4,7,6] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.0 [5,9] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.3 [4,9] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.6 [3,8] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.14 [1,9] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.1b [4,8] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.23 [4,7] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.28 [5,6] 2024-03-28T18:43:30.015 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/3359592084 expires 2024-03-29T18:33:28.825026+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/4209865453 expires 2024-03-29T18:32:54.930081+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2033406365 expires 2024-03-29T18:33:28.825026+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/3103230972 expires 2024-03-29T18:32:54.930081+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/1303314658 expires 2024-03-29T18:32:54.930081+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/3984787959 expires 2024-03-29T18:32:54.930081+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2617254040 expires 2024-03-29T18:33:28.825026+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/3103230972 expires 2024-03-29T18:32:54.930081+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/3359592084 expires 2024-03-29T18:33:28.825026+0000 2024-03-28T18:43:30.016 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/4220166218 expires 2024-03-29T18:33:28.825026+0000 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 msgr2=0x7f96d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 0x7f96d00a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96d80098e0 tx=0x7f96d8005fb0 comp rx=0 tx=0).stop 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 msgr2=0x7f96ec19e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00d980 tx=0x7f96dc00dc90 comp rx=0 tx=0).stop 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 shutdown_connections 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f96d00a2890 0x7f96d00a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec10a2a0 0x7f96ec1a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:30.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96ec108700 0x7f96ec19e8b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 --2- 172.21.15.119:0/1260740160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96ec109900 0x7f96ec19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 >> 172.21.15.119:0/1260740160 conn(0x7f96ec0776d0 msgr2=0x7f96ec10cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 shutdown_connections 2024-03-28T18:43:30.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:30.012+0000 7f96eb200700 1 -- 172.21.15.119:0/1260740160 wait complete. 2024-03-28T18:43:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:30 smithi119 ceph-mon[31813]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:30.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:30 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1260740160' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:43:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:30 smithi196 ceph-mon[36300]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:30 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1260740160' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:43:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:30 smithi131 ceph-mon[36065]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:30.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:30 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1260740160' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T18:43:30.832 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T18:43:30.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 -- 172.21.15.119:0/1211569008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c105490 msgr2=0x7ff02c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 --2- 172.21.15.119:0/1211569008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c105490 0x7ff02c10bd20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff024009ca0 tx=0x7ff02402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 -- 172.21.15.119:0/1211569008 shutdown_connections 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 --2- 172.21.15.119:0/1211569008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c105490 0x7ff02c10bd20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 --2- 172.21.15.119:0/1211569008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c104ad0 0x7ff02c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 --2- 172.21.15.119:0/1211569008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.054+0000 7ff034000640 1 -- 172.21.15.119:0/1211569008 >> 172.21.15.119:0/1211569008 conn(0x7ff02c0fee50 msgr2=0x7ff02c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.055+0000 7ff034000640 1 -- 172.21.15.119:0/1211569008 shutdown_connections 2024-03-28T18:43:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.055+0000 7ff034000640 1 -- 172.21.15.119:0/1211569008 wait complete. 2024-03-28T18:43:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.055+0000 7ff034000640 1 Processor -- start 2024-03-28T18:43:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 -- start start 2024-03-28T18:43:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 0x7ff02c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c105490 0x7ff02c1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c10f560 con 0x7ff02c105490 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c10f3e0 con 0x7ff02c104ad0 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff034000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02c10f6e0 con 0x7ff02c1038e0 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff031600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff031600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36114/0 (socket says 172.21.15.119:36114) 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff031600640 1 -- 172.21.15.119:0/300961556 learned_addr learned my addr 172.21.15.119:0/300961556 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff030c00640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 0x7ff02c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.056+0000 7ff032000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c105490 0x7ff02c1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 -- 172.21.15.119:0/300961556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 msgr2=0x7ff02c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 0x7ff02c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 -- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c105490 msgr2=0x7ff02c1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c105490 0x7ff02c1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 -- 172.21.15.119:0/300961556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c1a17c0 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff030c00640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 0x7ff02c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff031600640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e9c0 tx=0x7ff02000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02000cd80 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02c1a1ab0 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0200035f0 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020010690 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.057+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02c1a1fb0 con 0x7ff02c1038e0 2024-03-28T18:43:31.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.058+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff02c10aba0 con 0x7ff02c1038e0 2024-03-28T18:43:31.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.060+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff0200037a0 con 0x7ff02c1038e0 2024-03-28T18:43:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.062+0000 7ff02a000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.062+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ff0200cee90 con 0x7ff02c1038e0 2024-03-28T18:43:31.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.062+0000 7ff030c00640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.062+0000 7ff030c00640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 0x7ff0140a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff02c19b0b0 tx=0x7ff018005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.064+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff02009be50 con 0x7ff02c1038e0 2024-03-28T18:43:31.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.264+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff02c02d070 con 0x7ff02c1038e0 2024-03-28T18:43:31.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.266+0000 7ff02a000640 1 -- 172.21.15.119:0/300961556 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7ff02009b750 con 0x7ff02c1038e0 2024-03-28T18:43:31.267 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:31.267 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 msgr2=0x7ff0140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 0x7ff0140a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff02c19b0b0 tx=0x7ff018005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 msgr2=0x7ff02c199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e9c0 tx=0x7ff02000ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 shutdown_connections 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff0140a2800 0x7ff0140a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff02c105490 0x7ff02c1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff02c104ad0 0x7ff02c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 --2- 172.21.15.119:0/300961556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff02c1038e0 0x7ff02c199ba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 >> 172.21.15.119:0/300961556 conn(0x7ff02c0fee50 msgr2=0x7ff02c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:31.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 shutdown_connections 2024-03-28T18:43:31.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.269+0000 7ff034000640 1 -- 172.21.15.119:0/300961556 wait complete. 2024-03-28T18:43:31.289 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:31.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:31.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 -- 172.21.15.119:0/1096003076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 msgr2=0x7ffb0c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 --2- 172.21.15.119:0/1096003076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c077700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ffaf4009860 tx=0x7ffaf402bb70 comp rx=0 tx=0).stop 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 -- 172.21.15.119:0/1096003076 shutdown_connections 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 --2- 172.21.15.119:0/1096003076 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 --2- 172.21.15.119:0/1096003076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c077700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 --2- 172.21.15.119:0/1096003076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 0x7ffb0c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 -- 172.21.15.119:0/1096003076 >> 172.21.15.119:0/1096003076 conn(0x7ffb0c0fee50 msgr2=0x7ffb0c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 -- 172.21.15.119:0/1096003076 shutdown_connections 2024-03-28T18:43:31.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.497+0000 7ffb11200640 1 -- 172.21.15.119:0/1096003076 wait complete. 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.498+0000 7ffb11200640 1 Processor -- start 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.498+0000 7ffb11200640 1 -- start start 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 0x7ffb0c19e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c113970 con 0x7ffb0c0772a0 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c1137f0 con 0x7ffb0c077c40 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb11200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c113af0 con 0x7ffb0c076060 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb09e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 0x7ffb0c19e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57344/0 (socket says 172.21.15.119:57344) 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 -- 172.21.15.119:0/1218223650 learned_addr learned my addr 172.21.15.119:0/1218223650 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb01400640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 -- 172.21.15.119:0/1218223650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 msgr2=0x7ffb0c19e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 0x7ffb0c19e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 -- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 msgr2=0x7ffb0c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.499+0000 7ffb0a800640 1 -- 172.21.15.119:0/1218223650 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c1a5d00 con 0x7ffb0c077c40 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb0a800640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400efc0 tx=0x7ffb0400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003970 con 0x7ffb0c077c40 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb04004020 con 0x7ffb0c077c40 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c1a5ff0 con 0x7ffb0c077c40 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400b9f0 con 0x7ffb0c077c40 2024-03-28T18:43:31.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.500+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c1a6460 con 0x7ffb0c077c40 2024-03-28T18:43:31.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.501+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c110250 con 0x7ffb0c077c40 2024-03-28T18:43:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.503+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffb0401d020 con 0x7ffb0c077c40 2024-03-28T18:43:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.504+0000 7ffb03400640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 0x7ffaf00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.504+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ffb040ce940 con 0x7ffb0c077c40 2024-03-28T18:43:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.504+0000 7ffb09e00640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 0x7ffaf00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.505+0000 7ffb09e00640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 0x7ffaf00a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009bd0 tx=0x7ffafc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.508+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffb0409b900 con 0x7ffb0c077c40 2024-03-28T18:43:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.706+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb0c0622a0 con 0x7ffb0c077c40 2024-03-28T18:43:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.706+0000 7ffb03400640 1 -- 172.21.15.119:0/1218223650 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb0409b200 con 0x7ffb0c077c40 2024-03-28T18:43:31.706 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 msgr2=0x7ffaf00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 0x7ffaf00a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009bd0 tx=0x7ffafc005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 msgr2=0x7ffb0c1a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400efc0 tx=0x7ffb0400c800 comp rx=0 tx=0).stop 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 shutdown_connections 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ffaf00a2800 0x7ffaf00a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb0c077c40 0x7ffb0c1a5690 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb0c0772a0 0x7ffb0c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 --2- 172.21.15.119:0/1218223650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0c076060 0x7ffb0c19e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 >> 172.21.15.119:0/1218223650 conn(0x7ffb0c0fee50 msgr2=0x7ffb0c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:31.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 shutdown_connections 2024-03-28T18:43:31.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.711+0000 7ffb11200640 1 -- 172.21.15.119:0/1218223650 wait complete. 2024-03-28T18:43:31.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 -- 172.21.15.119:0/992415144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d880772a0 msgr2=0x7f3d88077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 --2- 172.21.15.119:0/992415144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d880772a0 0x7f3d88077700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3d74009900 tx=0x7f3d7402bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 -- 172.21.15.119:0/992415144 shutdown_connections 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 --2- 172.21.15.119:0/992415144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88077c40 0x7f3d880784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 --2- 172.21.15.119:0/992415144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d880772a0 0x7f3d88077700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 --2- 172.21.15.119:0/992415144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d88076060 0x7f3d88076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 -- 172.21.15.119:0/992415144 >> 172.21.15.119:0/992415144 conn(0x7f3d880fee50 msgr2=0x7f3d88101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 -- 172.21.15.119:0/992415144 shutdown_connections 2024-03-28T18:43:31.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.947+0000 7f3d90000640 1 -- 172.21.15.119:0/992415144 wait complete. 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.948+0000 7f3d90000640 1 Processor -- start 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.948+0000 7f3d90000640 1 -- start start 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.948+0000 7f3d90000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 0x7f3d8819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d90000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d880772a0 0x7f3d8819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d90000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 0x7f3d881a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d90000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88113970 con 0x7f3d88076060 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d90000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d881137f0 con 0x7f3d880772a0 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d90000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d88113af0 con 0x7f3d88077c40 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8d600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 0x7f3d8819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8cc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d880772a0 0x7f3d8819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 0x7f3d881a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8d600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 0x7f3d8819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38034/0 (socket says 172.21.15.119:38034) 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8d600640 1 -- 172.21.15.119:0/931557224 learned_addr learned my addr 172.21.15.119:0/931557224 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 -- 172.21.15.119:0/931557224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d880772a0 msgr2=0x7f3d8819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d880772a0 0x7f3d8819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 -- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 msgr2=0x7f3d8819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 0x7f3d8819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:31.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 -- 172.21.15.119:0/931557224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d881a5c20 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.949+0000 7f3d8e000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 0x7f3d881a55b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e970 tx=0x7f3d7800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.950+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800ccc0 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.950+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d78005020 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.950+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d881a5f10 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.950+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78004a00 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.950+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d881a6380 con 0x7f3d88077c40 2024-03-28T18:43:31.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.951+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d88110250 con 0x7f3d88077c40 2024-03-28T18:43:31.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.953+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3d780051d0 con 0x7f3d88077c40 2024-03-28T18:43:31.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.954+0000 7f3d86000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:31.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.954+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f3d780ceaa0 con 0x7f3d88077c40 2024-03-28T18:43:31.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.954+0000 7f3d8d600640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:31.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.955+0000 7f3d8d600640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3d800049d0 tx=0x7f3d80009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:31.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:31.958+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d7809ba60 con 0x7f3d88077c40 2024-03-28T18:43:32.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.156+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d880622d0 con 0x7f3d88077c40 2024-03-28T18:43:32.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.157+0000 7f3d86000640 1 -- 172.21.15.119:0/931557224 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d7809b360 con 0x7f3d88077c40 2024-03-28T18:43:32.157 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 msgr2=0x7f3d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3d800049d0 tx=0x7f3d80009500 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 msgr2=0x7f3d881a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 0x7f3d881a55b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e970 tx=0x7f3d7800ee30 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 shutdown_connections 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3d6c0a2800 0x7f3d6c0a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d88077c40 0x7f3d881a55b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d880772a0 0x7f3d8819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 --2- 172.21.15.119:0/931557224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d88076060 0x7f3d8819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.159+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 >> 172.21.15.119:0/931557224 conn(0x7f3d880fee50 msgr2=0x7f3d88100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.160+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 shutdown_connections 2024-03-28T18:43:32.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.160+0000 7f3d90000640 1 -- 172.21.15.119:0/931557224 wait complete. 2024-03-28T18:43:32.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.394+0000 7f1244c00640 1 -- 172.21.15.119:0/1413164734 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1240104ab0 msgr2=0x7f1240104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.394+0000 7f1244c00640 1 --2- 172.21.15.119:0/1413164734 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1240104ab0 0x7f1240104f30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1228009900 tx=0x7f122802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 -- 172.21.15.119:0/1413164734 shutdown_connections 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 --2- 172.21.15.119:0/1413164734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 0x7f124010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 --2- 172.21.15.119:0/1413164734 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1240104ab0 0x7f1240104f30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 --2- 172.21.15.119:0/1413164734 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12401038c0 0x7f1240103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 -- 172.21.15.119:0/1413164734 >> 172.21.15.119:0/1413164734 conn(0x7f12400fee50 msgr2=0x7f1240101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 -- 172.21.15.119:0/1413164734 shutdown_connections 2024-03-28T18:43:32.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.395+0000 7f1244c00640 1 -- 172.21.15.119:0/1413164734 wait complete. 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.396+0000 7f1244c00640 1 Processor -- start 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.396+0000 7f1244c00640 1 -- start start 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.396+0000 7f1244c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.396+0000 7f1244c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1240104ab0 0x7f124019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f1244c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 0x7f12401a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f1244c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124010f520 con 0x7f1240105470 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f1244c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124010f3a0 con 0x7f1240104ab0 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f1244c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124010f6a0 con 0x7f12401038c0 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36182/0 (socket says 172.21.15.119:36182) 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 -- 172.21.15.119:0/747926750 learned_addr learned my addr 172.21.15.119:0/747926750 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123e800640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 0x7f12401a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f1237e00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1240104ab0 0x7f124019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 -- 172.21.15.119:0/747926750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1240104ab0 msgr2=0x7f124019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1240104ab0 0x7f124019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 -- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 msgr2=0x7f12401a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 0x7f12401a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.397+0000 7f123de00640 1 -- 172.21.15.119:0/747926750 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12401a1790 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f123de00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f123000e9c0 tx=0x7f123000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123000cd80 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12300035f0 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230010690 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12401a1a80 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.398+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12401a1fa0 con 0x7f12401038c0 2024-03-28T18:43:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.399+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124010abe0 con 0x7f12401038c0 2024-03-28T18:43:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.401+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f12300037a0 con 0x7f12401038c0 2024-03-28T18:43:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.402+0000 7f1236a00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 0x7f12240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.402+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f12300ceef0 con 0x7f12401038c0 2024-03-28T18:43:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.403+0000 7f1237e00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 0x7f12240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.403+0000 7f1237e00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 0x7f12240a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f122802c7b0 tx=0x7f1228005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.406+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f123009beb0 con 0x7f12401038c0 2024-03-28T18:43:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:32 smithi196 ceph-mon[36300]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:32 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/300961556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:32 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1218223650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:32 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/931557224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:32 smithi131 ceph-mon[36065]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:32 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/300961556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:32 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1218223650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:32 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/931557224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:32 smithi119 ceph-mon[31813]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:32 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/300961556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:32 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1218223650' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:32 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/931557224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:32.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.605+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12400622a0 con 0x7f12401038c0 2024-03-28T18:43:32.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.606+0000 7f1236a00640 1 -- 172.21.15.119:0/747926750 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f123009b7b0 con 0x7f12401038c0 2024-03-28T18:43:32.606 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 msgr2=0x7f12240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 0x7f12240a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f122802c7b0 tx=0x7f1228005fd0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 msgr2=0x7f1240199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f123000e9c0 tx=0x7f123000ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 shutdown_connections 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f12240a2800 0x7f12240a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1240105470 0x7f12401a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1240104ab0 0x7f124019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 --2- 172.21.15.119:0/747926750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12401038c0 0x7f1240199b70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.608+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 >> 172.21.15.119:0/747926750 conn(0x7f12400fee50 msgr2=0x7f1240107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.609+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 shutdown_connections 2024-03-28T18:43:32.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.609+0000 7f1244c00640 1 -- 172.21.15.119:0/747926750 wait complete. 2024-03-28T18:43:32.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 -- 172.21.15.119:0/4111142744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 msgr2=0x7f2e640784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 --2- 172.21.15.119:0/4111142744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e640784e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c009be0 tx=0x7f2e5c02bef0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 -- 172.21.15.119:0/4111142744 shutdown_connections 2024-03-28T18:43:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 --2- 172.21.15.119:0/4111142744 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e640784e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 --2- 172.21.15.119:0/4111142744 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e640772a0 0x7f2e64077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 --2- 172.21.15.119:0/4111142744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e64076060 0x7f2e64076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 -- 172.21.15.119:0/4111142744 >> 172.21.15.119:0/4111142744 conn(0x7f2e640fee50 msgr2=0x7f2e64101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 -- 172.21.15.119:0/4111142744 shutdown_connections 2024-03-28T18:43:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.840+0000 7f2e6a400640 1 -- 172.21.15.119:0/4111142744 wait complete. 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 Processor -- start 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 -- start start 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e640772a0 0x7f2e6419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e641a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64113950 con 0x7f2e640772a0 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.841+0000 7f2e6a400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e641137d0 con 0x7f2e64077c40 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e6a400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64113ad0 con 0x7f2e64076060 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36206/0 (socket says 172.21.15.119:36206) 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 -- 172.21.15.119:0/3489769233 learned_addr learned my addr 172.21.15.119:0/3489769233 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63e00640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e641a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e62a00640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e640772a0 0x7f2e6419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 -- 172.21.15.119:0/3489769233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 msgr2=0x7f2e641a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e641a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 -- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e640772a0 msgr2=0x7f2e6419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e640772a0 0x7f2e6419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 -- 172.21.15.119:0/3489769233 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e641a5b50 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.842+0000 7f2e63400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400e9f0 tx=0x7f2e5400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.843+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5400cd80 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.843+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e54010910 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.843+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e641a5e40 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.843+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54018990 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.843+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e641a62a0 con 0x7f2e64076060 2024-03-28T18:43:32.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.844+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64110230 con 0x7f2e64076060 2024-03-28T18:43:32.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.846+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2e54010ac0 con 0x7f2e64076060 2024-03-28T18:43:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.847+0000 7f2e5be00640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.847+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f2e540cefe0 con 0x7f2e64076060 2024-03-28T18:43:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.847+0000 7f2e62a00640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:32.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.848+0000 7f2e62a00640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 0x7f2e480a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2e6419f440 tx=0x7f2e50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:32.854+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2e5409b950 con 0x7f2e64076060 2024-03-28T18:43:33.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.092+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e6402d070 con 0x7f2e64076060 2024-03-28T18:43:33.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.094+0000 7f2e5be00640 1 -- 172.21.15.119:0/3489769233 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f2e5409f9f0 con 0x7f2e64076060 2024-03-28T18:43:33.094 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:33.094 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:43:33.094 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.096+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 msgr2=0x7f2e480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.096+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 0x7f2e480a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2e6419f440 tx=0x7f2e50009260 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.096+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 msgr2=0x7f2e6419df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.096+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400e9f0 tx=0x7f2e5400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 shutdown_connections 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2e480a2800 0x7f2e480a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e64077c40 0x7f2e641a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e640772a0 0x7f2e6419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 --2- 172.21.15.119:0/3489769233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e64076060 0x7f2e6419df30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 >> 172.21.15.119:0/3489769233 conn(0x7f2e640fee50 msgr2=0x7f2e64100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 shutdown_connections 2024-03-28T18:43:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.097+0000 7f2e6a400640 1 -- 172.21.15.119:0/3489769233 wait complete. 2024-03-28T18:43:33.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 -- 172.21.15.119:0/2873915020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f01016d0 msgr2=0x7fa8f0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 --2- 172.21.15.119:0/2873915020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f01016d0 0x7fa8f0101af0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009860 tx=0x7fa8e402bb70 comp rx=0 tx=0).stop 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 -- 172.21.15.119:0/2873915020 shutdown_connections 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 --2- 172.21.15.119:0/2873915020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f0103280 0x7fa8f0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 --2- 172.21.15.119:0/2873915020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 --2- 172.21.15.119:0/2873915020 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f01016d0 0x7fa8f0101af0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 -- 172.21.15.119:0/2873915020 >> 172.21.15.119:0/2873915020 conn(0x7fa8f00fcc40 msgr2=0x7fa8f00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:33.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 -- 172.21.15.119:0/2873915020 shutdown_connections 2024-03-28T18:43:33.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.325+0000 7fa8f8000640 1 -- 172.21.15.119:0/2873915020 wait complete. 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.326+0000 7fa8f8000640 1 Processor -- start 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.326+0000 7fa8f8000640 1 -- start start 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 0x7fa8f0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f0103280 0x7fa8f0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f010d340 con 0x7fa8f01016d0 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f010d1c0 con 0x7fa8f0103280 2024-03-28T18:43:33.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f8000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f010d4c0 con 0x7fa8f01028c0 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f5600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 0x7fa8f0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36218/0 (socket says 172.21.15.119:36218) 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 -- 172.21.15.119:0/1199498919 learned_addr learned my addr 172.21.15.119:0/1199498919 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f6000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f0103280 0x7fa8f0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 -- 172.21.15.119:0/1199498919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f0103280 msgr2=0x7fa8f0073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f0103280 0x7fa8f0073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 -- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 msgr2=0x7fa8f0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 0x7fa8f0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f4c00640 1 -- 172.21.15.119:0/1199498919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f0074860 con 0x7fa8f01028c0 2024-03-28T18:43:33.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.327+0000 7fa8f5600640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 0x7fa8f0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8f4c00640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9f0 tx=0x7fa8dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc00ccc0 con 0x7fa8f01028c0 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8dc005500 con 0x7fa8f01028c0 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f01a1b40 con 0x7fa8f01028c0 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc010690 con 0x7fa8f01028c0 2024-03-28T18:43:33.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.328+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f01a2040 con 0x7fa8f01028c0 2024-03-28T18:43:33.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.329+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f0108a70 con 0x7fa8f01028c0 2024-03-28T18:43:33.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.332+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa8dc0046c0 con 0x7fa8f01028c0 2024-03-28T18:43:33.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.333+0000 7fa8ee000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 0x7fa8d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.333+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fa8dc0cee50 con 0x7fa8f01028c0 2024-03-28T18:43:33.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.333+0000 7fa8f5600640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 0x7fa8d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.334+0000 7fa8f5600640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 0x7fa8d80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40349e0 tx=0x7fa8e402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:33.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.337+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8dc09be10 con 0x7fa8f01028c0 2024-03-28T18:43:33.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:33 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/747926750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:33.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:33 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3489769233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.538+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f002d270 con 0x7fa8f01028c0 2024-03-28T18:43:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.540+0000 7fa8ee000640 1 -- 172.21.15.119:0/1199498919 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7fa8dc09b710 con 0x7fa8f01028c0 2024-03-28T18:43:33.540 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:33.540 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.542+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 msgr2=0x7fa8d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.542+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 0x7fa8d80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40349e0 tx=0x7fa8e402e040 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 msgr2=0x7fa8f0073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00e9f0 tx=0x7fa8dc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 shutdown_connections 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa8d80a2800 0x7fa8d80a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8f0103280 0x7fa8f0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8f01028c0 0x7fa8f0073a90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 --2- 172.21.15.119:0/1199498919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8f01016d0 0x7fa8f0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 >> 172.21.15.119:0/1199498919 conn(0x7fa8f00fcc40 msgr2=0x7fa8f0105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 shutdown_connections 2024-03-28T18:43:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.543+0000 7fa8f8000640 1 -- 172.21.15.119:0/1199498919 wait complete. 2024-03-28T18:43:33.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:33 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/747926750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:33.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:33 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3489769233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:33.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:33.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:33.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:33 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/747926750' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:33.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:33 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3489769233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:33.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 -- 172.21.15.119:0/1642285629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c104ad0 msgr2=0x7fdf8c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 --2- 172.21.15.119:0/1642285629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c104f50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009900 tx=0x7fdf7402bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:33.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 -- 172.21.15.119:0/1642285629 shutdown_connections 2024-03-28T18:43:33.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 --2- 172.21.15.119:0/1642285629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c105490 0x7fdf8c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 --2- 172.21.15.119:0/1642285629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c104f50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 --2- 172.21.15.119:0/1642285629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 -- 172.21.15.119:0/1642285629 >> 172.21.15.119:0/1642285629 conn(0x7fdf8c0fee50 msgr2=0x7fdf8c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 -- 172.21.15.119:0/1642285629 shutdown_connections 2024-03-28T18:43:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.771+0000 7fdf93000640 1 -- 172.21.15.119:0/1642285629 wait complete. 2024-03-28T18:43:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.772+0000 7fdf93000640 1 Processor -- start 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.772+0000 7fdf93000640 1 -- start start 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.772+0000 7fdf93000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf93000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf93000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c105490 0x7fdf8c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf93000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c10f5b0 con 0x7fdf8c105490 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf93000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c10f430 con 0x7fdf8c1038e0 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf93000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c10f730 con 0x7fdf8c104ad0 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57424/0 (socket says 172.21.15.119:57424) 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36236/0 (socket says 172.21.15.119:36236) 2024-03-28T18:43:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf8be00640 1 -- 172.21.15.119:0/3371553283 learned_addr learned my addr 172.21.15.119:0/3371553283 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf91000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c105490 0x7fdf8c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 -- 172.21.15.119:0/3371553283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 msgr2=0x7fdf8c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 -- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c105490 msgr2=0x7fdf8c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c105490 0x7fdf8c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 -- 172.21.15.119:0/3371553283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c1a1810 con 0x7fdf8c104ad0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.773+0000 7fdf83400640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf74036040 tx=0x7fdf7402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.774+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74043020 con 0x7fdf8c104ad0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.774+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c1a1aa0 con 0x7fdf8c104ad0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.774+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf740054e0 con 0x7fdf8c104ad0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.774+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003c80 con 0x7fdf8c104ad0 2024-03-28T18:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.774+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c1a1fc0 con 0x7fdf8c104ad0 2024-03-28T18:43:33.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.775+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c10acd0 con 0x7fdf8c104ad0 2024-03-28T18:43:33.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.777+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdf74046020 con 0x7fdf8c104ad0 2024-03-28T18:43:33.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.778+0000 7fdf89600640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:33.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.778+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fdf740f0b80 con 0x7fdf8c104ad0 2024-03-28T18:43:33.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.778+0000 7fdf8be00640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:33.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.779+0000 7fdf8be00640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c005fd0 tx=0x7fdf7c00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:33.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.786+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf7404d020 con 0x7fdf8c104ad0 2024-03-28T18:43:33.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.985+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf8c0622a0 con 0x7fdf8c104ad0 2024-03-28T18:43:33.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.986+0000 7fdf89600640 1 -- 172.21.15.119:0/3371553283 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf740bdb40 con 0x7fdf8c104ad0 2024-03-28T18:43:33.986 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:33.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 msgr2=0x7fdf700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c005fd0 tx=0x7fdf7c00b4a0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 msgr2=0x7fdf8c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf74036040 tx=0x7fdf7402c6c0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 shutdown_connections 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf8c105490 0x7fdf8c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf8c104ad0 0x7fdf8c19a130 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 --2- 172.21.15.119:0/3371553283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdf8c1038e0 0x7fdf8c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.988+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 >> 172.21.15.119:0/3371553283 conn(0x7fdf8c0fee50 msgr2=0x7fdf8c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.989+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 shutdown_connections 2024-03-28T18:43:33.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:33.989+0000 7fdf93000640 1 -- 172.21.15.119:0/3371553283 wait complete. 2024-03-28T18:43:34.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 -- 172.21.15.119:0/39551183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 msgr2=0x7fb5c0076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/39551183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c0076480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009900 tx=0x7fb5b002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 -- 172.21.15.119:0/39551183 shutdown_connections 2024-03-28T18:43:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/39551183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/39551183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c0077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/39551183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c0076480 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 -- 172.21.15.119:0/39551183 >> 172.21.15.119:0/39551183 conn(0x7fb5c00fee50 msgr2=0x7fb5c0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 -- 172.21.15.119:0/39551183 shutdown_connections 2024-03-28T18:43:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.217+0000 7fb5c7c00640 1 -- 172.21.15.119:0/39551183 wait complete. 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 Processor -- start 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 -- start start 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0113970 con 0x7fb5c0076060 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c01137f0 con 0x7fb5c0077c40 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.218+0000 7fb5c7c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0113af0 con 0x7fb5c00772a0 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36262/0 (socket says 172.21.15.119:36262) 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5b7e00640 1 -- 172.21.15.119:0/1327798596 learned_addr learned my addr 172.21.15.119:0/1327798596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 -- 172.21.15.119:0/1327798596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 msgr2=0x7fb5c019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 -- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 msgr2=0x7fb5c019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c019e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 -- 172.21.15.119:0/1327798596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c01a5c70 con 0x7fb5c0077c40 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5b7e00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.219+0000 7fb5c5c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c01a5600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800ec50 tx=0x7fb5b800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.220+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8003b70 con 0x7fb5c0077c40 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.220+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5b8003d10 con 0x7fb5c0077c40 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.220+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8010890 con 0x7fb5c0077c40 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.220+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c01a5f60 con 0x7fb5c0077c40 2024-03-28T18:43:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.220+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c01a63d0 con 0x7fb5c0077c40 2024-03-28T18:43:34.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.221+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c0110250 con 0x7fb5c0077c40 2024-03-28T18:43:34.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.222+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb5b8003fc0 con 0x7fb5c0077c40 2024-03-28T18:43:34.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.224+0000 7fb5be000640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 0x7fb5a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.224+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fb5b80cefe0 con 0x7fb5c0077c40 2024-03-28T18:43:34.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.224+0000 7fb5c5200640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 0x7fb5a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.225+0000 7fb5c5200640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 0x7fb5a40a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002c240 tx=0x7fb5b002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:34.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.228+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5b809c000 con 0x7fb5c0077c40 2024-03-28T18:43:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:34 smithi119 ceph-mon[31813]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1199498919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:34.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3371553283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:34.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.431+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5c00622a0 con 0x7fb5c0077c40 2024-03-28T18:43:34.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.432+0000 7fb5be000640 1 -- 172.21.15.119:0/1327798596 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb5b809b900 con 0x7fb5c0077c40 2024-03-28T18:43:34.433 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.434+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 msgr2=0x7fb5a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.434+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 0x7fb5a40a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5b002c240 tx=0x7fb5b002df90 comp rx=0 tx=0).stop 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.434+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 msgr2=0x7fb5c01a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.434+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c01a5600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800ec50 tx=0x7fb5b800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 shutdown_connections 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb5a40a2880 0x7fb5a40a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb5c0077c40 0x7fb5c01a5600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5c00772a0 0x7fb5c019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 --2- 172.21.15.119:0/1327798596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c0076060 0x7fb5c019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 >> 172.21.15.119:0/1327798596 conn(0x7fb5c00fee50 msgr2=0x7fb5c0100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 shutdown_connections 2024-03-28T18:43:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.435+0000 7fb5c7c00640 1 -- 172.21.15.119:0/1327798596 wait complete. 2024-03-28T18:43:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:34 smithi131 ceph-mon[36065]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1199498919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3371553283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:34.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.673+0000 7f0a18600640 1 -- 172.21.15.119:0/820984803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a10104ab0 msgr2=0x7f0a10104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.673+0000 7f0a18600640 1 --2- 172.21.15.119:0/820984803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a10104ab0 0x7f0a10104f30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009900 tx=0x7f09fc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 -- 172.21.15.119:0/820984803 shutdown_connections 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 --2- 172.21.15.119:0/820984803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 0x7f0a1010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 --2- 172.21.15.119:0/820984803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a10104ab0 0x7f0a10104f30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 --2- 172.21.15.119:0/820984803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a101038c0 0x7f0a10103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 -- 172.21.15.119:0/820984803 >> 172.21.15.119:0/820984803 conn(0x7f0a100fee50 msgr2=0x7f0a10101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 -- 172.21.15.119:0/820984803 shutdown_connections 2024-03-28T18:43:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.674+0000 7f0a18600640 1 -- 172.21.15.119:0/820984803 wait complete. 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.675+0000 7f0a18600640 1 Processor -- start 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.675+0000 7f0a18600640 1 -- start start 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.675+0000 7f0a18600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.675+0000 7f0a18600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a10104ab0 0x7f0a1019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a18600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 0x7f0a101a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a18600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1010f590 con 0x7f0a10104ab0 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a18600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1010f410 con 0x7f0a101038c0 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a18600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1010f710 con 0x7f0a10105470 2024-03-28T18:43:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57466/0 (socket says 172.21.15.119:57466) 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 -- 172.21.15.119:0/2695264966 learned_addr learned my addr 172.21.15.119:0/2695264966 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15200640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a10104ab0 0x7f0a1019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a16600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 0x7f0a101a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 -- 172.21.15.119:0/2695264966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 msgr2=0x7f0a101a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 0x7f0a101a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 -- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a10104ab0 msgr2=0x7f0a1019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a10104ab0 0x7f0a1019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.676+0000 7f0a15c00640 1 -- 172.21.15.119:0/2695264966 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a101a17c0 con 0x7f0a101038c0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a15c00640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400e9c0 tx=0x7f0a0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0400cd80 con 0x7f0a101038c0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a040035f0 con 0x7f0a101038c0 2024-03-28T18:43:34.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04010690 con 0x7f0a101038c0 2024-03-28T18:43:34.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a101a1ab0 con 0x7f0a101038c0 2024-03-28T18:43:34.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.677+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a101a1f80 con 0x7f0a101038c0 2024-03-28T18:43:34.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.678+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1010abe0 con 0x7f0a101038c0 2024-03-28T18:43:34.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.680+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a040037a0 con 0x7f0a101038c0 2024-03-28T18:43:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.681+0000 7f0a0e000640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 0x7f09f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.682+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0a040cee50 con 0x7f0a101038c0 2024-03-28T18:43:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.682+0000 7f0a15200640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 0x7f09f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:34.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.682+0000 7f0a15200640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 0x7f09f80a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09fc02c240 tx=0x7f09fc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:34 smithi196 ceph-mon[36300]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1199498919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3371553283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.685+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a0409be10 con 0x7f0a101038c0 2024-03-28T18:43:34.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.883+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a100622d0 con 0x7f0a101038c0 2024-03-28T18:43:34.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.884+0000 7f0a0e000640 1 -- 172.21.15.119:0/2695264966 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a0409b710 con 0x7f0a101038c0 2024-03-28T18:43:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 msgr2=0x7f09f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 0x7f09f80a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09fc02c240 tx=0x7f09fc02df90 comp rx=0 tx=0).stop 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 msgr2=0x7f0a10199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400e9c0 tx=0x7f0a0400ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 shutdown_connections 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09f80a2880 0x7f09f80a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a10105470 0x7f0a101a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a10104ab0 0x7f0a1019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 --2- 172.21.15.119:0/2695264966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a101038c0 0x7f0a10199ba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 >> 172.21.15.119:0/2695264966 conn(0x7f0a100fee50 msgr2=0x7f0a10107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 shutdown_connections 2024-03-28T18:43:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:34.887+0000 7f0a18600640 1 -- 172.21.15.119:0/2695264966 wait complete. 2024-03-28T18:43:34.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:35.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 -- 172.21.15.119:0/1436355413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714077c40 msgr2=0x7fc7140784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 --2- 172.21.15.119:0/1436355413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714077c40 0x7fc7140784e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009ca0 tx=0x7fc70c02c100 comp rx=0 tx=0).stop 2024-03-28T18:43:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 -- 172.21.15.119:0/1436355413 shutdown_connections 2024-03-28T18:43:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 --2- 172.21.15.119:0/1436355413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714077c40 0x7fc7140784e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 --2- 172.21.15.119:0/1436355413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 0x7fc714077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 --2- 172.21.15.119:0/1436355413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714076060 0x7fc714076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.116+0000 7fc71ae00640 1 -- 172.21.15.119:0/1436355413 >> 172.21.15.119:0/1436355413 conn(0x7fc7140fee90 msgr2=0x7fc7141012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.117+0000 7fc71ae00640 1 -- 172.21.15.119:0/1436355413 shutdown_connections 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.117+0000 7fc71ae00640 1 -- 172.21.15.119:0/1436355413 wait complete. 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.117+0000 7fc71ae00640 1 Processor -- start 2024-03-28T18:43:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 -- start start 2024-03-28T18:43:35.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 0x7fc71419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 0x7fc71419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714113970 con 0x7fc7140772a0 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141137f0 con 0x7fc714077c40 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc71ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714113af0 con 0x7fc714076060 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc713e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 0x7fc71419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc718e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc70b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 0x7fc71419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc718e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57478/0 (socket says 172.21.15.119:57478) 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc713e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 0x7fc71419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36294/0 (socket says 172.21.15.119:36294) 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.118+0000 7fc718e00640 1 -- 172.21.15.119:0/103015880 learned_addr learned my addr 172.21.15.119:0/103015880 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 -- 172.21.15.119:0/103015880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 msgr2=0x7fc71419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 0x7fc71419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 -- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 msgr2=0x7fc71419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 0x7fc71419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 -- 172.21.15.119:0/103015880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7141a5bb0 con 0x7fc714077c40 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc718e00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc70c02c640 tx=0x7fc70c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c039040 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc70c044070 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7141a5e40 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0043c0 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.119+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7141a6300 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.120+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7141103e0 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.122+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc70c004560 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.123+0000 7fc711600640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 0x7fc6f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.124+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fc70c0f1070 con 0x7fc714077c40 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.124+0000 7fc713e00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 0x7fc6f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.124+0000 7fc713e00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 0x7fc6f80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc704009bd0 tx=0x7fc704005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.127+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc70c0be030 con 0x7fc714077c40 2024-03-28T18:43:35.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.368+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc71402d070 con 0x7fc714077c40 2024-03-28T18:43:35.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.369+0000 7fc711600640 1 -- 172.21.15.119:0/103015880 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fc70c0098b0 con 0x7fc714077c40 2024-03-28T18:43:35.369 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:35.369 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:41:28.190103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24538,24517,24563,24515,24505]},"id":1}]} 2024-03-28T18:43:35.370 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 27 2024-03-28T18:43:35.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.371+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 msgr2=0x7fc6f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.371+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 0x7fc6f80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc704009bd0 tx=0x7fc704005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.371+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 msgr2=0x7fc7141a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.371+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc70c02c640 tx=0x7fc70c035d10 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 shutdown_connections 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc6f80a2880 0x7fc6f80a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc714077c40 0x7fc7141a5540 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7140772a0 0x7fc71419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 --2- 172.21.15.119:0/103015880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc714076060 0x7fc71419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 >> 172.21.15.119:0/103015880 conn(0x7fc7140fee90 msgr2=0x7fc714100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 shutdown_connections 2024-03-28T18:43:35.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.372+0000 7fc71ae00640 1 -- 172.21.15.119:0/103015880 wait complete. 2024-03-28T18:43:35.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:35.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:35 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1327798596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:35 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2695264966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 -- 172.21.15.119:0/3807530233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89400772a0 msgr2=0x7f8940077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 --2- 172.21.15.119:0/3807530233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89400772a0 0x7f8940077700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8928009900 tx=0x7f892802bc50 comp rx=0 tx=0).stop 2024-03-28T18:43:35.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 -- 172.21.15.119:0/3807530233 shutdown_connections 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 --2- 172.21.15.119:0/3807530233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89400784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 --2- 172.21.15.119:0/3807530233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f89400772a0 0x7f8940077700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 --2- 172.21.15.119:0/3807530233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8940076060 0x7f8940076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 -- 172.21.15.119:0/3807530233 >> 172.21.15.119:0/3807530233 conn(0x7f89400fee50 msgr2=0x7f8940101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 -- 172.21.15.119:0/3807530233 shutdown_connections 2024-03-28T18:43:35.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.601+0000 7f8946200640 1 -- 172.21.15.119:0/3807530233 wait complete. 2024-03-28T18:43:35.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 Processor -- start 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 -- start start 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89400772a0 0x7f894019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89401a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401139b0 con 0x7f89400772a0 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940113830 con 0x7f8940076060 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.602+0000 7f8946200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940113b30 con 0x7f8940077c40 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f8936a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89400772a0 0x7f894019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57496/0 (socket says 172.21.15.119:57496) 2024-03-28T18:43:35.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 -- 172.21.15.119:0/443410213 learned_addr learned my addr 172.21.15.119:0/443410213 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893fe00640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89401a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 -- 172.21.15.119:0/443410213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 msgr2=0x7f89401a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89401a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 -- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89400772a0 msgr2=0x7f894019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89400772a0 0x7f894019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 -- 172.21.15.119:0/443410213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89401a5c00 con 0x7f8940076060 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893fe00640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89401a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.603+0000 7f893f400640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f893000e9f0 tx=0x7f893000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.604+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000ccc0 con 0x7f8940076060 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.604+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8930010910 con 0x7f8940076060 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.604+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300188a0 con 0x7f8940076060 2024-03-28T18:43:35.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.604+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89401a5ef0 con 0x7f8940076060 2024-03-28T18:43:35.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.604+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401a6300 con 0x7f8940076060 2024-03-28T18:43:35.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.605+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89401103f0 con 0x7f8940076060 2024-03-28T18:43:35.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.607+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8930010ac0 con 0x7f8940076060 2024-03-28T18:43:35.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.608+0000 7f893cc00640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 0x7f89240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:35.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.608+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f89300cee50 con 0x7f8940076060 2024-03-28T18:43:35.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.608+0000 7f8936a00640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 0x7f89240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:35.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.609+0000 7f8936a00640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 0x7f89240a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f892802c260 tx=0x7f8928000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:35.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.612+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f893009be70 con 0x7f8940076060 2024-03-28T18:43:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:35 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1327798596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:35 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2695264966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:35 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1327798596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:35 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2695264966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:35.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.811+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89400623a0 con 0x7f8940076060 2024-03-28T18:43:35.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.813+0000 7f893cc00640 1 -- 172.21.15.119:0/443410213 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f893009b770 con 0x7f8940076060 2024-03-28T18:43:35.813 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:35.814 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:35.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 msgr2=0x7f89240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 0x7f89240a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f892802c260 tx=0x7f8928000bc0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 msgr2=0x7f894019dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f893000e9f0 tx=0x7f893000edf0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 shutdown_connections 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f89240a2880 0x7f89240a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8940077c40 0x7f89401a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89400772a0 0x7f894019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 --2- 172.21.15.119:0/443410213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8940076060 0x7f894019dfe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:35.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 >> 172.21.15.119:0/443410213 conn(0x7f89400fee50 msgr2=0x7f8940100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:35.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 shutdown_connections 2024-03-28T18:43:35.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:35.816+0000 7f8946200640 1 -- 172.21.15.119:0/443410213 wait complete. 2024-03-28T18:43:35.836 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T18:41:20.510917+0000', 'modified': '2024-03-28T18:41:28.190103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24563, 'mds_1': 24538, 'mds_2': 24517, 'mds_3': 24515, 'mds_4': 24505}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14448': {'gid': 14448, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6839/1834295684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 1834295684}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 1834295684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14460': {'gid': 14460, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6841/683166305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6840', 'nonce': 683166305}, {'type': 'v1', 'addr': '172.21.15.119:6841', 'nonce': 683166305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6833/4160748351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6832', 'nonce': 4160748351}, {'type': 'v1', 'addr': '172.21.15.131:6833', 'nonce': 4160748351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6833/1244574561', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6832', 'nonce': 1244574561}, {'type': 'v1', 'addr': '172.21.15.196:6833', 'nonce': 1244574561}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24485': {'gid': 24485, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.131:6835/1538863538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6834', 'nonce': 1538863538}, {'type': 'v1', 'addr': '172.21.15.131:6835', 'nonce': 1538863538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24505': {'gid': 24505, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.196:6835/480911404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 480911404}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 480911404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.131:6837/2241845025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6836', 'nonce': 2241845025}, {'type': 'v1', 'addr': '172.21.15.131:6837', 'nonce': 2241845025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.196:6837/4226502296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 4226502296}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 4226502296}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.196:6839/2667663000', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2667663000}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2667663000}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24563': {'gid': 24563, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.131:6839/2855046426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.131:6838', 'nonce': 2855046426}, {'type': 'v1', 'addr': '172.21.15.131:6839', 'nonce': 2855046426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24563, 'qdb_cluster': [24538, 24517, 24563, 24515, 24505]} 2024-03-28T18:43:35.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T18:43:35.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 -- 172.21.15.119:0/1651000343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d8103280 msgr2=0x7f20d8109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 --2- 172.21.15.119:0/1651000343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d8103280 0x7f20d8109b10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009860 tx=0x7f20d002bb70 comp rx=0 tx=0).stop 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 -- 172.21.15.119:0/1651000343 shutdown_connections 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 --2- 172.21.15.119:0/1651000343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d8103280 0x7f20d8109b10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 --2- 172.21.15.119:0/1651000343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81028c0 0x7f20d8102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 --2- 172.21.15.119:0/1651000343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d81016d0 0x7f20d8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 -- 172.21.15.119:0/1651000343 >> 172.21.15.119:0/1651000343 conn(0x7f20d80fcc40 msgr2=0x7f20d80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 -- 172.21.15.119:0/1651000343 shutdown_connections 2024-03-28T18:43:36.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.056+0000 7f20dde00640 1 -- 172.21.15.119:0/1651000343 wait complete. 2024-03-28T18:43:36.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 Processor -- start 2024-03-28T18:43:36.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 -- start start 2024-03-28T18:43:36.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81016d0 0x7f20d8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d8103280 0x7f20d81a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d810d330 con 0x7f20d8103280 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d810d1b0 con 0x7f20d81028c0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20dde00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d810d4b0 con 0x7f20d81016d0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57512/0 (socket says 172.21.15.119:57512) 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81016d0 0x7f20d8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20d6000640 1 -- 172.21.15.119:0/3303153827 learned_addr learned my addr 172.21.15.119:0/3303153827 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.057+0000 7f20d7400640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d8103280 0x7f20d81a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 -- 172.21.15.119:0/3303153827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81016d0 msgr2=0x7f20d8199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81016d0 0x7f20d8199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 -- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d8103280 msgr2=0x7f20d81a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d8103280 0x7f20d81a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 -- 172.21.15.119:0/3303153827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d81a1790 con 0x7f20d81028c0 2024-03-28T18:43:36.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20d6000640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20c000efb0 tx=0x7f20c000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20dca00640 1 -- 172.21.15.119:0/3303153827 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c001b070 con 0x7f20d81028c0 2024-03-28T18:43:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20dca00640 1 -- 172.21.15.119:0/3303153827 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20c000ba50 con 0x7f20d81028c0 2024-03-28T18:43:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20dca00640 1 -- 172.21.15.119:0/3303153827 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0016c00 con 0x7f20d81028c0 2024-03-28T18:43:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d81a1a80 con 0x7f20d81028c0 2024-03-28T18:43:36.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.058+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d81a1f80 con 0x7f20d81028c0 2024-03-28T18:43:36.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.061+0000 7f20e0bea740 1 Processor -- start 2024-03-28T18:43:36.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 -- start start 2024-03-28T18:43:36.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20dca00640 1 -- 172.21.15.119:0/3303153827 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f20c001e070 con 0x7f20d81028c0 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d4590 0x5635751d9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751dbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563575138c90 con 0x5635751d4590 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635750e1980 con 0x5635751d3ba0 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20e0bea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563575134d30 con 0x5635751d9700 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20d6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751dbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57528/0 (socket says 172.21.15.119:57528) 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d4590 0x5635751d9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.062+0000 7f20d7400640 1 -- 172.21.15.119:0/3189649252 learned_addr learned my addr 172.21.15.119:0/3189649252 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 -- 172.21.15.119:0/3189649252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 msgr2=0x5635751dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751dbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 -- 172.21.15.119:0/3189649252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d4590 msgr2=0x5635751d9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d4590 0x5635751d9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 -- 172.21.15.119:0/3189649252 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635750f9510 con 0x5635751d3ba0 2024-03-28T18:43:36.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20d7400640 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f20d0034b10 tx=0x7f20d0005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=758305458393d75f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20c7e00640 1 -- 172.21.15.119:0/3189649252 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0043020 con 0x5635751d3ba0 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20c7e00640 1 -- 172.21.15.119:0/3189649252 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20d0034db0 con 0x5635751d3ba0 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20c7e00640 1 -- 172.21.15.119:0/3189649252 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0039040 con 0x5635751d3ba0 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.063+0000 7f20dca00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 0x7f20b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20dca00640 1 -- 172.21.15.119:0/3303153827 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f20c00ceac0 con 0x7f20d81028c0 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20d6a00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 0x7f20b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 -- 172.21.15.119:0/3189649252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 msgr2=0x5635751d3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f20d0034b10 tx=0x7f20d0005b20 comp rx=0 tx=0).stop 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 -- 172.21.15.119:0/3189649252 shutdown_connections 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751dbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d4590 0x5635751d9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 --2- 172.21.15.119:0/3189649252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d3fc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 -- 172.21.15.119:0/3189649252 >> 172.21.15.119:0/3189649252 conn(0x5635751cf110 msgr2=0x5635751cf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 -- 172.21.15.119:0/3189649252 shutdown_connections 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20e0bea740 1 -- 172.21.15.119:0/3189649252 wait complete. 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.064+0000 7f20d6a00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 0x7f20b80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5635751d0080 tx=0x7f20c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 -- start start 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 Processor -- start 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d3ba0 0x5635751de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751e5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751dc650 con 0x5635751d3ba0 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635750f4dd0 con 0x5635751d4590 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751dc7d0 con 0x5635751d9700 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20e0bea740 10 client.0 _ll_register_callbacks cb 0x563574e4cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.065+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d3ba0 0x5635751de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57538/0 (socket says 172.21.15.119:57538) 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 -- 172.21.15.119:0/505509261 learned_addr learned my addr 172.21.15.119:0/505509261 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d7400640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 -- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 msgr2=0x5635751e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 -- 172.21.15.119:0/505509261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d3ba0 msgr2=0x5635751de090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d3ba0 0x5635751de090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 -- 172.21.15.119:0/505509261 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563574e4d2d0 con 0x5635751d4590 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20d6000640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f20c00a35f0 tx=0x7f20c0016660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4268ce3bd3d55a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c009b430 con 0x5635751d4590 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20d0034db0 con 0x5635751d4590 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f20c009f120 con 0x5635751d4590 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20c009f4d0 con 0x5635751d4590 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20e0bea740 10 client.25166 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.066+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563574f176b0 con 0x5635751d4590 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20d0034db0 con 0x5635751d4590 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f20c000b040 con 0x5635751d4590 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 10 client.25166 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 10 client.25166 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 20 client.25166 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5635751ea810 0x5635751e9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 4 client.25166 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635750da520 con 0x5635751ea810 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.067+0000 7f20d6a00640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5635751ea810 0x5635751e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.068+0000 7f20d6a00640 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5635751ea810 0x5635751e9770 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=52fab9bdc98746a3 server_cookie=7cf10139714c8f86 in_seq=0 out_seq=0 2024-03-28T18:43:36.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.068+0000 7f20c4c00640 10 client.25166 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:36.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20c4c00640 1 -- 172.21.15.119:0/505509261 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635750da520 con 0x5635751ea810 2024-03-28T18:43:36.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20c4c00640 10 client.25166 handle_command_reply: tid=0 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 client.25166 shutdown 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 msgr2=0x5635751de5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f20c00a35f0 tx=0x7f20c0016660 comp rx=0 tx=0).stop 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 shutdown_connections 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5635751ea810 0x5635751e9770 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d9700 0x5635751e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751de5d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 --2- 172.21.15.119:0/505509261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d3ba0 0x5635751de090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 >> 172.21.15.119:0/505509261 conn(0x5635751cf110 msgr2=0x5635751ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20c4c00640 0 client.25166 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 shutdown_connections 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 1 -- 172.21.15.119:0/505509261 wait complete. 2024-03-28T18:43:36.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.069+0000 7f20e0bea740 20 client.25166 trim_cache size 0 max 16384 2024-03-28T18:43:36.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 Processor -- start 2024-03-28T18:43:36.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 -- start start 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d4590 0x5635751e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751ea030 con 0x5635751d9700 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751ea690 con 0x5635751d3ba0 2024-03-28T18:43:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20e0bea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751eacf0 con 0x5635751d4590 2024-03-28T18:43:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57540/0 (socket says 172.21.15.119:57540) 2024-03-28T18:43:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.089+0000 7f20d6a00640 1 -- 172.21.15.119:0/970670419 learned_addr learned my addr 172.21.15.119:0/970670419 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d7400640 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6000640 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d4590 0x5635751e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 -- 172.21.15.119:0/970670419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d4590 msgr2=0x5635751e55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d4590 0x5635751e55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 -- 172.21.15.119:0/970670419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 msgr2=0x5635751d2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751d2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 -- 172.21.15.119:0/970670419 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635751d4e20 con 0x5635751d3ba0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20d6a00640 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5635751d4990 tx=0x7f20c8022040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ce04b77bf74a85f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.090+0000 7f20c7400640 1 -- 172.21.15.119:0/970670419 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c80043b0 con 0x5635751d3ba0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20c7400640 1 -- 172.21.15.119:0/970670419 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20c8030070 con 0x5635751d3ba0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20c7400640 1 -- 172.21.15.119:0/970670419 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c801f890 con 0x5635751d3ba0 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 -- 172.21.15.119:0/970670419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 msgr2=0x5635751d90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5635751d4990 tx=0x7f20c8022040 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 -- 172.21.15.119:0/970670419 shutdown_connections 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751d2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d4590 0x5635751e55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 --2- 172.21.15.119:0/970670419 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d3ba0 0x5635751d90c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 -- 172.21.15.119:0/970670419 >> 172.21.15.119:0/970670419 conn(0x5635751cf110 msgr2=0x5635751d2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 -- 172.21.15.119:0/970670419 shutdown_connections 2024-03-28T18:43:36.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.091+0000 7f20e0bea740 1 -- 172.21.15.119:0/970670419 wait complete. 2024-03-28T18:43:36.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 -- start start 2024-03-28T18:43:36.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 Processor -- start 2024-03-28T18:43:36.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751e2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751eb9b0 con 0x5635751d9700 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751eb350 con 0x5635751d4590 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.092+0000 7f20e0bea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635751ec010 con 0x5635751d3ba0 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20e0bea740 10 client.0 _ll_register_callbacks cb 0x7f20c0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36366/0 (socket says 172.21.15.119:36366) 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 -- 172.21.15.119:0/868540414 learned_addr learned my addr 172.21.15.119:0/868540414 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57548/0 (socket says 172.21.15.119:57548) 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d7400640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751e2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 -- 172.21.15.119:0/868540414 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 msgr2=0x5635751e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 -- 172.21.15.119:0/868540414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 msgr2=0x5635751e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751e2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 -- 172.21.15.119:0/868540414 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635751e2940 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6000640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20d6a00640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f20c80041c0 tx=0x7f20c802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45739d89bd0a3415 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.093+0000 7f20c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8038030 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f20c8018070 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20c8030070 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20c8044050 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20e0bea740 10 client.25195 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56357524f0f0 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.094+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20c8030070 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f20c801f9a0 con 0x5635751d3ba0 2024-03-28T18:43:36.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 10 client.25195 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 10 client.25195 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 20 client.25195 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563575253460 0x56357524f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 4 client.25195 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5635751461d0 con 0x563575253460 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20d6000640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563575253460 0x56357524f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20d6000640 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563575253460 0x56357524f3d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9c85aa13a4da2f1 server_cookie=556a660ea2fc49e7 in_seq=0 out_seq=0 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.095+0000 7f20c4c00640 10 client.25195 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20c4c00640 1 -- 172.21.15.119:0/868540414 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5635751461d0 con 0x563575253460 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20c4c00640 10 client.25195 handle_command_reply: tid=0 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 client.25195 shutdown 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 msgr2=0x5635751e5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f20c80041c0 tx=0x7f20c802bec0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 shutdown_connections 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563575253460 0x56357524f3d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635751d9700 0x5635751e2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635751d4590 0x5635751e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20c4c00640 0 client.25195 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 --2- 172.21.15.119:0/868540414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635751d3ba0 0x5635751e5320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 >> 172.21.15.119:0/868540414 conn(0x5635751cf110 msgr2=0x5635751d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 shutdown_connections 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 1 -- 172.21.15.119:0/868540414 wait complete. 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.096+0000 7f20e0bea740 20 client.25195 trim_cache size 0 max 16384 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 4, 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24505, 2024-03-28T18:43:36.100 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 128.91586394399999, 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 161.771550292 2024-03-28T18:43:36.101 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 msgr2=0x7f20b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 0x7f20b80a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5635751d0080 tx=0x7f20c8005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 msgr2=0x7f20d819a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20c000efb0 tx=0x7f20c000c800 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 shutdown_connections 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f20b80a2800 0x7f20b80a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20d8103280 0x7f20d81a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20d81028c0 0x7f20d819a0b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 --2- 172.21.15.119:0/3303153827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20d81016d0 0x7f20d8199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 >> 172.21.15.119:0/3303153827 conn(0x7f20d80fcc40 msgr2=0x7f20d80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 shutdown_connections 2024-03-28T18:43:36.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.097+0000 7f20dde00640 1 -- 172.21.15.119:0/3303153827 wait complete. 2024-03-28T18:43:36.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2624300830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f4105490 msgr2=0x7fe2f410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2624300830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f4105490 0x7fe2f410bd20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009ca0 tx=0x7fe2ec02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2624300830 shutdown_connections 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2624300830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f4105490 0x7fe2f410bd20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2624300830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4104ad0 0x7fe2f4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2624300830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f41038e0 0x7fe2f4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2624300830 >> 172.21.15.119:0/2624300830 conn(0x7fe2f40fee50 msgr2=0x7fe2f4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2624300830 shutdown_connections 2024-03-28T18:43:36.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.304+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2624300830 wait complete. 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.305+0000 7fe2f9a00640 1 Processor -- start 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.305+0000 7fe2f9a00640 1 -- start start 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.305+0000 7fe2f9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f9a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f4104ad0 0x7fe2f419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4105490 0x7fe2f41a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f9a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f410f540 con 0x7fe2f4105490 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f9a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f410f3c0 con 0x7fe2f4104ad0 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f9a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f410f6c0 con 0x7fe2f41038e0 2024-03-28T18:43:36.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36388/0 (socket says 172.21.15.119:36388) 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2027988830 learned_addr learned my addr 172.21.15.119:0/2027988830 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f4104ad0 0x7fe2f419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f3400640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4105490 0x7fe2f41a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2027988830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f4104ad0 msgr2=0x7fe2f419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f4104ad0 0x7fe2f419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4105490 msgr2=0x7fe2f41a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4105490 0x7fe2f41a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2027988830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f41a17b0 con 0x7fe2f41038e0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.306+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400e9c0 tx=0x7fe2e400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.307+0000 7fe2eb400640 1 -- 172.21.15.119:0/2027988830 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400cd80 con 0x7fe2f41038e0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.307+0000 7fe2eb400640 1 -- 172.21.15.119:0/2027988830 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2e40035f0 con 0x7fe2f41038e0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.307+0000 7fe2eb400640 1 -- 172.21.15.119:0/2027988830 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4010690 con 0x7fe2f41038e0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.307+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f41a1aa0 con 0x7fe2f41038e0 2024-03-28T18:43:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.307+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f41a2040 con 0x7fe2f41038e0 2024-03-28T18:43:36.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2eb400640 1 -- 172.21.15.119:0/2027988830 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe2e40037a0 con 0x7fe2f41038e0 2024-03-28T18:43:36.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 Processor -- start 2024-03-28T18:43:36.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 -- start start 2024-03-28T18:43:36.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befc590 0x55ec5bf01090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 0x55ec5bf03b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5be60c90 con 0x55ec5befbba0 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5be09980 con 0x55ec5bf01700 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.310+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5be5cd30 con 0x55ec5befc590 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2f3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2f3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38282/0 (socket says 172.21.15.119:38282) 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2f3400640 1 -- 172.21.15.119:0/1207341674 learned_addr learned my addr 172.21.15.119:0/1207341674 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befc590 0x55ec5bf01090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 0x55ec5bf03b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 -- 172.21.15.119:0/1207341674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befc590 msgr2=0x55ec5bf01090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befc590 0x55ec5bf01090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 -- 172.21.15.119:0/1207341674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 msgr2=0x55ec5befbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 -- 172.21.15.119:0/1207341674 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec5be21510 con 0x55ec5bf01700 2024-03-28T18:43:36.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2f3400640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 0x55ec5bf03b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc005d00 tx=0x7fe2dc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c90557d3d9a797 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.311+0000 7fe2eb400640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 0x7fe2d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2eb400640 1 -- 172.21.15.119:0/2027988830 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe2e40cee70 con 0x7fe2f41038e0 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2e3e00640 1 -- 172.21.15.119:0/1207341674 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc01c480 con 0x55ec5bf01700 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2f3400640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 0x7fe2d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2e3e00640 1 -- 172.21.15.119:0/1207341674 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2dc01cb00 con 0x55ec5bf01700 2024-03-28T18:43:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2e3e00640 1 -- 172.21.15.119:0/1207341674 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc011c30 con 0x55ec5bf01700 2024-03-28T18:43:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2fc78d740 1 -- 172.21.15.119:0/1207341674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 msgr2=0x55ec5bf03b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 0x55ec5bf03b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc005d00 tx=0x7fe2dc0107a0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.312+0000 7fe2f3400640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 0x7fe2d40a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec032420 tx=0x7fe2ec032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 -- 172.21.15.119:0/1207341674 shutdown_connections 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5bf01700 0x55ec5bf03b90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befc590 0x55ec5bf01090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/1207341674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5befbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 -- 172.21.15.119:0/1207341674 >> 172.21.15.119:0/1207341674 conn(0x55ec5bef7110 msgr2=0x55ec5bef7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 -- 172.21.15.119:0/1207341674 shutdown_connections 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 -- 172.21.15.119:0/1207341674 wait complete. 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 -- start start 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.313+0000 7fe2fc78d740 1 Processor -- start 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5bf060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf0d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf046b0 con 0x55ec5befbba0 2024-03-28T18:43:36.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5be1cdd0 con 0x55ec5befc590 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf04830 con 0x55ec5bf01700 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2fc78d740 10 client.0 _ll_register_callbacks cb 0x55ec5bb74ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5bf060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57582/0 (socket says 172.21.15.119:57582) 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 -- 172.21.15.119:0/2300062639 learned_addr learned my addr 172.21.15.119:0/2300062639 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2f3400640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf0d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 -- 172.21.15.119:0/2300062639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 msgr2=0x55ec5bf0d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf0d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 -- 172.21.15.119:0/2300062639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 msgr2=0x55ec5bf060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5bf060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2ebe00640 1 -- 172.21.15.119:0/2300062639 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec5bb752d0 con 0x55ec5befc590 2024-03-28T18:43:36.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.314+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5bf060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc005bc0 tx=0x7fe2dc00bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad40de0c4663125f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc01c690 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe2dc01c830 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2dc01cb00 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2dc004020 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2fc78d740 10 client.25184 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ec5bc3f6b0 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.315+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2dc01cb00 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe2dc0038c0 con 0x55ec5befc590 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 10 client.25184 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 10 client.25184 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 20 client.25184 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf11800 0x55ec5bf16890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 4 client.25184 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:36.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec5be02520 con 0x55ec5bf11800 2024-03-28T18:43:36.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf11800 0x55ec5bf16890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf11800 0x55ec5bf16890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3d912c2515c5d84 server_cookie=8a56f231fb13f1e6 in_seq=0 out_seq=0 2024-03-28T18:43:36.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.316+0000 7fe2e0c00640 10 client.25184 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.317+0000 7fe2e0c00640 1 -- 172.21.15.119:0/2300062639 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec5be02520 con 0x55ec5bf11800 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.317+0000 7fe2e0c00640 10 client.25184 handle_command_reply: tid=0 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.317+0000 7fe2fc78d740 1 client.25184 shutdown 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.317+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 msgr2=0x55ec5bf06600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.317+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc005bc0 tx=0x7fe2dc00bf30 comp rx=0 tx=0).stop 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 shutdown_connections 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf0d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf06600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf11800 0x55ec5bf16890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2300062639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befbba0 0x55ec5bf060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 >> 172.21.15.119:0/2300062639 conn(0x55ec5bef7110 msgr2=0x55ec5bef6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2e0c00640 0 client.25184 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 shutdown_connections 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2300062639 wait complete. 2024-03-28T18:43:36.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.318+0000 7fe2fc78d740 20 client.25184 trim_cache size 0 max 16384 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 Processor -- start 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 -- start start 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 0x55ec5bf0d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf03af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf11c30 con 0x55ec5befc590 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf12290 con 0x55ec5befbba0 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf128f0 con 0x55ec5bf01700 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57584/0 (socket says 172.21.15.119:57584) 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.336+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2407578591 learned_addr learned my addr 172.21.15.119:0/2407578591 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f3400640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf03af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 0x55ec5bf0d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2407578591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 msgr2=0x55ec5bf03af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf03af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2407578591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 msgr2=0x55ec5bf0d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 0x55ec5bf0d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 -- 172.21.15.119:0/2407578591 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec5bf071f0 con 0x55ec5befbba0 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 0x55ec5bf0d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:36.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4097b90 tx=0x7fe2e40cc760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca4633d464d3540 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2e3400640 1 -- 172.21.15.119:0/2407578591 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e409b500 con 0x55ec5befbba0 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2e3400640 1 -- 172.21.15.119:0/2407578591 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2e409be30 con 0x55ec5befbba0 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.337+0000 7fe2e3400640 1 -- 172.21.15.119:0/2407578591 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e409b090 con 0x55ec5befbba0 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2407578591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 msgr2=0x55ec5bf0dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4097b90 tx=0x7fe2e40cc760 comp rx=0 tx=0).stop 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2407578591 shutdown_connections 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5bf01700 0x55ec5bf03af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5befc590 0x55ec5bf0d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/2407578591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befbba0 0x55ec5bf0dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2407578591 >> 172.21.15.119:0/2407578591 conn(0x55ec5bef7110 msgr2=0x55ec5bf04c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2407578591 shutdown_connections 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- 172.21.15.119:0/2407578591 wait complete. 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.338+0000 7fe2fc78d740 1 -- start start 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 Processor -- start 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf08e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5bf01700 0x55ec5bf0be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf135b0 con 0x55ec5bf01700 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf12f50 con 0x55ec5befc590 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec5bf13c10 con 0x55ec5befbba0 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2fc78d740 10 client.0 _ll_register_callbacks cb 0x7fe2e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36416/0 (socket says 172.21.15.119:36416) 2024-03-28T18:43:36.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2f2a00640 1 -- 172.21.15.119:0/970835453 learned_addr learned my addr 172.21.15.119:0/970835453 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2f3400640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5bf01700 0x55ec5bf0be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.339+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf08e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 -- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 msgr2=0x55ec5bf08e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf08e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 -- 172.21.15.119:0/970835453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5bf01700 msgr2=0x55ec5bf0be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5bf01700 0x55ec5bf0be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 -- 172.21.15.119:0/970835453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec5bf0c460 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2f2a00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c6a0 tx=0x7fe2e4099b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94b5d6ca097bd879 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40d7070 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe2e4003b30 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2e409be30 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2e409b270 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2fc78d740 10 client.25213 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ec5bf0d6d0 con 0x55ec5befbba0 2024-03-28T18:43:36.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.340+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2e409be30 con 0x55ec5befbba0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe2e40b4270 con 0x55ec5befbba0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 10 client.25213 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 10 client.25213 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 20 client.25213 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf715c0 0x55ec5bf6d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 4 client.25213 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec5be6e1d0 con 0x55ec5bf715c0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf715c0 0x55ec5bf6d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2ebe00640 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf715c0 0x55ec5bf6d9b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=31883ece098647a7 server_cookie=4b357a1d9af3b8c5 in_seq=0 out_seq=0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2e0c00640 10 client.25213 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2e0c00640 1 -- 172.21.15.119:0/970835453 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ec5be6e1d0 con 0x55ec5bf715c0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2e0c00640 10 client.25213 handle_command_reply: tid=0 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.341+0000 7fe2fc78d740 1 client.25213 shutdown 2024-03-28T18:43:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 msgr2=0x55ec5bf08910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c6a0 tx=0x7fe2e4099b80 comp rx=0 tx=0).stop 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 shutdown_connections 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ec5bf715c0 0x55ec5bf6d9b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec5bf01700 0x55ec5bf0be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec5befc590 0x55ec5bf08e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2e0c00640 0 client.25213 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 --2- 172.21.15.119:0/970835453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec5befbba0 0x55ec5bf08910 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 >> 172.21.15.119:0/970835453 conn(0x55ec5bef7110 msgr2=0x55ec5bf0dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 shutdown_connections 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 1 -- 172.21.15.119:0/970835453 wait complete. 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.342+0000 7fe2fc78d740 20 client.25213 trim_cache size 0 max 16384 2024-03-28T18:43:36.346 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 3, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24515, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 130.18263537799999, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 156.52190761899999 2024-03-28T18:43:36.347 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 msgr2=0x7fe2d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 0x7fe2d40a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec032420 tx=0x7fe2ec032590 comp rx=0 tx=0).stop 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 msgr2=0x7fe2f4199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400e9c0 tx=0x7fe2e400ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 shutdown_connections 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe2d40a2880 0x7fe2d40a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2f4105490 0x7fe2f41a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe2f4104ad0 0x7fe2f419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 --2- 172.21.15.119:0/2027988830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2f41038e0 0x7fe2f4199b90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 >> 172.21.15.119:0/2027988830 conn(0x7fe2f40fee50 msgr2=0x7fe2f4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 shutdown_connections 2024-03-28T18:43:36.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.343+0000 7fe2f9a00640 1 -- 172.21.15.119:0/2027988830 wait complete. 2024-03-28T18:43:36.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T18:43:36.434 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:36 smithi119 ceph-mon[31813]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:36.434 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/103015880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.434 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:36 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/443410213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 -- 172.21.15.119:0/596399585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb100104ab0 msgr2=0x7fb100104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 --2- 172.21.15.119:0/596399585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb100104ab0 0x7fb100104f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009900 tx=0x7fb0ec02bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 -- 172.21.15.119:0/596399585 shutdown_connections 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 --2- 172.21.15.119:0/596399585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100105470 0x7fb10010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 --2- 172.21.15.119:0/596399585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb100104ab0 0x7fb100104f30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 --2- 172.21.15.119:0/596399585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1001038c0 0x7fb100103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.570+0000 7fb108400640 1 -- 172.21.15.119:0/596399585 >> 172.21.15.119:0/596399585 conn(0x7fb1000fee50 msgr2=0x7fb100101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.571+0000 7fb108400640 1 -- 172.21.15.119:0/596399585 shutdown_connections 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.571+0000 7fb108400640 1 -- 172.21.15.119:0/596399585 wait complete. 2024-03-28T18:43:36.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 Processor -- start 2024-03-28T18:43:36.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 -- start start 2024-03-28T18:43:36.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 0x7fb10019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 0x7fb1001a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10010f590 con 0x7fb100105470 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10010f410 con 0x7fb1001038c0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.572+0000 7fb108400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10010f710 con 0x7fb100104ab0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57600/0 (socket says 172.21.15.119:57600) 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 -- 172.21.15.119:0/1724177034 learned_addr learned my addr 172.21.15.119:0/1724177034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105000640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 0x7fb10019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb106400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 0x7fb1001a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 -- 172.21.15.119:0/1724177034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 msgr2=0x7fb10019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 0x7fb10019a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 -- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 msgr2=0x7fb1001a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 0x7fb1001a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 -- 172.21.15.119:0/1724177034 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1001a17d0 con 0x7fb1001038c0 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105000640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 0x7fb10019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb106400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 0x7fb1001a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb105a00640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400d770 tx=0x7fb0f400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.573+0000 7fb0fe000640 1 -- 172.21.15.119:0/1724177034 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4017070 con 0x7fb1001038c0 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.574+0000 7fb0fe000640 1 -- 172.21.15.119:0/1724177034 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0f4009c30 con 0x7fb1001038c0 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.574+0000 7fb0fe000640 1 -- 172.21.15.119:0/1724177034 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4003dd0 con 0x7fb1001038c0 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.574+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1001a1ac0 con 0x7fb1001038c0 2024-03-28T18:43:36.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.574+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1001a1f90 con 0x7fb1001038c0 2024-03-28T18:43:36.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 Processor -- start 2024-03-28T18:43:36.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb0fe000640 1 -- 172.21.15.119:0/1724177034 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb0f4003f70 con 0x7fb1001038c0 2024-03-28T18:43:36.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 -- start start 2024-03-28T18:43:36.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423eba0 0x5561c423efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4244090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c41a3c90 con 0x5561c423eba0 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c414c980 con 0x5561c423f590 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.577+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c419fd30 con 0x5561c4244700 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4244090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb106400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423eba0 0x5561c423efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36438/0 (socket says 172.21.15.119:36438) 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4244090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57604/0 (socket says 172.21.15.119:57604) 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 -- 172.21.15.119:0/3098749850 learned_addr learned my addr 172.21.15.119:0/3098749850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 -- 172.21.15.119:0/3098749850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 msgr2=0x5561c4244090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4244090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 -- 172.21.15.119:0/3098749850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423eba0 msgr2=0x5561c423efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423eba0 0x5561c423efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 -- 172.21.15.119:0/3098749850 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561c4164510 con 0x5561c4244700 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.578+0000 7fb105000640 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec006010 tx=0x7fb0ec000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9486487e25ebd250 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb0f2a00640 1 -- 172.21.15.119:0/3098749850 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec042070 con 0x5561c4244700 2024-03-28T18:43:36.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb0f2a00640 1 -- 172.21.15.119:0/3098749850 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec004ba0 con 0x5561c4244700 2024-03-28T18:43:36.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb0f2a00640 1 -- 172.21.15.119:0/3098749850 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0053c0 con 0x5561c4244700 2024-03-28T18:43:36.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb0fe000640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 0x7fb0e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb0fe000640 1 -- 172.21.15.119:0/1724177034 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fb0f40ce440 con 0x7fb1001038c0 2024-03-28T18:43:36.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb106400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 0x7fb0e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 -- 172.21.15.119:0/3098749850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 msgr2=0x5561c4246b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec006010 tx=0x7fb0ec000bc0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 -- 172.21.15.119:0/3098749850 shutdown_connections 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c4246b90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4244090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/3098749850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423eba0 0x5561c423efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.579+0000 7fb10b1d3740 1 -- 172.21.15.119:0/3098749850 >> 172.21.15.119:0/3098749850 conn(0x5561c423a110 msgr2=0x5561c423a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.580+0000 7fb10b1d3740 1 -- 172.21.15.119:0/3098749850 shutdown_connections 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.580+0000 7fb10b1d3740 1 -- 172.21.15.119:0/3098749850 wait complete. 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.580+0000 7fb106400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 0x7fb0e80a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5561c423fdf0 tx=0x7fb0f8009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.580+0000 7fb10b1d3740 1 -- start start 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.580+0000 7fb10b1d3740 1 Processor -- start 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4249650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4250620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c4247650 con 0x5561c4244700 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c415fdd0 con 0x5561c423f590 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c42477d0 con 0x5561c423eba0 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb10b1d3740 10 client.0 _ll_register_callbacks cb 0x5561c3eb7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4249650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36452/0 (socket says 172.21.15.119:36452) 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 -- 172.21.15.119:0/643923486 learned_addr learned my addr 172.21.15.119:0/643923486 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb106400640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4250620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 -- 172.21.15.119:0/643923486 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 msgr2=0x5561c4249650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4249650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 -- 172.21.15.119:0/643923486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 msgr2=0x5561c4250620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4250620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 -- 172.21.15.119:0/643923486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561c3eb82d0 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105000640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4249650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb106400640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4250620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.581+0000 7fb105a00640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40976d0 tx=0x7fb0f4005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80dca9d32ab25fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4020070 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fb0f4097370 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0ec004ba0 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f409a660 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 10 client.25225 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5561c3f826b0 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec004ba0 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb0f409a150 con 0x5561c423eba0 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 10 client.25225 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 10 client.25225 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 20 client.25225 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42558a0 0x5561c4254960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 4 client.25225 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.582+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561c4145520 con 0x5561c42558a0 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.583+0000 7fb105000640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42558a0 0x5561c4254960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.583+0000 7fb105000640 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42558a0 0x5561c4254960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1773fa8a20bb6399 server_cookie=4f97e49d3be5c49f in_seq=0 out_seq=0 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.583+0000 7fb0e7400640 10 client.25225 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb0e7400640 1 -- 172.21.15.119:0/643923486 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5561c4145520 con 0x5561c42558a0 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb0e7400640 10 client.25225 handle_command_reply: tid=0 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 client.25225 shutdown 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 msgr2=0x5561c4249110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40976d0 tx=0x7fb0f4005f30 comp rx=0 tx=0).stop 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 shutdown_connections 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42558a0 0x5561c4254960 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4250620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423f590 0x5561c4249650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb0e7400640 0 client.25225 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/643923486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423eba0 0x5561c4249110 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 >> 172.21.15.119:0/643923486 conn(0x5561c423a110 msgr2=0x5561c4239310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 shutdown_connections 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 1 -- 172.21.15.119:0/643923486 wait complete. 2024-03-28T18:43:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.584+0000 7fb10b1d3740 20 client.25225 trim_cache size 0 max 16384 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.602+0000 7fb10b1d3740 1 Processor -- start 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 -- start start 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c4249430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4247c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c42550c0 con 0x5561c4244700 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c4255720 con 0x5561c423eba0 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c4255d80 con 0x5561c423f590 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c4249430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb106400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4247c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36454/0 (socket says 172.21.15.119:36454) 2024-03-28T18:43:36.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 -- 172.21.15.119:0/2904200300 learned_addr learned my addr 172.21.15.119:0/2904200300 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 -- 172.21.15.119:0/2904200300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 msgr2=0x5561c4249430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c4249430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 -- 172.21.15.119:0/2904200300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 msgr2=0x5561c4247c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4247c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105000640 1 -- 172.21.15.119:0/2904200300 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561c423f970 con 0x5561c423f590 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.603+0000 7fb105a00640 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c4249430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb105000640 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0096c0 tx=0x7fb0ec02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db1d79276b8978c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb0f2000640 1 -- 172.21.15.119:0/2904200300 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec055070 con 0x5561c423f590 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb0f2000640 1 -- 172.21.15.119:0/2904200300 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec0055d0 con 0x5561c423f590 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb0f2000640 1 -- 172.21.15.119:0/2904200300 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec042040 con 0x5561c423f590 2024-03-28T18:43:36.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2904200300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 msgr2=0x5561c42505e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.604+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0096c0 tx=0x7fb0ec02e040 comp rx=0 tx=0).stop 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2904200300 shutdown_connections 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c4244700 0x5561c4247c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c423f590 0x5561c42505e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2904200300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c4249430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2904200300 >> 172.21.15.119:0/2904200300 conn(0x5561c423a110 msgr2=0x5561c4248d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2904200300 shutdown_connections 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2904200300 wait complete. 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 -- start start 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 Processor -- start 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.605+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423f590 0x5561c424c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c424f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c4256a40 con 0x5561c423f590 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c42563e0 con 0x5561c423eba0 2024-03-28T18:43:36.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c42570a0 con 0x5561c4244700 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb10b1d3740 10 client.0 _ll_register_callbacks cb 0x7fb0f40046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb106400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c424f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57630/0 (socket says 172.21.15.119:57630) 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 -- 172.21.15.119:0/2862914168 learned_addr learned my addr 172.21.15.119:0/2862914168 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105000640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423f590 0x5561c424c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 -- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 msgr2=0x5561c424f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c424f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 -- 172.21.15.119:0/2862914168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423f590 msgr2=0x5561c424c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423f590 0x5561c424c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 -- 172.21.15.119:0/2862914168 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561c424fd60 con 0x5561c423eba0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb105a00640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009610 tx=0x7fb0f409a380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da1369387877c403 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb0e7400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.606+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4020040 con 0x5561c423eba0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fb0f40b92f0 con 0x5561c423eba0 2024-03-28T18:43:36.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0ec0055d0 con 0x5561c423eba0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f409ee50 con 0x5561c423eba0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 10 client.25208 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5561c42b9670 con 0x5561c423eba0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec0055d0 con 0x5561c423eba0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb0f4096a50 con 0x5561c423eba0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 10 client.25208 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 10 client.25208 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 20 client.25208 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42bda90 0x5561c42b98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 4 client.25208 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5561c41b11d0 con 0x5561c42bda90 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.607+0000 7fb105000640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42bda90 0x5561c42b98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb105000640 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42bda90 0x5561c42b98a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8f53bb81475c9cf server_cookie=cf28c0dc3c7bb893 in_seq=0 out_seq=0 2024-03-28T18:43:36.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb0e7400640 10 client.25208 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb0e7400640 1 -- 172.21.15.119:0/2862914168 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5561c41b11d0 con 0x5561c42bda90 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb0e7400640 10 client.25208 handle_command_reply: tid=0 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 client.25208 shutdown 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 msgr2=0x5561c424c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009610 tx=0x7fb0f409a380 comp rx=0 tx=0).stop 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 shutdown_connections 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5561c42bda90 0x5561c42b98a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5561c4244700 0x5561c424f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb0e7400640 0 client.25208 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5561c423f590 0x5561c424c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 --2- 172.21.15.119:0/2862914168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5561c423eba0 0x5561c424c210 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.608+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 >> 172.21.15.119:0/2862914168 conn(0x5561c423a110 msgr2=0x5561c423dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 shutdown_connections 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb10b1d3740 1 -- 172.21.15.119:0/2862914168 wait complete. 2024-03-28T18:43:36.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb10b1d3740 20 client.25208 trim_cache size 0 max 16384 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 2, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24517, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 131.463118356, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:36.616 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 159.445752255 2024-03-28T18:43:36.617 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:36.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 msgr2=0x7fb0e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 0x7fb0e80a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5561c423fdf0 tx=0x7fb0f8009700 comp rx=0 tx=0).stop 2024-03-28T18:43:36.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 msgr2=0x7fb100199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400d770 tx=0x7fb0f400dc30 comp rx=0 tx=0).stop 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 shutdown_connections 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fb0e80a2880 0x7fb0e80a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb100105470 0x7fb1001a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100104ab0 0x7fb10019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 --2- 172.21.15.119:0/1724177034 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb1001038c0 0x7fb100199bb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 >> 172.21.15.119:0/1724177034 conn(0x7fb1000fee50 msgr2=0x7fb100107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 shutdown_connections 2024-03-28T18:43:36.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.609+0000 7fb108400640 1 -- 172.21.15.119:0/1724177034 wait complete. 2024-03-28T18:43:36.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T18:43:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:36 smithi196 ceph-mon[36300]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:36 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/103015880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:36 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/443410213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:36 smithi131 ceph-mon[36065]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:36 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/103015880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:36 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/443410213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:36.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.836+0000 7f97a0c00640 1 -- 172.21.15.119:0/69855168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 msgr2=0x7f979c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.836+0000 7f97a0c00640 1 --2- 172.21.15.119:0/69855168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c109330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9784009900 tx=0x7f978402bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 -- 172.21.15.119:0/69855168 shutdown_connections 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 --2- 172.21.15.119:0/69855168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c109870 0x7f979c110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 --2- 172.21.15.119:0/69855168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c109330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 --2- 172.21.15.119:0/69855168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c107cc0 0x7f979c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 -- 172.21.15.119:0/69855168 >> 172.21.15.119:0/69855168 conn(0x7f979c076c60 msgr2=0x7f979c0790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 -- 172.21.15.119:0/69855168 shutdown_connections 2024-03-28T18:43:36.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.837+0000 7f97a0c00640 1 -- 172.21.15.119:0/69855168 wait complete. 2024-03-28T18:43:36.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.838+0000 7f97a0c00640 1 Processor -- start 2024-03-28T18:43:36.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.838+0000 7f97a0c00640 1 -- start start 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.838+0000 7f97a0c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c107cc0 0x7f979c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f97a0c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f97a0c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c109870 0x7f979c1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f97a0c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c113a10 con 0x7f979c107cc0 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f97a0c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c113890 con 0x7f979c108eb0 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f97a0c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c113b90 con 0x7f979c109870 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57658/0 (socket says 172.21.15.119:57658) 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 -- 172.21.15.119:0/1982514546 learned_addr learned my addr 172.21.15.119:0/1982514546 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9799e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c107cc0 0x7f979c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f979a800640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c109870 0x7f979c1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 -- 172.21.15.119:0/1982514546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c109870 msgr2=0x7f979c1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c109870 0x7f979c1a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 -- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c107cc0 msgr2=0x7f979c19e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c107cc0 0x7f979c19e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.839+0000 7f9793e00640 1 -- 172.21.15.119:0/1982514546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c1a5c80 con 0x7f979c108eb0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f9793e00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f978402c8f0 tx=0x7f9784005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f9792a00640 1 -- 172.21.15.119:0/1982514546 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784041070 con 0x7f979c108eb0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f9792a00640 1 -- 172.21.15.119:0/1982514546 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97840044d0 con 0x7f979c108eb0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c1a5f10 con 0x7f979c108eb0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f9792a00640 1 -- 172.21.15.119:0/1982514546 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784034440 con 0x7f979c108eb0 2024-03-28T18:43:36.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.840+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c1a6370 con 0x7f979c108eb0 2024-03-28T18:43:36.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.843+0000 7f97a39d0740 1 Processor -- start 2024-03-28T18:43:36.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.843+0000 7f97a39d0740 1 -- start start 2024-03-28T18:43:36.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f384700 0x564b7f386b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f2e3c90 con 0x564b7f384700 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f28c980 con 0x564b7f37eba0 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f97a39d0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f2dfd30 con 0x564b7f37f590 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f9792a00640 1 -- 172.21.15.119:0/1982514546 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9784044070 con 0x7f979c108eb0 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f9799e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57662/0 (socket says 172.21.15.119:57662) 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f9799e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36490/0 (socket says 172.21.15.119:36490) 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 -- 172.21.15.119:0/3696360968 learned_addr learned my addr 172.21.15.119:0/3696360968 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f9793e00640 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f384700 0x564b7f386b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 -- 172.21.15.119:0/3696360968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 msgr2=0x564b7f384090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 -- 172.21.15.119:0/3696360968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f384700 msgr2=0x564b7f386b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f384700 0x564b7f386b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f979a800640 1 -- 172.21.15.119:0/3696360968 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b7f2a4510 con 0x564b7f37eba0 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.844+0000 7f9799e00640 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f979a800640 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f979400a430 tx=0x7f9794009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa76125ad1baad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f978b000640 1 -- 172.21.15.119:0/3696360968 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9794013790 con 0x564b7f37eba0 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f978b000640 1 -- 172.21.15.119:0/3696360968 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9794019070 con 0x564b7f37eba0 2024-03-28T18:43:36.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f978b000640 1 -- 172.21.15.119:0/3696360968 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9794014de0 con 0x564b7f37eba0 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f9792a00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 0x7f977c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f97a39d0740 1 -- 172.21.15.119:0/3696360968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 msgr2=0x564b7f37efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f979400a430 tx=0x7f9794009650 comp rx=0 tx=0).stop 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f97a39d0740 1 -- 172.21.15.119:0/3696360968 shutdown_connections 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f384700 0x564b7f386b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f384090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.845+0000 7f9792a00640 1 -- 172.21.15.119:0/1982514546 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f97840f06d0 con 0x7f979c108eb0 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3696360968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37eba0 0x564b7f37efc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 -- 172.21.15.119:0/3696360968 >> 172.21.15.119:0/3696360968 conn(0x564b7f37a110 msgr2=0x564b7f37a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f9799e00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 0x7f977c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 -- 172.21.15.119:0/3696360968 shutdown_connections 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 -- 172.21.15.119:0/3696360968 wait complete. 2024-03-28T18:43:36.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 -- start start 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f9799e00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 0x7f977c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x564b7f37b080 tx=0x7f978c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.846+0000 7f97a39d0740 1 Processor -- start 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f389110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f384700 0x564b7f390690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f3876b0 con 0x564b7f37eba0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f29fdd0 con 0x564b7f37f590 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f387830 con 0x564b7f384700 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f97a39d0740 10 client.0 _ll_register_callbacks cb 0x564b7eff7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9799e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f389110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f979a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f384700 0x564b7f390690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57670/0 (socket says 172.21.15.119:57670) 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 -- 172.21.15.119:0/2330233973 learned_addr learned my addr 172.21.15.119:0/2330233973 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 -- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f384700 msgr2=0x564b7f390690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f384700 0x564b7f390690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 -- 172.21.15.119:0/2330233973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 msgr2=0x564b7f389110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f389110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.847+0000 7f9793e00640 1 -- 172.21.15.119:0/2330233973 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b7eff82d0 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9793e00640 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f97840bcfd0 tx=0x7f9784009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36af7123959e65c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978404b050 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f97840c54a0 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9794019070 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97840c10e0 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 10 client.25220 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564b7f0c26b0 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9794019070 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f97840bc490 con 0x564b7f37f590 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 10 client.25220 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 10 client.25220 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.848+0000 7f97a39d0740 20 client.25220 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f395910 0x564b7f3947f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f97a39d0740 4 client.25220 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b7f285520 con 0x564b7f395910 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f9799e00640 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f395910 0x564b7f3947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f9799e00640 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f395910 0x564b7f3947f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=51d83fb7d11eedbd server_cookie=642307905f008e95 in_seq=0 out_seq=0 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.849+0000 7f9783400640 10 client.25220 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f9783400640 1 -- 172.21.15.119:0/2330233973 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564b7f285520 con 0x564b7f395910 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f9783400640 10 client.25220 handle_command_reply: tid=0 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 client.25220 shutdown 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 msgr2=0x564b7f389650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f97840bcfd0 tx=0x7f9784009260 comp rx=0 tx=0).stop 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 shutdown_connections 2024-03-28T18:43:36.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f395910 0x564b7f3947f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f384700 0x564b7f390690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f37f590 0x564b7f389650 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f9783400640 0 client.25220 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2330233973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f389110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 >> 172.21.15.119:0/2330233973 conn(0x564b7f37a110 msgr2=0x564b7f379310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 shutdown_connections 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.850+0000 7f97a39d0740 1 -- 172.21.15.119:0/2330233973 wait complete. 2024-03-28T18:43:36.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.851+0000 7f97a39d0740 20 client.25220 trim_cache size 0 max 16384 2024-03-28T18:43:36.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.868+0000 7f97a39d0740 1 Processor -- start 2024-03-28T18:43:36.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 -- start start 2024-03-28T18:43:36.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f37a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f395130 con 0x564b7f37eba0 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f395790 con 0x564b7f384700 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f97a39d0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f395df0 con 0x564b7f37f590 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36496/0 (socket says 172.21.15.119:36496) 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 -- 172.21.15.119:0/3059832263 learned_addr learned my addr 172.21.15.119:0/3059832263 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57674/0 (socket says 172.21.15.119:57674) 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9799e00640 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f37a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 -- 172.21.15.119:0/3059832263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 msgr2=0x564b7f388d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 -- 172.21.15.119:0/3059832263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 msgr2=0x564b7f37a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f37a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f9793e00640 1 -- 172.21.15.119:0/3059832263 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b7f387590 con 0x564b7f37f590 2024-03-28T18:43:36.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.869+0000 7f979a800640 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f9793e00640 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f978402c180 tx=0x7f97840f60d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3255c54a70f56ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f978a600640 1 -- 172.21.15.119:0/3059832263 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978404b070 con 0x564b7f37f590 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f978a600640 1 -- 172.21.15.119:0/3059832263 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97840c5730 con 0x564b7f37f590 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f978a600640 1 -- 172.21.15.119:0/3059832263 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840b7e10 con 0x564b7f37f590 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f97a39d0740 1 -- 172.21.15.119:0/3059832263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 msgr2=0x564b7f390650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.870+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f978402c180 tx=0x7f97840f60d0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 -- 172.21.15.119:0/3059832263 shutdown_connections 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f388d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37f590 0x564b7f390650 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 --2- 172.21.15.119:0/3059832263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37eba0 0x564b7f37a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 -- 172.21.15.119:0/3059832263 >> 172.21.15.119:0/3059832263 conn(0x564b7f37a110 msgr2=0x564b7f387c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 -- 172.21.15.119:0/3059832263 shutdown_connections 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 -- 172.21.15.119:0/3059832263 wait complete. 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 -- start start 2024-03-28T18:43:36.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.871+0000 7f97a39d0740 1 Processor -- start 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 0x564b7f38c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37f590 0x564b7f38ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f396ab0 con 0x564b7f37f590 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f396450 con 0x564b7f384700 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b7f397110 con 0x564b7f37eba0 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f97a39d0740 10 client.0 _ll_register_callbacks cb 0x7f9784003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 0x564b7f38c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9793e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37f590 0x564b7f38ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f979a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57676/0 (socket says 172.21.15.119:57676) 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f979a800640 1 -- 172.21.15.119:0/2938612630 learned_addr learned my addr 172.21.15.119:0/2938612630 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 -- 172.21.15.119:0/2938612630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 msgr2=0x564b7f38fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 -- 172.21.15.119:0/2938612630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37f590 msgr2=0x564b7f38ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:36.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37f590 0x564b7f38ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f9799e00640 1 -- 172.21.15.119:0/2938612630 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b7f390270 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.872+0000 7f979a800640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9799e00640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 0x564b7f38c770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x564b7f397c60 tx=0x7f978c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=719b818eea415d1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9783400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c003ac0 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f978c003c60 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97840c5730 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f978c038030 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f97a39d0740 10 client.25249 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564b7f3f1380 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.873+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97840c5730 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f978c03d550 con 0x564b7f37eba0 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 10 client.25249 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 10 client.25249 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T18:43:36.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 20 client.25249 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:36.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f3f5a60 0x564b7f3f1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:36.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 4 client.25249 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:36.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564b7f2f11d0 con 0x564b7f3f5a60 2024-03-28T18:43:36.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f9793e00640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f3f5a60 0x564b7f3f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f9793e00640 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f3f5a60 0x564b7f3f1670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6828d7c2823898cf server_cookie=369792e257fb1845 in_seq=0 out_seq=0 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f9783400640 10 client.25249 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.874+0000 7f9783400640 1 -- 172.21.15.119:0/2938612630 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564b7f2f11d0 con 0x564b7f3f5a60 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f9783400640 10 client.25249 handle_command_reply: tid=0 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 client.25249 shutdown 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 msgr2=0x564b7f38c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 0x564b7f38c770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x564b7f397c60 tx=0x7f978c0046c0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 shutdown_connections 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x564b7f3f5a60 0x564b7f3f1670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564b7f384700 0x564b7f38fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f9783400640 0 client.25249 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564b7f37f590 0x564b7f38ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 --2- 172.21.15.119:0/2938612630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564b7f37eba0 0x564b7f38c770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 >> 172.21.15.119:0/2938612630 conn(0x564b7f37a110 msgr2=0x564b7f37a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 shutdown_connections 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 1 -- 172.21.15.119:0/2938612630 wait complete. 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.875+0000 7f97a39d0740 20 client.25249 trim_cache size 0 max 16384 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:36.879 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 1, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24538, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 132.740045506, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 154.14993751399999 2024-03-28T18:43:36.880 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 msgr2=0x7f977c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 0x7f977c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x564b7f37b080 tx=0x7f978c005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 msgr2=0x7f979c19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f978402c8f0 tx=0x7f9784005d50 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 shutdown_connections 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f977c0a2800 0x7f977c0a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f979c109870 0x7f979c1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f979c108eb0 0x7f979c19e5a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 --2- 172.21.15.119:0/1982514546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f979c107cc0 0x7f979c19e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 >> 172.21.15.119:0/1982514546 conn(0x7f979c076c60 msgr2=0x7f979c10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 shutdown_connections 2024-03-28T18:43:36.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:36.876+0000 7f97a0c00640 1 -- 172.21.15.119:0/1982514546 wait complete. 2024-03-28T18:43:36.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.091+0000 7f5ff4800640 1 -- 172.21.15.119:0/774799259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec104a70 msgr2=0x7f5fec106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.091+0000 7f5ff4800640 1 --2- 172.21.15.119:0/774799259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec104a70 0x7f5fec106e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009900 tx=0x7f5fdc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 -- 172.21.15.119:0/774799259 shutdown_connections 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 --2- 172.21.15.119:0/774799259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec1073a0 0x7f5fec1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 --2- 172.21.15.119:0/774799259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec104a70 0x7f5fec106e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 --2- 172.21.15.119:0/774799259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec102140 0x7f5fec104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 -- 172.21.15.119:0/774799259 >> 172.21.15.119:0/774799259 conn(0x7f5fec0fbb50 msgr2=0x7f5fec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 -- 172.21.15.119:0/774799259 shutdown_connections 2024-03-28T18:43:37.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.092+0000 7f5ff4800640 1 -- 172.21.15.119:0/774799259 wait complete. 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 Processor -- start 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 -- start start 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec102140 0x7f5fec101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 0x7f5fec100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec10d380 con 0x7f5fec102140 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec10d200 con 0x7f5fec104a70 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.093+0000 7f5ff4800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec10d500 con 0x7f5fec1073a0 2024-03-28T18:43:37.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 0x7f5fec100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57696/0 (socket says 172.21.15.119:57696) 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 -- 172.21.15.119:0/3952339943 learned_addr learned my addr 172.21.15.119:0/3952339943 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec102140 0x7f5fec101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 -- 172.21.15.119:0/3952339943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 msgr2=0x7f5fec100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 0x7f5fec100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 -- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec102140 msgr2=0x7f5fec101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec102140 0x7f5fec101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 -- 172.21.15.119:0/3952339943 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec100ec0 con 0x7f5fec104a70 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff2800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 0x7f5fec100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:37.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.094+0000 7f5ff1400640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc000c00 tx=0x7f5fdc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.095+0000 7f5fe6a00640 1 -- 172.21.15.119:0/3952339943 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc043020 con 0x7f5fec104a70 2024-03-28T18:43:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.095+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec1a1b10 con 0x7f5fec104a70 2024-03-28T18:43:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.095+0000 7f5fe6a00640 1 -- 172.21.15.119:0/3952339943 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fdc0054e0 con 0x7f5fec104a70 2024-03-28T18:43:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.095+0000 7f5fe6a00640 1 -- 172.21.15.119:0/3952339943 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc003ce0 con 0x7f5fec104a70 2024-03-28T18:43:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.095+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec1a2080 con 0x7f5fec104a70 2024-03-28T18:43:37.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.098+0000 7f5ff7597740 1 Processor -- start 2024-03-28T18:43:37.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.098+0000 7f5ff7597740 1 -- start start 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcb6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb6700 0x5601bbcb8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbc15c90 con 0x5601bbcb1590 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbbbe980 con 0x5601bbcb0ba0 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff7597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbc11d30 con 0x5601bbcb6700 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5fe6a00640 1 -- 172.21.15.119:0/3952339943 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5fdc046020 con 0x7f5fec104a70 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57702/0 (socket says 172.21.15.119:57702) 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 -- 172.21.15.119:0/1769991104 learned_addr learned my addr 172.21.15.119:0/1769991104 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcb6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff1400640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb6700 0x5601bbcb8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 -- 172.21.15.119:0/1769991104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb6700 msgr2=0x5601bbcb8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb6700 0x5601bbcb8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 -- 172.21.15.119:0/1769991104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 msgr2=0x5601bbcb6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcb6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff2800640 1 -- 172.21.15.119:0/1769991104 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bbbd6510 con 0x5601bbcb0ba0 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.099+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcb6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.100+0000 7f5ff2800640 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8011790 tx=0x7f5fe8011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51a239a19c8cf0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.100+0000 7f5fdb000640 1 -- 172.21.15.119:0/1769991104 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe8003c30 con 0x5601bbcb0ba0 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.100+0000 7f5fdb000640 1 -- 172.21.15.119:0/1769991104 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe800fe80 con 0x5601bbcb0ba0 2024-03-28T18:43:37.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.100+0000 7f5fdb000640 1 -- 172.21.15.119:0/1769991104 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80193e0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.100+0000 7f5fe6a00640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 0x7f5fd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- 172.21.15.119:0/1769991104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 msgr2=0x5601bbcb0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8011790 tx=0x7f5fe8011c50 comp rx=0 tx=0).stop 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5fe6a00640 1 -- 172.21.15.119:0/3952339943 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f5fdc0f0bb0 con 0x7f5fec104a70 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 0x7f5fd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- 172.21.15.119:0/1769991104 shutdown_connections 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb6700 0x5601bbcb8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcb6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 --2- 172.21.15.119:0/1769991104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcb0fc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- 172.21.15.119:0/1769991104 >> 172.21.15.119:0/1769991104 conn(0x5601bbcac110 msgr2=0x5601bbcac510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- 172.21.15.119:0/1769991104 shutdown_connections 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- 172.21.15.119:0/1769991104 wait complete. 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.101+0000 7f5ff7597740 1 -- start start 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 0x7f5fd40a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5601bbcad080 tx=0x7f5fe000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 Processor -- start 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb6700 0x5601bbcc26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcb96b0 con 0x5601bbcb6700 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbbd1dd0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff7597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcb9830 con 0x5601bbcb1590 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.102+0000 7f5ff1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57706/0 (socket says 172.21.15.119:57706) 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36540/0 (socket says 172.21.15.119:36540) 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2477007724 learned_addr learned my addr 172.21.15.119:0/2477007724 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff2800640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb6700 0x5601bbcc26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff7597740 10 client.0 _ll_register_callbacks cb 0x5601bb929ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2477007724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 msgr2=0x5601bbcbb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2477007724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb6700 msgr2=0x5601bbcc26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb6700 0x5601bbcc26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2477007724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601bb92a2d0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5fe001f430 tx=0x7f5fe0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7027827fa334b2a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5fcf400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.103+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe002c060 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f5fe0017030 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601bbbb7520 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe001c600 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 10 client.25238 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601bb9f46b0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5601bbbb7520 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5fe00109c0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 10 client.25238 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 10 client.25238 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 20 client.25238 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcc79c0 0x5601bbcc6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 4 client.25238 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:37.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.104+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601bbc231d0 con 0x5601bbcc79c0 2024-03-28T18:43:37.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.105+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcc79c0 0x5601bbcc6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.105+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcc79c0 0x5601bbcc6a30 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cdefff675718005 server_cookie=16aedde20a19f47 in_seq=0 out_seq=0 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.105+0000 7f5fcf400640 10 client.25238 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.106+0000 7f5fcf400640 1 -- 172.21.15.119:0/2477007724 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5601bbc231d0 con 0x5601bbcc79c0 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.106+0000 7f5fcf400640 10 client.25238 handle_command_reply: tid=0 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.106+0000 7f5ff7597740 1 client.25238 shutdown 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.106+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 msgr2=0x5601bbcbb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.106+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5fe001f430 tx=0x7f5fe0004730 comp rx=0 tx=0).stop 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 shutdown_connections 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcc79c0 0x5601bbcc6a30 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb6700 0x5601bbcc26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbcbb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2477007724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbb170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 >> 172.21.15.119:0/2477007724 conn(0x5601bbcac110 msgr2=0x5601bbcab310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5fcf400640 0 client.25238 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:37.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 shutdown_connections 2024-03-28T18:43:37.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 1 -- 172.21.15.119:0/2477007724 wait complete. 2024-03-28T18:43:37.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.107+0000 7f5ff7597740 20 client.25238 trim_cache size 0 max 16384 2024-03-28T18:43:37.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 Processor -- start 2024-03-28T18:43:37.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 -- start start 2024-03-28T18:43:37.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 0x5601bbcb8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcc26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbca6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc71e0 con 0x5601bbcac110 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc7840 con 0x5601bbcb0ba0 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.127+0000 7f5ff7597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc7ea0 con 0x5601bbcb1590 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 0x5601bbcb8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcc26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 0x5601bbcb8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38426/0 (socket says 172.21.15.119:38426) 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbca6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2755051405 learned_addr learned my addr 172.21.15.119:0/2755051405 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 -- 172.21.15.119:0/2755051405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 msgr2=0x5601bbca6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbca6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 -- 172.21.15.119:0/2755051405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 msgr2=0x5601bbcb8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 0x5601bbcb8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff1400640 1 -- 172.21.15.119:0/2755051405 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bbcace80 con 0x5601bbcb0ba0 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.128+0000 7f5ff2800640 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbca6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcc26b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0ee680 tx=0x7f5fdc0dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ce234081a17c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5fda600640 1 -- 172.21.15.119:0/2755051405 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0bd010 con 0x5601bbcb0ba0 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5fda600640 1 -- 172.21.15.119:0/2755051405 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fdc004c80 con 0x5601bbcb0ba0 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5fda600640 1 -- 172.21.15.119:0/2755051405 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0bce30 con 0x5601bbcb0ba0 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5ff7597740 1 -- 172.21.15.119:0/2755051405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 msgr2=0x5601bbcc26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.129+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcc26b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0ee680 tx=0x7f5fdc0dd040 comp rx=0 tx=0).stop 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 -- 172.21.15.119:0/2755051405 shutdown_connections 2024-03-28T18:43:37.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb1590 0x5601bbca6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcb0ba0 0x5601bbcc26b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2755051405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcac110 0x5601bbcb8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 -- 172.21.15.119:0/2755051405 >> 172.21.15.119:0/2755051405 conn(0x5601bbcbc0f0 msgr2=0x5601bbcb1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 -- 172.21.15.119:0/2755051405 shutdown_connections 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 -- 172.21.15.119:0/2755051405 wait complete. 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 -- start start 2024-03-28T18:43:37.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.130+0000 7f5ff7597740 1 Processor -- start 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcc1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc8b60 con 0x5601bbcb1590 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc8500 con 0x5601bbcac110 2024-03-28T18:43:37.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bbcc91c0 con 0x5601bbcb0ba0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff7597740 10 client.0 _ll_register_callbacks cb 0x7f5fe000bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57716/0 (socket says 172.21.15.119:57716) 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2731788201 learned_addr learned my addr 172.21.15.119:0/2731788201 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff2800640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcc1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2731788201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb0ba0 msgr2=0x5601bbcbe860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbe860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2731788201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 msgr2=0x5601bbcc1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcc1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 -- 172.21.15.119:0/2731788201 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601bbcc1e20 con 0x5601bbcac110 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5ff1e00640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0010d60 tx=0x7f5fe001cd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e2a61b4486dd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:37.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.131+0000 7f5fcf400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0010840 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f5fe00109e0 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fdc004c80 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe001fc00 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 10 client.25244 fetch_fsmap learned FSMap version 27 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601bbd22f50 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fdc004c80 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5fe001bc80 con 0x5601bbcac110 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 10 client.25244 fetch_fsmap finished waiting for FSMap version 27 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 10 client.25244 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 20 client.25244 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcb6700 0x5601bbd23180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 4 client.25244 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.132+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5601bbc1d520 con 0x5601bbcb6700 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcb6700 0x5601bbd23180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5ff1400640 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcb6700 0x5601bbd23180 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b09cacb539c254e server_cookie=4694dac912fd6819 in_seq=0 out_seq=0 2024-03-28T18:43:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5fcf400640 10 client.25244 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5fcf400640 1 -- 172.21.15.119:0/2731788201 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5601bbc1d520 con 0x5601bbcb6700 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5fcf400640 10 client.25244 handle_command_reply: tid=0 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.133+0000 7f5ff7597740 1 client.25244 shutdown 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 msgr2=0x5601bbcbe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0010d60 tx=0x7f5fe001cd60 comp rx=0 tx=0).stop 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 shutdown_connections 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5601bbcb6700 0x5601bbd23180 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601bbcb1590 0x5601bbcc1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601bbcb0ba0 0x5601bbcbe860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5fcf400640 0 client.25244 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 --2- 172.21.15.119:0/2731788201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601bbcac110 0x5601bbcbe320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 >> 172.21.15.119:0/2731788201 conn(0x5601bbcbc0f0 msgr2=0x5601bbca6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 shutdown_connections 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 1 -- 172.21.15.119:0/2731788201 wait complete. 2024-03-28T18:43:37.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff7597740 20 client.25244 trim_cache size 0 max 16384 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "5ff7da1e-ed31-11ee-95d0-87774f69a715", 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 24563, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 136.03277096799999, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 27, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 90, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 86, 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 143.82469181799999 2024-03-28T18:43:37.138 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 msgr2=0x7f5fd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.134+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 0x7f5fd40a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5601bbcad080 tx=0x7f5fe000b410 comp rx=0 tx=0).stop 2024-03-28T18:43:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 msgr2=0x7f5fec1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc000c00 tx=0x7f5fdc02c6c0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 shutdown_connections 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f5fd40a2800 0x7f5fd40a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fec1073a0 0x7f5fec100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5fec104a70 0x7f5fec1000f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 --2- 172.21.15.119:0/3952339943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fec102140 0x7f5fec101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 >> 172.21.15.119:0/3952339943 conn(0x7f5fec0fbb50 msgr2=0x7f5fec1029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 shutdown_connections 2024-03-28T18:43:37.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:37.135+0000 7f5ff4800640 1 -- 172.21.15.119:0/3952339943 wait complete. 2024-03-28T18:43:37.152 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T18:43:37.176 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T18:43:37.176 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T18:43:37.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:37.176 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-03-28T18:43:37.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:37.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link delete ceph-brx 2024-03-28T18:43:37.324 INFO:teuthology.orchestra.run.smithi119.stderr:Cannot find device "ceph-brx" 2024-03-28T18:43:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:43:37.329 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T18:43:37.329 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T18:43:37.330 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:43:37.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T18:43:37.777 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:43:38.486 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:38 smithi196 ceph-mon[36300]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:38 smithi119 ceph-mon[31813]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:38 smithi131 ceph-mon[36065]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:43:39.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 -- 172.21.15.119:0/1510172157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 msgr2=0x7fd3fc1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:39.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 --2- 172.21.15.119:0/1510172157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1049e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009990 tx=0x7fd3ec009ca0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 -- 172.21.15.119:0/1510172157 shutdown_connections 2024-03-28T18:43:39.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 --2- 172.21.15.119:0/1510172157 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 --2- 172.21.15.119:0/1510172157 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1049e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 --2- 172.21.15.119:0/1510172157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 -- 172.21.15.119:0/1510172157 >> 172.21.15.119:0/1510172157 conn(0x7fd3fc0ffe90 msgr2=0x7fd3fc1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 -- 172.21.15.119:0/1510172157 shutdown_connections 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.835+0000 7fd402e00700 1 -- 172.21.15.119:0/1510172157 wait complete. 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.836+0000 7fd402e00700 1 Processor -- start 2024-03-28T18:43:39.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 -- start start 2024-03-28T18:43:39.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc114710 con 0x7fd3fc1096e0 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc114590 con 0x7fd3fc104600 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd402e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3fc114890 con 0x7fd3fc104fb0 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd3fbe00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd3fbe00700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54014/0 (socket says 172.21.15.119:54014) 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd3fbe00700 1 -- 172.21.15.119:0/1988298144 learned_addr learned my addr 172.21.15.119:0/1988298144 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd3f3e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42518/0 (socket says 172.21.15.119:42518) 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.837+0000 7fd400e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 -- 172.21.15.119:0/1988298144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 msgr2=0x7fd3fc1a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc1a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 -- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 msgr2=0x7fd3fc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 -- 172.21.15.119:0/1988298144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3fc19b0f0 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd400e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3fbe00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec000c00 tx=0x7fd3ec005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec027070 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ec0056b0 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec017650 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.838+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3fc19b380 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.839+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3fc1a76a0 con 0x7fd3fc104600 2024-03-28T18:43:39.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.840+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3fc105ae0 con 0x7fd3fc104600 2024-03-28T18:43:39.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.841+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd3ec02a070 con 0x7fd3fc104600 2024-03-28T18:43:39.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.843+0000 7fd3f9600700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 0x7fd3e00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:39.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.843+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fd3ec0cfb30 con 0x7fd3fc104600 2024-03-28T18:43:39.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.843+0000 7fd3f3e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 0x7fd3e00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:39.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.844+0000 7fd3f3e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 0x7fd3e00a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc10ac90 tx=0x7fd3e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:39.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:39.845+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd3ec013070 con 0x7fd3fc104600 2024-03-28T18:43:40.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.120+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd3fc02da30 con 0x7fd3fc104600 2024-03-28T18:43:40.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.120+0000 7fd3f9600700 1 -- 172.21.15.119:0/1988298144 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd3ec09cb80 con 0x7fd3fc104600 2024-03-28T18:43:40.120 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 msgr2=0x7fd3e00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 0x7fd3e00a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc10ac90 tx=0x7fd3e4009260 comp rx=0 tx=0).stop 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 msgr2=0x7fd3fc1a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec000c00 tx=0x7fd3ec005590 comp rx=0 tx=0).stop 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 shutdown_connections 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd3e00a2910 0x7fd3e00a4dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:40.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3fc1096e0 0x7fd3fc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:40.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3fc104600 0x7fd3fc1a07d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:40.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 --2- 172.21.15.119:0/1988298144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3fc104fb0 0x7fd3fc1a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:40.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 >> 172.21.15.119:0/1988298144 conn(0x7fd3fc0ffe90 msgr2=0x7fd3fc1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:40.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 shutdown_connections 2024-03-28T18:43:40.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:40.123+0000 7fd402e00700 1 -- 172.21.15.119:0/1988298144 wait complete. 2024-03-28T18:43:40.490 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:40 smithi119 ceph-mon[31813]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:40.490 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:40 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1988298144' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:43:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:40 smithi196 ceph-mon[36300]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:40 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1988298144' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:43:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:40 smithi131 ceph-mon[36065]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:40 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1988298144' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T18:43:40.894 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T18:43:40.895 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T18:43:40.895 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:43:40.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:40.895 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-03-28T18:43:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T18:43:40.949 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: altname enp7s0f0 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: altname enp7s0f1 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T18:43:40.950 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-03-28T18:43:40.951 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-28T18:43:40.951 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 41967sec preferred_lft 41967sec 2024-03-28T18:43:40.951 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-03-28T18:43:40.951 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-03-28T18:43:40.952 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T18:43:40.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add name ceph-brx type bridge 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr flush dev ceph-brx 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set ceph-brx up 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T18:43:40.952 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-03-28T18:43:41.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:41.401 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T18:43:41.474 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-03-28T18:43:41.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:41.477 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-03-28T18:43:41.533 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-03-28T18:43:41.534 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T18:43:41.534 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-28T18:43:41.534 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-28T18:43:41.534 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T18:43:41.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:41.534 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-03-28T18:43:41.534 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-03-28T18:43:41.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-28T18:43:41.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-28T18:43:41.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-28T18:43:41.535 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-03-28T18:43:42.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:42.288 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-03-28T18:43:42.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:42.344 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list-id 2024-03-28T18:43:42.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:42.395 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-03-28T18:43:42.395 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-03-28T18:43:42.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:43:42.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T18:43:42.396 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-03-28T18:43:42.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:42 smithi119 ceph-mon[31813]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:42.541 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T18:43:42.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T18:43:42.541 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-03-28T18:43:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:42 smithi196 ceph-mon[36300]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:42 smithi131 ceph-mon[36065]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:43:42.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:42.820 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-03-28T18:43:42.820 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-03-28T18:43:42.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-03-28T18:43:42.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T18:43:42.820 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-03-28T18:43:42.986 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:43:42.986 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T18:43:43.033 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T18:43:43.033 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T18:43:43.033 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:43:43.125 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T18:43:43.125 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T18:43:43.125 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/a7a67cbb-052a-445c-b58b-3818e042bf37 2024-03-28T18:43:43.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:43.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/a7a67cbb-052a-445c-b58b-3818e042bf37 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:43:43.257 INFO:teuthology.orchestra.run.smithi119.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:43:43.257 INFO:teuthology.orchestra.run.smithi119.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T18:43:43.894 INFO:teuthology.orchestra.run.smithi119.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-28T18:43:43.894 INFO:teuthology.orchestra.run.smithi119.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-28T18:43:43.895 INFO:teuthology.orchestra.run.smithi119.stdout:mount.ceph: resolved to: "172.21.15.119:6789,172.21.15.131:6789,172.21.15.196:6789" 2024-03-28T18:43:43.895 INFO:teuthology.orchestra.run.smithi119.stdout:mount.ceph: trying mount with new device syntax: 0@5ff7da1e-ed31-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/a7a67cbb-052a-445c-b58b-3818e042bf37 2024-03-28T18:43:43.895 INFO:teuthology.orchestra.run.smithi119.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.119:6789/172.21.15.131:6789/172.21.15.196:6789" will pass to kernel 2024-03-28T18:43:43.898 INFO:teuthology.orchestra.run.smithi119.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T18:43:43.898 INFO:teuthology.orchestra.run.smithi119.stdout: You just mounted a file system that supports labels which does not 2024-03-28T18:43:43.898 INFO:teuthology.orchestra.run.smithi119.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T18:43:43.898 INFO:teuthology.orchestra.run.smithi119.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T18:43:43.898 INFO:teuthology.orchestra.run.smithi119.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T18:43:43.903 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T18:43:43.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:43.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T18:43:43.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:43.998 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:43:44.058 INFO:teuthology.orchestra.run.smithi119.stdout:client25276 2024-03-28T18:43:44.058 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:43:44.058 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:43:44.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:44.109 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:43:44.165 INFO:teuthology.orchestra.run.smithi119.stdout:client25276 2024-03-28T18:43:44.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:44.165 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T18:43:44.215 INFO:teuthology.orchestra.run.smithi119.stdout:client25276 2024-03-28T18:43:44.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:44.215 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715.client25276/status') 2024-03-28T18:43:44.305 INFO:teuthology.orchestra.run.smithi119.stdout:instance: client.25276 (3)172.21.15.119:0/3750074906 2024-03-28T18:43:44.305 INFO:teuthology.orchestra.run.smithi119.stdout:blocklisted: false 2024-03-28T18:43:44.306 INFO:teuthology.orchestra.run.smithi119.stderr:0+1 records in 2024-03-28T18:43:44.306 INFO:teuthology.orchestra.run.smithi119.stderr:0+1 records out 2024-03-28T18:43:44.306 INFO:teuthology.orchestra.run.smithi119.stderr:72 bytes copied, 8.6603e-05 s, 831 kB/s 2024-03-28T18:43:44.310 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T18:43:44.320 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:43:44.320 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:43:44.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-28T18:43:44.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 -- 172.21.15.119:0/913130492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c0104af0 msgr2=0x7f03c0104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:44.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 --2- 172.21.15.119:0/913130492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c0104af0 0x7f03c0104f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009ca0 tx=0x7f03ac02c100 comp rx=0 tx=0).stop 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 -- 172.21.15.119:0/913130492 shutdown_connections 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 --2- 172.21.15.119:0/913130492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c01054b0 0x7f03c010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 --2- 172.21.15.119:0/913130492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c0104af0 0x7f03c0104f70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 --2- 172.21.15.119:0/913130492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 -- 172.21.15.119:0/913130492 >> 172.21.15.119:0/913130492 conn(0x7f03c00fee90 msgr2=0x7f03c01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 -- 172.21.15.119:0/913130492 shutdown_connections 2024-03-28T18:43:44.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.532+0000 7f03c8200640 1 -- 172.21.15.119:0/913130492 wait complete. 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.533+0000 7f03c8200640 1 Processor -- start 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.533+0000 7f03c8200640 1 -- start start 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c01054b0 0x7f03c01a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c010f560 con 0x7f03c0103900 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c010f3e0 con 0x7f03c0104af0 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c8200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c010f6e0 con 0x7f03c01054b0 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:44.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58476/0 (socket says 172.21.15.119:58476) 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c5800640 1 -- 172.21.15.119:0/2748112023 learned_addr learned my addr 172.21.15.119:0/2748112023 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54030/0 (socket says 172.21.15.119:54030) 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c6200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c01054b0 0x7f03c01a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 -- 172.21.15.119:0/2748112023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c01054b0 msgr2=0x7f03c01a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c01054b0 0x7f03c01a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 -- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 msgr2=0x7f03c0199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.534+0000 7f03c4e00640 1 -- 172.21.15.119:0/2748112023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c01a1760 con 0x7f03c0104af0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03c4e00640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009bf0 tx=0x7f03ac035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac039040 con 0x7f03c0104af0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03ac003970 con 0x7f03c0104af0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c01a19f0 con 0x7f03c0104af0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac004360 con 0x7f03c0104af0 2024-03-28T18:43:44.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.535+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c01a1eb0 con 0x7f03c0104af0 2024-03-28T18:43:44.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.536+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c010abc0 con 0x7f03c0104af0 2024-03-28T18:43:44.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.538+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f03ac046030 con 0x7f03c0104af0 2024-03-28T18:43:44.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.540+0000 7f03be000640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 0x7f03a40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:44.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.540+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f03ac0f0650 con 0x7f03c0104af0 2024-03-28T18:43:44.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.540+0000 7f03c5800640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 0x7f03a40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:44.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.541+0000 7f03c5800640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 0x7f03a40a4d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03b00058c0 tx=0x7f03b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:44.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.542+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f03ac04d060 con 0x7f03c0104af0 2024-03-28T18:43:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:44 smithi119 ceph-mon[31813]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:44 smithi196 ceph-mon[36300]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:44 smithi131 ceph-mon[36065]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-28T18:43:44.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:44.781+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f03c002d280 con 0x7f03c0104af0 2024-03-28T18:43:45.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.235+0000 7f03be000640 1 -- 172.21.15.119:0/2748112023 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) v1 ==== 117+0+0 (secure 0 0 0) 0x7f03ac0bd6a0 con 0x7f03c0104af0 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 msgr2=0x7f03a40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 0x7f03a40a4d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03b00058c0 tx=0x7f03b0005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 msgr2=0x7f03c019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009bf0 tx=0x7f03ac035d10 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 shutdown_connections 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f03a40a28a0 0x7f03a40a4d60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03c01054b0 0x7f03c01a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03c0104af0 0x7f03c019a080 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 --2- 172.21.15.119:0/2748112023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03c0103900 0x7f03c0199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 >> 172.21.15.119:0/2748112023 conn(0x7f03c00fee90 msgr2=0x7f03c0107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:45.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 shutdown_connections 2024-03-28T18:43:45.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.239+0000 7f03c8200640 1 -- 172.21.15.119:0/2748112023 wait complete. 2024-03-28T18:43:45.261 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T18:43:45.271 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-28T18:43:45.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:45.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 -- 172.21.15.119:0/3958539656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634107b00 msgr2=0x7f4634107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 --2- 172.21.15.119:0/3958539656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634107b00 0x7f4634107f20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4624009900 tx=0x7f462402bc50 comp rx=0 tx=0).stop 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 -- 172.21.15.119:0/3958539656 shutdown_connections 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 --2- 172.21.15.119:0/3958539656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 0x7f463410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 --2- 172.21.15.119:0/3958539656 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634108cf0 0x7f4634109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 --2- 172.21.15.119:0/3958539656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634107b00 0x7f4634107f20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 -- 172.21.15.119:0/3958539656 >> 172.21.15.119:0/3958539656 conn(0x7f4634076ab0 msgr2=0x7f4634078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 -- 172.21.15.119:0/3958539656 shutdown_connections 2024-03-28T18:43:45.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.523+0000 7f4639a00640 1 -- 172.21.15.119:0/3958539656 wait complete. 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 Processor -- start 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 -- start start 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634108cf0 0x7f463419e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 0x7f46341a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46341137a0 con 0x7f46341096b0 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634113620 con 0x7f4634107b00 2024-03-28T18:43:45.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4639a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4634113920 con 0x7f4634108cf0 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.525+0000 7f4633e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54054/0 (socket says 172.21.15.119:54054) 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 -- 172.21.15.119:0/1990459716 learned_addr learned my addr 172.21.15.119:0/1990459716 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633400640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634108cf0 0x7f463419e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4639000640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 0x7f46341a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 -- 172.21.15.119:0/1990459716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634108cf0 msgr2=0x7f463419e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634108cf0 0x7f463419e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 -- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 msgr2=0x7f46341a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 0x7f46341a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 -- 172.21.15.119:0/1990459716 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46341a5a30 con 0x7f4634107b00 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4633e00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f462402c260 tx=0x7f462402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.526+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624041070 con 0x7f4634107b00 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.527+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4624004450 con 0x7f4634107b00 2024-03-28T18:43:45.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.527+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46341a5cc0 con 0x7f4634107b00 2024-03-28T18:43:45.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.527+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4624034440 con 0x7f4634107b00 2024-03-28T18:43:45.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.527+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46341a61b0 con 0x7f4634107b00 2024-03-28T18:43:45.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.528+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463410ee20 con 0x7f4634107b00 2024-03-28T18:43:45.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.530+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4624044070 con 0x7f4634107b00 2024-03-28T18:43:45.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.531+0000 7f4630c00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 0x7f46180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.531+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f46240f0930 con 0x7f4634107b00 2024-03-28T18:43:45.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.531+0000 7f4633400640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 0x7f46180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.532+0000 7f4633400640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 0x7f46180a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46200095f0 tx=0x7f4620005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:45.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.538+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46240bd950 con 0x7f4634107b00 2024-03-28T18:43:45.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:45 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2748112023' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:45 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:45 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2748112023' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:45 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:45 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2748112023' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:45 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T18:43:45.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.738+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46340622a0 con 0x7f4634107b00 2024-03-28T18:43:45.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.740+0000 7f4630c00640 1 -- 172.21.15.119:0/1990459716 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f46240bd250 con 0x7f4634107b00 2024-03-28T18:43:45.741 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:45.741 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 msgr2=0x7f46180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 0x7f46180a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f46200095f0 tx=0x7f4620005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 msgr2=0x7f463419de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f462402c260 tx=0x7f462402c670 comp rx=0 tx=0).stop 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 shutdown_connections 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f46180a2880 0x7f46180a4d40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46341096b0 0x7f46341a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4634108cf0 0x7f463419e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 --2- 172.21.15.119:0/1990459716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4634107b00 0x7f463419de10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 >> 172.21.15.119:0/1990459716 conn(0x7f4634076ab0 msgr2=0x7f463410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:45.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 shutdown_connections 2024-03-28T18:43:45.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.743+0000 7f4639a00640 1 -- 172.21.15.119:0/1990459716 wait complete. 2024-03-28T18:43:45.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:45.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 -- 172.21.15.119:0/3641329592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f4102880 msgr2=0x7f26f4104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 --2- 172.21.15.119:0/3641329592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f4102880 0x7f26f4104cf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009900 tx=0x7f26dc02bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 -- 172.21.15.119:0/3641329592 shutdown_connections 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 --2- 172.21.15.119:0/3641329592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 0x7f26f41056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 --2- 172.21.15.119:0/3641329592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f4102880 0x7f26f4104cf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 --2- 172.21.15.119:0/3641329592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f40fff10 0x7f26f4102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 -- 172.21.15.119:0/3641329592 >> 172.21.15.119:0/3641329592 conn(0x7f26f40fbb30 msgr2=0x7f26f40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 -- 172.21.15.119:0/3641329592 shutdown_connections 2024-03-28T18:43:45.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.981+0000 7f26fb800640 1 -- 172.21.15.119:0/3641329592 wait complete. 2024-03-28T18:43:45.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.982+0000 7f26fb800640 1 Processor -- start 2024-03-28T18:43:45.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.982+0000 7f26fb800640 1 -- start start 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 0x7f26f419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 0x7f26f41a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4113960 con 0x7f26f4105230 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f41137e0 con 0x7f26f4102880 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26fb800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4113ae0 con 0x7f26f40fff10 2024-03-28T18:43:45.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 0x7f26f419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54080/0 (socket says 172.21.15.119:54080) 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 0x7f26f419e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42582/0 (socket says 172.21.15.119:42582) 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 -- 172.21.15.119:0/3348029231 learned_addr learned my addr 172.21.15.119:0/3348029231 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f9800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 0x7f26f41a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 -- 172.21.15.119:0/3348029231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 msgr2=0x7f26f419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 0x7f26f419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 -- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 msgr2=0x7f26f41a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 0x7f26f41a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.983+0000 7f26f3e00640 1 -- 172.21.15.119:0/3348029231 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f41a5bd0 con 0x7f26f4102880 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26f3e00640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02c8f0 tx=0x7f26dc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc041070 con 0x7f26f4102880 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26dc0044d0 con 0x7f26f4102880 2024-03-28T18:43:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc034440 con 0x7f26f4102880 2024-03-28T18:43:45.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41a5e60 con 0x7f26f4102880 2024-03-28T18:43:45.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.984+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41a62c0 con 0x7f26f4102880 2024-03-28T18:43:45.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.985+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f40fd620 con 0x7f26f4102880 2024-03-28T18:43:45.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.987+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f26dc044070 con 0x7f26f4102880 2024-03-28T18:43:45.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.988+0000 7f26f1600640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:45.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.989+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f26dc0f0730 con 0x7f26f4102880 2024-03-28T18:43:45.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.989+0000 7f26f8e00640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:45.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.989+0000 7f26f8e00640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26e4005fd0 tx=0x7f26e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:45.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:45.992+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26dc0bd6f0 con 0x7f26f4102880 2024-03-28T18:43:46.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.190+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26f4062340 con 0x7f26f4102880 2024-03-28T18:43:46.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.191+0000 7f26f1600640 1 -- 172.21.15.119:0/3348029231 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f26dc0bcff0 con 0x7f26f4102880 2024-03-28T18:43:46.191 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 msgr2=0x7f26d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26e4005fd0 tx=0x7f26e4009500 comp rx=0 tx=0).stop 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 msgr2=0x7f26f419e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f26dc02c8f0 tx=0x7f26dc005d50 comp rx=0 tx=0).stop 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 shutdown_connections 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26f4105230 0x7f26f41a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f26f4102880 0x7f26f419e540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 --2- 172.21.15.119:0/3348029231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26f40fff10 0x7f26f419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 >> 172.21.15.119:0/3348029231 conn(0x7f26f40fbb30 msgr2=0x7f26f4100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:46.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 shutdown_connections 2024-03-28T18:43:46.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.194+0000 7f26fb800640 1 -- 172.21.15.119:0/3348029231 wait complete. 2024-03-28T18:43:46.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 -- 172.21.15.119:0/3360924369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 msgr2=0x7f3a80104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 --2- 172.21.15.119:0/3360924369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a80104f30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009900 tx=0x7f3a6802bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 -- 172.21.15.119:0/3360924369 shutdown_connections 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 --2- 172.21.15.119:0/3360924369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a8010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 --2- 172.21.15.119:0/3360924369 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a80104f30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 --2- 172.21.15.119:0/3360924369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 -- 172.21.15.119:0/3360924369 >> 172.21.15.119:0/3360924369 conn(0x7f3a800fee50 msgr2=0x7f3a80101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:46.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 -- 172.21.15.119:0/3360924369 shutdown_connections 2024-03-28T18:43:46.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.422+0000 7f3a86400640 1 -- 172.21.15.119:0/3360924369 wait complete. 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.423+0000 7f3a86400640 1 Processor -- start 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.423+0000 7f3a86400640 1 -- start start 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a801a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8010f570 con 0x7f3a801038c0 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8010f3f0 con 0x7f3a80104ab0 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a86400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8010f6f0 con 0x7f3a80105470 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a7f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a7fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a801a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a7f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58542/0 (socket says 172.21.15.119:58542) 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54088/0 (socket says 172.21.15.119:54088) 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a7f400640 1 -- 172.21.15.119:0/2957737632 learned_addr learned my addr 172.21.15.119:0/2957737632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 -- 172.21.15.119:0/2957737632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 msgr2=0x7f3a801a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a801a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 -- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 msgr2=0x7f3a80199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a76a00640 1 -- 172.21.15.119:0/2957737632 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a801a17e0 con 0x7f3a80104ab0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.424+0000 7f3a7fe00640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a801a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a76a00640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036040 tx=0x7f3a6802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68041070 con 0x7f3a80104ab0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a68004450 con 0x7f3a80104ab0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a801a1a70 con 0x7f3a80104ab0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68003df0 con 0x7f3a80104ab0 2024-03-28T18:43:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.425+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a801a1f80 con 0x7f3a80104ab0 2024-03-28T18:43:46.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.426+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8010abe0 con 0x7f3a80104ab0 2024-03-28T18:43:46.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.428+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3a68044070 con 0x7f3a80104ab0 2024-03-28T18:43:46.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.429+0000 7f3a7cc00640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 0x7f3a640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.429+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f3a680f0520 con 0x7f3a80104ab0 2024-03-28T18:43:46.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.429+0000 7f3a7f400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 0x7f3a640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.430+0000 7f3a7f400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 0x7f3a640a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3a700058c0 tx=0x7f3a70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:46.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.437+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a680bd4e0 con 0x7f3a80104ab0 2024-03-28T18:43:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:46 smithi119 ceph-mon[31813]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:46 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:43:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:46 smithi119 ceph-mon[31813]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:43:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:46 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1990459716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:46 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3348029231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:46.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.622+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a800622a0 con 0x7f3a80104ab0 2024-03-28T18:43:46.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.623+0000 7f3a7cc00640 1 -- 172.21.15.119:0/2957737632 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3a680bcde0 con 0x7f3a80104ab0 2024-03-28T18:43:46.623 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 msgr2=0x7f3a640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 0x7f3a640a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3a700058c0 tx=0x7f3a70005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 msgr2=0x7f3a8019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a68036040 tx=0x7f3a6802c660 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 shutdown_connections 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f3a640a2880 0x7f3a640a4d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a80105470 0x7f3a801a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80104ab0 0x7f3a8019a100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 --2- 172.21.15.119:0/2957737632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a801038c0 0x7f3a80199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.625+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 >> 172.21.15.119:0/2957737632 conn(0x7f3a800fee50 msgr2=0x7f3a80107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.626+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 shutdown_connections 2024-03-28T18:43:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.626+0000 7f3a86400640 1 -- 172.21.15.119:0/2957737632 wait complete. 2024-03-28T18:43:46.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:46 smithi196 ceph-mon[36300]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:46 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:43:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:46 smithi196 ceph-mon[36300]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:43:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:46 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1990459716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:46 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3348029231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:46 smithi131 ceph-mon[36065]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:43:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:46 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T18:43:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:46 smithi131 ceph-mon[36065]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T18:43:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:46 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1990459716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:46.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:46 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3348029231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:46.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 -- 172.21.15.119:0/2510264486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50077c40 msgr2=0x7fee500784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 --2- 172.21.15.119:0/2510264486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50077c40 0x7fee500784e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fee48009ca0 tx=0x7fee4802c100 comp rx=0 tx=0).stop 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 -- 172.21.15.119:0/2510264486 shutdown_connections 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 --2- 172.21.15.119:0/2510264486 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50077c40 0x7fee500784e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 --2- 172.21.15.119:0/2510264486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 0x7fee50077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 --2- 172.21.15.119:0/2510264486 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50076060 0x7fee50076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 -- 172.21.15.119:0/2510264486 >> 172.21.15.119:0/2510264486 conn(0x7fee500fee50 msgr2=0x7fee50101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 -- 172.21.15.119:0/2510264486 shutdown_connections 2024-03-28T18:43:46.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.848+0000 7fee56400640 1 -- 172.21.15.119:0/2510264486 wait complete. 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.849+0000 7fee56400640 1 Processor -- start 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.849+0000 7fee56400640 1 -- start start 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 0x7fee5019e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 0x7fee501a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50113970 con 0x7fee500772a0 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee501137f0 con 0x7fee50077c40 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee56400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50113af0 con 0x7fee50076060 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42624/0 (socket says 172.21.15.119:42624) 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee46a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 0x7fee5019e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4f400640 1 -- 172.21.15.119:0/3016569522 learned_addr learned my addr 172.21.15.119:0/3016569522 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 0x7fee501a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 -- 172.21.15.119:0/3016569522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 msgr2=0x7fee5019dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 -- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 msgr2=0x7fee5019e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 0x7fee5019e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4fe00640 1 -- 172.21.15.119:0/3016569522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee501a5bc0 con 0x7fee50077c40 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.850+0000 7fee4f400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee4fe00640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 0x7fee501a5550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee48037040 tx=0x7fee48035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:46.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48039040 con 0x7fee50077c40 2024-03-28T18:43:46.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee501a5e50 con 0x7fee50077c40 2024-03-28T18:43:46.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee48044070 con 0x7fee50077c40 2024-03-28T18:43:46.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48004450 con 0x7fee50077c40 2024-03-28T18:43:46.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.851+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee501a6310 con 0x7fee50077c40 2024-03-28T18:43:46.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.852+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee50110250 con 0x7fee50077c40 2024-03-28T18:43:46.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.854+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee480045f0 con 0x7fee50077c40 2024-03-28T18:43:46.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.855+0000 7fee4cc00640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 0x7fee340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:46.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.855+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fee480f12e0 con 0x7fee50077c40 2024-03-28T18:43:46.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.856+0000 7fee4f400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 0x7fee340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.856+0000 7fee4f400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 0x7fee340a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee40011550 tx=0x7fee40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:46.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:46.860+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee480be2a0 con 0x7fee50077c40 2024-03-28T18:43:47.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.045+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fee5002d220 con 0x7fee50077c40 2024-03-28T18:43:47.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.046+0000 7fee4cc00640 1 -- 172.21.15.119:0/3016569522 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fee480098b0 con 0x7fee50077c40 2024-03-28T18:43:47.046 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 msgr2=0x7fee340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 0x7fee340a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee40011550 tx=0x7fee40005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 msgr2=0x7fee501a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 0x7fee501a5550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee48037040 tx=0x7fee48035d10 comp rx=0 tx=0).stop 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 shutdown_connections 2024-03-28T18:43:47.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee340a2800 0x7fee340a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee50077c40 0x7fee501a5550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee500772a0 0x7fee5019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 --2- 172.21.15.119:0/3016569522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee50076060 0x7fee5019dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 >> 172.21.15.119:0/3016569522 conn(0x7fee500fee50 msgr2=0x7fee50100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 shutdown_connections 2024-03-28T18:43:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.049+0000 7fee56400640 1 -- 172.21.15.119:0/3016569522 wait complete. 2024-03-28T18:43:47.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 -- 172.21.15.119:0/1610179491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 msgr2=0x7fa9f0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 --2- 172.21.15.119:0/1610179491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0103d00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009900 tx=0x7fa9e002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 -- 172.21.15.119:0/1610179491 shutdown_connections 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 --2- 172.21.15.119:0/1610179491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0105490 0x7fa9f010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 --2- 172.21.15.119:0/1610179491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0104ad0 0x7fa9f0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 --2- 172.21.15.119:0/1610179491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0103d00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 -- 172.21.15.119:0/1610179491 >> 172.21.15.119:0/1610179491 conn(0x7fa9f00fee50 msgr2=0x7fa9f0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:47.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 -- 172.21.15.119:0/1610179491 shutdown_connections 2024-03-28T18:43:47.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.267+0000 7fa9f6800640 1 -- 172.21.15.119:0/1610179491 wait complete. 2024-03-28T18:43:47.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.268+0000 7fa9f6800640 1 Processor -- start 2024-03-28T18:43:47.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.268+0000 7fa9f6800640 1 -- start start 2024-03-28T18:43:47.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.268+0000 7fa9f6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.268+0000 7fa9f6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0104ad0 0x7fa9f019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.268+0000 7fa9f6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 0x7fa9f01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9f6800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f010f580 con 0x7fa9f0104ad0 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9f6800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f010f400 con 0x7fa9f0105490 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9f6800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f010f700 con 0x7fa9f01038e0 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42632/0 (socket says 172.21.15.119:42632) 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9ef400640 1 -- 172.21.15.119:0/3291741073 learned_addr learned my addr 172.21.15.119:0/3291741073 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 0x7fa9f01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9e6a00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0104ad0 0x7fa9f019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 -- 172.21.15.119:0/3291741073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 msgr2=0x7fa9f0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 -- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0104ad0 msgr2=0x7fa9f019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0104ad0 0x7fa9f019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 -- 172.21.15.119:0/3291741073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f01a1850 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.269+0000 7fa9efe00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 0x7fa9f01a11b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009fd0 tx=0x7fa9e800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.270+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800ce20 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.270+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9e8005020 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.270+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8010720 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.270+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f01a1b40 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.270+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f01a2090 con 0x7fa9f0105490 2024-03-28T18:43:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.271+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f010ac00 con 0x7fa9f0105490 2024-03-28T18:43:47.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.272+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa9e8005540 con 0x7fa9f0105490 2024-03-28T18:43:47.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.274+0000 7fa9ecc00640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 0x7fa9d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.274+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fa9e80cf260 con 0x7fa9f0105490 2024-03-28T18:43:47.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.274+0000 7fa9ef400640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 0x7fa9d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.275+0000 7fa9ef400640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 0x7fa9d40a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9f019b110 tx=0x7fa9e002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:47.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.277+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa9e809c220 con 0x7fa9f0105490 2024-03-28T18:43:47.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:47 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2957737632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:47 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3016569522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.469+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9f00622a0 con 0x7fa9f0105490 2024-03-28T18:43:47.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.471+0000 7fa9ecc00640 1 -- 172.21.15.119:0/3291741073 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7fa9e809bb20 con 0x7fa9f0105490 2024-03-28T18:43:47.471 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:47.472 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.473+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 msgr2=0x7fa9d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.473+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 0x7fa9d40a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9f019b110 tx=0x7fa9e002df90 comp rx=0 tx=0).stop 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.473+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 msgr2=0x7fa9f01a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.473+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 0x7fa9f01a11b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009fd0 tx=0x7fa9e800ef10 comp rx=0 tx=0).stop 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 shutdown_connections 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9d40a2880 0x7fa9d40a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9f0105490 0x7fa9f01a11b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9f0104ad0 0x7fa9f019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 --2- 172.21.15.119:0/3291741073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9f01038e0 0x7fa9f0199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 >> 172.21.15.119:0/3291741073 conn(0x7fa9f00fee50 msgr2=0x7fa9f0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:47.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 shutdown_connections 2024-03-28T18:43:47.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.474+0000 7fa9f6800640 1 -- 172.21.15.119:0/3291741073 wait complete. 2024-03-28T18:43:47.492 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:47.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:47.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:47 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2957737632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:47 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3016569522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:47 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2957737632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:47 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3016569522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 -- 172.21.15.119:0/1345162751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 msgr2=0x7ff828103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 --2- 172.21.15.119:0/1345162751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828103d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009860 tx=0x7ff81c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 -- 172.21.15.119:0/1345162751 shutdown_connections 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 --2- 172.21.15.119:0/1345162751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828105490 0x7ff82810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 --2- 172.21.15.119:0/1345162751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828104ad0 0x7ff828104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 --2- 172.21.15.119:0/1345162751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828103d00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 -- 172.21.15.119:0/1345162751 >> 172.21.15.119:0/1345162751 conn(0x7ff8280fee50 msgr2=0x7ff828101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 -- 172.21.15.119:0/1345162751 shutdown_connections 2024-03-28T18:43:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.703+0000 7ff830400640 1 -- 172.21.15.119:0/1345162751 wait complete. 2024-03-28T18:43:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 Processor -- start 2024-03-28T18:43:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 -- start start 2024-03-28T18:43:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828105490 0x7ff8281a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82810f560 con 0x7ff828105490 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82810f3e0 con 0x7ff8281038e0 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff830400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82810f6e0 con 0x7ff828104ad0 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82da00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82e400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828105490 0x7ff8281a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82da00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54150/0 (socket says 172.21.15.119:54150) 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82da00640 1 -- 172.21.15.119:0/2990051649 learned_addr learned my addr 172.21.15.119:0/2990051649 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.704+0000 7ff82d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42644/0 (socket says 172.21.15.119:42644) 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 -- 172.21.15.119:0/2990051649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 msgr2=0x7ff82819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 -- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828105490 msgr2=0x7ff8281a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828105490 0x7ff8281a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 -- 172.21.15.119:0/2990051649 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8281a1860 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82d000640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff82da00640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0349e0 tx=0x7ff81c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c041020 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8281a1af0 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff81c02c9f0 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.705+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c039040 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.706+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8281a1fe0 con 0x7ff8281038e0 2024-03-28T18:43:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.707+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82810aca0 con 0x7ff8281038e0 2024-03-28T18:43:47.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.708+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff81c044020 con 0x7ff8281038e0 2024-03-28T18:43:47.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.709+0000 7ff826000640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 0x7ff8100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:47.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.710+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7ff81c0f0430 con 0x7ff8281038e0 2024-03-28T18:43:47.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.710+0000 7ff82d000640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 0x7ff8100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:47.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.710+0000 7ff82d000640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 0x7ff8100a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff814009fc0 tx=0x7ff814005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:47.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.714+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff81c04b020 con 0x7ff8281038e0 2024-03-28T18:43:47.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.914+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff82802d290 con 0x7ff8281038e0 2024-03-28T18:43:47.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.915+0000 7ff826000640 1 -- 172.21.15.119:0/2990051649 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff81c0bd3f0 con 0x7ff8281038e0 2024-03-28T18:43:47.915 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.917+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 msgr2=0x7ff8100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.917+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 0x7ff8100a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff814009fc0 tx=0x7ff814005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 msgr2=0x7ff828199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0349e0 tx=0x7ff81c032430 comp rx=0 tx=0).stop 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 shutdown_connections 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7ff8100a2800 0x7ff8100a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff828105490 0x7ff8281a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff828104ad0 0x7ff82819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 --2- 172.21.15.119:0/2990051649 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8281038e0 0x7ff828199c50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:47.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 >> 172.21.15.119:0/2990051649 conn(0x7ff8280fee50 msgr2=0x7ff828107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:47.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 shutdown_connections 2024-03-28T18:43:47.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:47.918+0000 7ff830400640 1 -- 172.21.15.119:0/2990051649 wait complete. 2024-03-28T18:43:47.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.145+0000 7f2d8da00640 1 -- 172.21.15.119:0/3974834810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88104ad0 msgr2=0x7f2d88104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 --2- 172.21.15.119:0/3974834810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88104ad0 0x7f2d88104f50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009900 tx=0x7f2d7402bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 -- 172.21.15.119:0/3974834810 shutdown_connections 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 --2- 172.21.15.119:0/3974834810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d88105490 0x7f2d8810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 --2- 172.21.15.119:0/3974834810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88104ad0 0x7f2d88104f50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 --2- 172.21.15.119:0/3974834810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d881038e0 0x7f2d88103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 -- 172.21.15.119:0/3974834810 >> 172.21.15.119:0/3974834810 conn(0x7f2d880fee50 msgr2=0x7f2d88101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 -- 172.21.15.119:0/3974834810 shutdown_connections 2024-03-28T18:43:48.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.146+0000 7f2d8da00640 1 -- 172.21.15.119:0/3974834810 wait complete. 2024-03-28T18:43:48.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 Processor -- start 2024-03-28T18:43:48.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 -- start start 2024-03-28T18:43:48.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d88104ad0 0x7f2d8819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 0x7f2d881a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8810f590 con 0x7f2d88104ad0 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8810f410 con 0x7f2d88105490 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d8da00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8810f710 con 0x7f2d881038e0 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d86a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d87400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 0x7f2d881a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d86000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d88104ad0 0x7f2d8819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d86a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42668/0 (socket says 172.21.15.119:42668) 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.147+0000 7f2d86a00640 1 -- 172.21.15.119:0/1559138543 learned_addr learned my addr 172.21.15.119:0/1559138543 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 -- 172.21.15.119:0/1559138543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 msgr2=0x7f2d881a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 0x7f2d881a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 -- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d88104ad0 msgr2=0x7f2d8819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d88104ad0 0x7f2d8819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 -- 172.21.15.119:0/1559138543 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d881a1870 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d87400640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 0x7f2d881a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d86a00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800d950 tx=0x7f2d7800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78017020 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d881a1b60 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d78009c30 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.148+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800f6b0 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.149+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d881a2100 con 0x7f2d881038e0 2024-03-28T18:43:48.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.150+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8810aba0 con 0x7f2d881038e0 2024-03-28T18:43:48.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.151+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2d7801a020 con 0x7f2d881038e0 2024-03-28T18:43:48.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.153+0000 7f2d7f400640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.153+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f2d780cee00 con 0x7f2d881038e0 2024-03-28T18:43:48.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.153+0000 7f2d86000640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.154+0000 7f2d86000640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2d7402efd0 tx=0x7f2d7402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:48.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.159+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d78021020 con 0x7f2d881038e0 2024-03-28T18:43:48.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.365+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d8802d070 con 0x7f2d881038e0 2024-03-28T18:43:48.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.365+0000 7f2d7f400640 1 -- 172.21.15.119:0/1559138543 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2d7809bdc0 con 0x7f2d881038e0 2024-03-28T18:43:48.365 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:48.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.368+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 msgr2=0x7f2d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.368+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2d7402efd0 tx=0x7f2d7402cf90 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.368+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 msgr2=0x7f2d88199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.368+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800d950 tx=0x7f2d7800de10 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 shutdown_connections 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2d88105490 0x7f2d881a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d88104ad0 0x7f2d8819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 --2- 172.21.15.119:0/1559138543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d881038e0 0x7f2d88199c60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 >> 172.21.15.119:0/1559138543 conn(0x7f2d880fee50 msgr2=0x7f2d88107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 shutdown_connections 2024-03-28T18:43:48.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.369+0000 7f2d8da00640 1 -- 172.21.15.119:0/1559138543 wait complete. 2024-03-28T18:43:48.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:48 smithi119 ceph-mon[31813]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:43:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:48 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3291741073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:48 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2990051649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:48.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 -- 172.21.15.119:0/2300814534 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f0101790 msgr2=0x7fc1f0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/2300814534 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f0101790 0x7fc1f0101c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc009900 tx=0x7fc1dc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 -- 172.21.15.119:0/2300814534 shutdown_connections 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/2300814534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0102150 0x7fc1f01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/2300814534 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f0101790 0x7fc1f0101c10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/2300814534 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f01005a0 0x7fc1f01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 -- 172.21.15.119:0/2300814534 >> 172.21.15.119:0/2300814534 conn(0x7fc1f00fbb30 msgr2=0x7fc1f00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.575+0000 7fc1f5a00640 1 -- 172.21.15.119:0/2300814534 shutdown_connections 2024-03-28T18:43:48.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.576+0000 7fc1f5a00640 1 -- 172.21.15.119:0/2300814534 wait complete. 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.576+0000 7fc1f5a00640 1 Processor -- start 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 -- start start 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0101790 0x7fc1f019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f0102150 0x7fc1f01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f010c230 con 0x7fc1f0101790 2024-03-28T18:43:48.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f010c0b0 con 0x7fc1f0102150 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1f5a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f010c3b0 con 0x7fc1f01005a0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1eea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1eea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42672/0 (socket says 172.21.15.119:42672) 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1eea00640 1 -- 172.21.15.119:0/1638260716 learned_addr learned my addr 172.21.15.119:0/1638260716 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1ee000640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0101790 0x7fc1f019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.577+0000 7fc1ef400640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f0102150 0x7fc1f01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 -- 172.21.15.119:0/1638260716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f0102150 msgr2=0x7fc1f01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f0102150 0x7fc1f01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 -- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0101790 msgr2=0x7fc1f019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0101790 0x7fc1f019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 -- 172.21.15.119:0/1638260716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f01a1840 con 0x7fc1f01005a0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1eea00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000e9c0 tx=0x7fc1e000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000cd80 con 0x7fc1f01005a0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e00035f0 con 0x7fc1f01005a0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f01a1b30 con 0x7fc1f01005a0 2024-03-28T18:43:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0010690 con 0x7fc1f01005a0 2024-03-28T18:43:48.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.578+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f01a2080 con 0x7fc1f01005a0 2024-03-28T18:43:48.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.579+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f01a2250 con 0x7fc1f01005a0 2024-03-28T18:43:48.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.581+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc1e00037a0 con 0x7fc1f01005a0 2024-03-28T18:43:48.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.582+0000 7fc1e7400640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 0x7fc1d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:48.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.583+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fc1e00ceea0 con 0x7fc1f01005a0 2024-03-28T18:43:48.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.583+0000 7fc1ee000640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 0x7fc1d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:48.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.583+0000 7fc1ee000640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 0x7fc1d40a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc02c7b0 tx=0x7fc1dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:48.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.586+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1e009be60 con 0x7fc1f01005a0 2024-03-28T18:43:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:48 smithi196 ceph-mon[36300]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:43:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:48 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3291741073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:48 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2990051649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:48 smithi131 ceph-mon[36065]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:43:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:48 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3291741073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:48 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2990051649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:48.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.784+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc1f01a2420 con 0x7fc1f01005a0 2024-03-28T18:43:48.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.785+0000 7fc1e7400640 1 -- 172.21.15.119:0/1638260716 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc1e009b760 con 0x7fc1f01005a0 2024-03-28T18:43:48.785 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 msgr2=0x7fc1d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 0x7fc1d40a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc02c7b0 tx=0x7fc1dc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 msgr2=0x7fc1f0199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000e9c0 tx=0x7fc1e000ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 shutdown_connections 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc1d40a2880 0x7fc1d40a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc1f0102150 0x7fc1f01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f0101790 0x7fc1f019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 --2- 172.21.15.119:0/1638260716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1f01005a0 0x7fc1f0199c30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.787+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 >> 172.21.15.119:0/1638260716 conn(0x7fc1f00fbb30 msgr2=0x7fc1f00fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.788+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 shutdown_connections 2024-03-28T18:43:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:48.788+0000 7fc1f5a00640 1 -- 172.21.15.119:0/1638260716 wait complete. 2024-03-28T18:43:48.807 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-28T18:43:48.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:49.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 -- 172.21.15.119:0/3720366972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d81030c0 msgr2=0x7fa9d8109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 --2- 172.21.15.119:0/3720366972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d81030c0 0x7fa9d8109950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0009be0 tx=0x7fa9d002bef0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 -- 172.21.15.119:0/3720366972 shutdown_connections 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 --2- 172.21.15.119:0/3720366972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d81030c0 0x7fa9d8109950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 --2- 172.21.15.119:0/3720366972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d8102700 0x7fa9d8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 --2- 172.21.15.119:0/3720366972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d8101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 -- 172.21.15.119:0/3720366972 >> 172.21.15.119:0/3720366972 conn(0x7fa9d80fca80 msgr2=0x7fa9d80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 -- 172.21.15.119:0/3720366972 shutdown_connections 2024-03-28T18:43:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.001+0000 7fa9dea00640 1 -- 172.21.15.119:0/3720366972 wait complete. 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 Processor -- start 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 -- start start 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d8102700 0x7fa9d8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 0x7fa9d81a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.002+0000 7fa9dea00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d810d1c0 con 0x7fa9d8102700 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dea00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d810d040 con 0x7fa9d81030c0 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dea00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d810d340 con 0x7fa9d8101510 2024-03-28T18:43:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35924/0 (socket says 172.21.15.119:35924) 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 -- 172.21.15.119:0/2636552283 learned_addr learned my addr 172.21.15.119:0/2636552283 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dcc00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d8102700 0x7fa9d8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9de000640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 0x7fa9d81a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 -- 172.21.15.119:0/2636552283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 msgr2=0x7fa9d81a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 0x7fa9d81a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 -- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d8102700 msgr2=0x7fa9d8199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d8102700 0x7fa9d8199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9dd600640 1 -- 172.21.15.119:0/2636552283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d81a15c0 con 0x7fa9d8101510 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.003+0000 7fa9de000640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 0x7fa9d81a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9dd600640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00e9a0 tx=0x7fa9cc00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc00cd80 con 0x7fa9d8101510 2024-03-28T18:43:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d81a18b0 con 0x7fa9d8101510 2024-03-28T18:43:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc010910 con 0x7fa9d8101510 2024-03-28T18:43:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc018a30 con 0x7fa9d8101510 2024-03-28T18:43:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.004+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d81a1e00 con 0x7fa9d8101510 2024-03-28T18:43:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.005+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d80683b0 con 0x7fa9d8101510 2024-03-28T18:43:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.007+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa9cc010ac0 con 0x7fa9d8101510 2024-03-28T18:43:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.008+0000 7fa9d6000640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 0x7fa9c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.008+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fa9cc0cee50 con 0x7fa9d8101510 2024-03-28T18:43:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.008+0000 7fa9dcc00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 0x7fa9c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.009+0000 7fa9dcc00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 0x7fa9c00a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40095f0 tx=0x7fa9c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.012+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa9cc053740 con 0x7fa9d8101510 2024-03-28T18:43:49.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.276+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9d81a27b0 con 0x7fa9d8101510 2024-03-28T18:43:49.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.277+0000 7fa9d6000640 1 -- 172.21.15.119:0/2636552283 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fa9cc09be10 con 0x7fa9d8101510 2024-03-28T18:43:49.277 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:49.278 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:43:49.278 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.279+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 msgr2=0x7fa9c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 0x7fa9c00a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40095f0 tx=0x7fa9c4005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 msgr2=0x7fa9d81999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00e9a0 tx=0x7fa9cc00ee60 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 shutdown_connections 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa9c00a2800 0x7fa9c00a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa9d81030c0 0x7fa9d81a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9d8102700 0x7fa9d8199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 --2- 172.21.15.119:0/2636552283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d8101510 0x7fa9d81999a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 >> 172.21.15.119:0/2636552283 conn(0x7fa9d80fca80 msgr2=0x7fa9d80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 shutdown_connections 2024-03-28T18:43:49.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.280+0000 7fa9dea00640 1 -- 172.21.15.119:0/2636552283 wait complete. 2024-03-28T18:43:49.297 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-28T18:43:49.297 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-28T18:43:49.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 -- 172.21.15.119:0/2292039405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c104ad0 msgr2=0x7fdc7c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 --2- 172.21.15.119:0/2292039405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c104f50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009900 tx=0x7fdc6802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 -- 172.21.15.119:0/2292039405 shutdown_connections 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 --2- 172.21.15.119:0/2292039405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c105490 0x7fdc7c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 --2- 172.21.15.119:0/2292039405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c104f50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 --2- 172.21.15.119:0/2292039405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 -- 172.21.15.119:0/2292039405 >> 172.21.15.119:0/2292039405 conn(0x7fdc7c0fee90 msgr2=0x7fdc7c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 -- 172.21.15.119:0/2292039405 shutdown_connections 2024-03-28T18:43:49.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.493+0000 7fdc82200640 1 -- 172.21.15.119:0/2292039405 wait complete. 2024-03-28T18:43:49.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 Processor -- start 2024-03-28T18:43:49.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 -- start start 2024-03-28T18:43:49.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 0x7fdc7c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c10f540 con 0x7fdc7c104ad0 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c10f3c0 con 0x7fdc7c1038e0 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc82200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c10f6c0 con 0x7fdc7c105490 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc7b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc7b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58984/0 (socket says 172.21.15.119:58984) 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc7b400640 1 -- 172.21.15.119:0/1690238656 learned_addr learned my addr 172.21.15.119:0/1690238656 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc7be00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 0x7fdc7c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.494+0000 7fdc7aa00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 -- 172.21.15.119:0/1690238656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 msgr2=0x7fdc7c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 0x7fdc7c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 -- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c104ad0 msgr2=0x7fdc7c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 -- 172.21.15.119:0/1690238656 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c1a1800 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7be00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 0x7fdc7c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc7b400640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00e9c0 tx=0x7fdc6c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c00cd80 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c1a1af0 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc6c0035f0 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.495+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c010690 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.496+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1a2010 con 0x7fdc7c1038e0 2024-03-28T18:43:49.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.497+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c10ac00 con 0x7fdc7c1038e0 2024-03-28T18:43:49.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.498+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdc6c0037a0 con 0x7fdc7c1038e0 2024-03-28T18:43:49.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.499+0000 7fdc73e00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 0x7fdc600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.500+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fdc6c0cef20 con 0x7fdc7c1038e0 2024-03-28T18:43:49.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.500+0000 7fdc7aa00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 0x7fdc600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.501+0000 7fdc7aa00640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 0x7fdc600a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c7b0 tx=0x7fdc68005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.503+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc6c09bee0 con 0x7fdc7c1038e0 2024-03-28T18:43:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:49 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1559138543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:49 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1638260716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:49 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:49.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:49 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1559138543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:49 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1638260716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:49 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:43:49.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:49 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1559138543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:49 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1638260716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:49 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:49.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.699+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc7c061ae0 con 0x7fdc600a2800 2024-03-28T18:43:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.708+0000 7fdc73e00640 1 -- 172.21.15.119:0/1690238656 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259126 (secure 0 0 0) 0x7fdc7c061ae0 con 0x7fdc600a2800 2024-03-28T18:43:49.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:49.713 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:43:49.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.712+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 msgr2=0x7fdc600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.712+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 0x7fdc600a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802c7b0 tx=0x7fdc68005fd0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.712+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 msgr2=0x7fdc7c199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.712+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00e9c0 tx=0x7fdc6c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 shutdown_connections 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fdc600a2800 0x7fdc600a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc7c105490 0x7fdc7c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc7c104ad0 0x7fdc7c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 --2- 172.21.15.119:0/1690238656 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc7c1038e0 0x7fdc7c199be0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 >> 172.21.15.119:0/1690238656 conn(0x7fdc7c0fee90 msgr2=0x7fdc7c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 shutdown_connections 2024-03-28T18:43:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.713+0000 7fdc82200640 1 -- 172.21.15.119:0/1690238656 wait complete. 2024-03-28T18:43:49.732 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-03-28T18:43:49.022675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660650,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1565,"num_read_kb":909,"num_write":438,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361260,"kb_used_data":12108,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580564,"statfs":{"total":1151940427776,"available":1151570497536,"internally_reserved":0,"allocated":12398592,"data_stored":7694490,"data_compressed":246444,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":72,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986177+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:42:58.986177+0000","last_peered":"2024-03-28T18:42:58.986177+0000","last_clean":"2024-03-28T18:42:58.986177+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:42:58.986177+0000","last_undegraded":"2024-03-28T18:42:58.986177+0000","last_fullsized":"2024-03-28T18:42:58.986177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988151+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:42:58.988151+0000","last_peered":"2024-03-28T18:42:58.988151+0000","last_clean":"2024-03-28T18:42:58.988151+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:42:58.988151+0000","last_undegraded":"2024-03-28T18:42:58.988151+0000","last_fullsized":"2024-03-28T18:42:58.988151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987731+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:42:58.987731+0000","last_peered":"2024-03-28T18:42:58.987731+0000","last_clean":"2024-03-28T18:42:58.987731+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:42:58.987731+0000","last_undegraded":"2024-03-28T18:42:58.987731+0000","last_fullsized":"2024-03-28T18:42:58.987731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964648+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:42:58.964648+0000","last_peered":"2024-03-28T18:42:58.964648+0000","last_clean":"2024-03-28T18:42:58.964648+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:42:58.964648+0000","last_undegraded":"2024-03-28T18:42:58.964648+0000","last_fullsized":"2024-03-28T18:42:58.964648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.973904+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:42:58.973904+0000","last_peered":"2024-03-28T18:42:58.973904+0000","last_clean":"2024-03-28T18:42:58.973904+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:42:58.973904+0000","last_undegraded":"2024-03-28T18:42:58.973904+0000","last_fullsized":"2024-03-28T18:42:58.973904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978449+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:42:58.978449+0000","last_peered":"2024-03-28T18:42:58.978449+0000","last_clean":"2024-03-28T18:42:58.978449+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:42:58.978449+0000","last_undegraded":"2024-03-28T18:42:58.978449+0000","last_fullsized":"2024-03-28T18:42:58.978449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983496+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:42:58.983496+0000","last_peered":"2024-03-28T18:42:58.983496+0000","last_clean":"2024-03-28T18:42:58.983496+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:42:58.983496+0000","last_undegraded":"2024-03-28T18:42:58.983496+0000","last_fullsized":"2024-03-28T18:42:58.983496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979669+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:42:58.979669+0000","last_peered":"2024-03-28T18:42:58.979669+0000","last_clean":"2024-03-28T18:42:58.979669+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:42:58.979669+0000","last_undegraded":"2024-03-28T18:42:58.979669+0000","last_fullsized":"2024-03-28T18:42:58.979669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972427+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:42:58.972427+0000","last_peered":"2024-03-28T18:42:58.972427+0000","last_clean":"2024-03-28T18:42:58.972427+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:42:58.972427+0000","last_undegraded":"2024-03-28T18:42:58.972427+0000","last_fullsized":"2024-03-28T18:42:58.972427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.961476+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:42:58.961476+0000","last_peered":"2024-03-28T18:42:58.961476+0000","last_clean":"2024-03-28T18:42:58.961476+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:42:58.961476+0000","last_undegraded":"2024-03-28T18:42:58.961476+0000","last_fullsized":"2024-03-28T18:42:58.961476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978753+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:42:58.978753+0000","last_peered":"2024-03-28T18:42:58.978753+0000","last_clean":"2024-03-28T18:42:58.978753+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:42:58.978753+0000","last_undegraded":"2024-03-28T18:42:58.978753+0000","last_fullsized":"2024-03-28T18:42:58.978753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957622+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:42:58.957622+0000","last_peered":"2024-03-28T18:42:58.957622+0000","last_clean":"2024-03-28T18:42:58.957622+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:42:58.957622+0000","last_undegraded":"2024-03-28T18:42:58.957622+0000","last_fullsized":"2024-03-28T18:42:58.957622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972016+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:42:58.972016+0000","last_peered":"2024-03-28T18:42:58.972016+0000","last_clean":"2024-03-28T18:42:58.972016+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:42:58.972016+0000","last_undegraded":"2024-03-28T18:42:58.972016+0000","last_fullsized":"2024-03-28T18:42:58.972016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987551+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:42:58.987551+0000","last_peered":"2024-03-28T18:42:58.987551+0000","last_clean":"2024-03-28T18:42:58.987551+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:42:58.987551+0000","last_undegraded":"2024-03-28T18:42:58.987551+0000","last_fullsized":"2024-03-28T18:42:58.987551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987723+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:42:58.987723+0000","last_peered":"2024-03-28T18:42:58.987723+0000","last_clean":"2024-03-28T18:42:58.987723+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:42:58.987723+0000","last_undegraded":"2024-03-28T18:42:58.987723+0000","last_fullsized":"2024-03-28T18:42:58.987723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989419+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:42:58.989419+0000","last_peered":"2024-03-28T18:42:58.989419+0000","last_clean":"2024-03-28T18:42:58.989419+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:42:58.989419+0000","last_undegraded":"2024-03-28T18:42:58.989419+0000","last_fullsized":"2024-03-28T18:42:58.989419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986304+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:42:58.986304+0000","last_peered":"2024-03-28T18:42:58.986304+0000","last_clean":"2024-03-28T18:42:58.986304+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:42:58.986304+0000","last_undegraded":"2024-03-28T18:42:58.986304+0000","last_fullsized":"2024-03-28T18:42:58.986304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980073+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:42:58.980073+0000","last_peered":"2024-03-28T18:42:58.980073+0000","last_clean":"2024-03-28T18:42:58.980073+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:42:58.980073+0000","last_undegraded":"2024-03-28T18:42:58.980073+0000","last_fullsized":"2024-03-28T18:42:58.980073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983143+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:42:58.983143+0000","last_peered":"2024-03-28T18:42:58.983143+0000","last_clean":"2024-03-28T18:42:58.983143+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:42:58.983143+0000","last_undegraded":"2024-03-28T18:42:58.983143+0000","last_fullsized":"2024-03-28T18:42:58.983143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.509497+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:43:43.509497+0000","last_peered":"2024-03-28T18:43:43.509497+0000","last_clean":"2024-03-28T18:43:43.509497+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:43:43.509497+0000","last_undegraded":"2024-03-28T18:43:43.509497+0000","last_fullsized":"2024-03-28T18:43:43.509497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976163+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:42:58.976163+0000","last_peered":"2024-03-28T18:42:58.976163+0000","last_clean":"2024-03-28T18:42:58.976163+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:42:58.976163+0000","last_undegraded":"2024-03-28T18:42:58.976163+0000","last_fullsized":"2024-03-28T18:42:58.976163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.405108+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:42:31.405108+0000","last_peered":"2024-03-28T18:42:31.405108+0000","last_clean":"2024-03-28T18:42:31.405108+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:42:31.405108+0000","last_undegraded":"2024-03-28T18:42:31.405108+0000","last_fullsized":"2024-03-28T18:42:31.405108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987959+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:42:58.987959+0000","last_peered":"2024-03-28T18:42:58.987959+0000","last_clean":"2024-03-28T18:42:58.987959+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:42:58.987959+0000","last_undegraded":"2024-03-28T18:42:58.987959+0000","last_fullsized":"2024-03-28T18:42:58.987959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239229+0000","last_change":"2024-03-28T18:41:30.335571+0000","last_active":"2024-03-28T18:42:31.239229+0000","last_peered":"2024-03-28T18:42:31.239229+0000","last_clean":"2024-03-28T18:42:31.239229+0000","last_became_active":"2024-03-28T18:41:30.311420+0000","last_became_peered":"2024-03-28T18:41:30.311420+0000","last_unstale":"2024-03-28T18:42:31.239229+0000","last_undegraded":"2024-03-28T18:42:31.239229+0000","last_fullsized":"2024-03-28T18:42:31.239229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:39.796667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972421+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:42:58.972421+0000","last_peered":"2024-03-28T18:42:58.972421+0000","last_clean":"2024-03-28T18:42:58.972421+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:42:58.972421+0000","last_undegraded":"2024-03-28T18:42:58.972421+0000","last_fullsized":"2024-03-28T18:42:58.972421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239193+0000","last_change":"2024-03-28T18:41:30.315981+0000","last_active":"2024-03-28T18:42:31.239193+0000","last_peered":"2024-03-28T18:42:31.239193+0000","last_clean":"2024-03-28T18:42:31.239193+0000","last_became_active":"2024-03-28T18:41:30.311208+0000","last_became_peered":"2024-03-28T18:41:30.311208+0000","last_unstale":"2024-03-28T18:42:31.239193+0000","last_undegraded":"2024-03-28T18:42:31.239193+0000","last_fullsized":"2024-03-28T18:42:31.239193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:10.817446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.962262+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:42:58.962262+0000","last_peered":"2024-03-28T18:42:58.962262+0000","last_clean":"2024-03-28T18:42:58.962262+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:42:58.962262+0000","last_undegraded":"2024-03-28T18:42:58.962262+0000","last_fullsized":"2024-03-28T18:42:58.962262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:46.346077+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:43:46.346077+0000","last_peered":"2024-03-28T18:43:46.346077+0000","last_clean":"2024-03-28T18:43:46.346077+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:43:46.346077+0000","last_undegraded":"2024-03-28T18:43:46.346077+0000","last_fullsized":"2024-03-28T18:43:46.346077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.969774+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:42:58.969774+0000","last_peered":"2024-03-28T18:42:58.969774+0000","last_clean":"2024-03-28T18:42:58.969774+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:42:58.969774+0000","last_undegraded":"2024-03-28T18:42:58.969774+0000","last_fullsized":"2024-03-28T18:42:58.969774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269565+0000","last_change":"2024-03-28T18:41:30.228521+0000","last_active":"2024-03-28T18:42:31.269565+0000","last_peered":"2024-03-28T18:42:31.269565+0000","last_clean":"2024-03-28T18:42:31.269565+0000","last_became_active":"2024-03-28T18:41:30.221867+0000","last_became_peered":"2024-03-28T18:41:30.221867+0000","last_unstale":"2024-03-28T18:42:31.269565+0000","last_undegraded":"2024-03-28T18:42:31.269565+0000","last_fullsized":"2024-03-28T18:42:31.269565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:45.207508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968706+0000","last_change":"2024-03-28T18:41:30.507663+0000","last_active":"2024-03-28T18:42:58.968706+0000","last_peered":"2024-03-28T18:42:58.968706+0000","last_clean":"2024-03-28T18:42:58.968706+0000","last_became_active":"2024-03-28T18:41:30.507284+0000","last_became_peered":"2024-03-28T18:41:30.507284+0000","last_unstale":"2024-03-28T18:42:58.968706+0000","last_undegraded":"2024-03-28T18:42:58.968706+0000","last_fullsized":"2024-03-28T18:42:58.968706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:03.045818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175185+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:42:31.175185+0000","last_peered":"2024-03-28T18:42:31.175185+0000","last_clean":"2024-03-28T18:42:31.175185+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:42:31.175185+0000","last_undegraded":"2024-03-28T18:42:31.175185+0000","last_fullsized":"2024-03-28T18:42:31.175185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982042+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:42:58.982042+0000","last_peered":"2024-03-28T18:42:58.982042+0000","last_clean":"2024-03-28T18:42:58.982042+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:42:58.982042+0000","last_undegraded":"2024-03-28T18:42:58.982042+0000","last_fullsized":"2024-03-28T18:42:58.982042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168096+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:42:31.168096+0000","last_peered":"2024-03-28T18:42:31.168096+0000","last_clean":"2024-03-28T18:42:31.168096+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:42:31.168096+0000","last_undegraded":"2024-03-28T18:42:31.168096+0000","last_fullsized":"2024-03-28T18:42:31.168096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980414+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:42:58.980414+0000","last_peered":"2024-03-28T18:42:58.980414+0000","last_clean":"2024-03-28T18:42:58.980414+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:42:58.980414+0000","last_undegraded":"2024-03-28T18:42:58.980414+0000","last_fullsized":"2024-03-28T18:42:58.980414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.439337+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:43:47.439337+0000","last_peered":"2024-03-28T18:43:47.439337+0000","last_clean":"2024-03-28T18:43:47.439337+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:43:47.439337+0000","last_undegraded":"2024-03-28T18:43:47.439337+0000","last_fullsized":"2024-03-28T18:43:47.439337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":280,"num_read_kb":280,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982861+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:42:58.982861+0000","last_peered":"2024-03-28T18:42:58.982861+0000","last_clean":"2024-03-28T18:42:58.982861+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:42:58.982861+0000","last_undegraded":"2024-03-28T18:42:58.982861+0000","last_fullsized":"2024-03-28T18:42:58.982861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360380+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:42:31.360380+0000","last_peered":"2024-03-28T18:42:31.360380+0000","last_clean":"2024-03-28T18:42:31.360380+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:42:31.360380+0000","last_undegraded":"2024-03-28T18:42:31.360380+0000","last_fullsized":"2024-03-28T18:42:31.360380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987661+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:42:58.987661+0000","last_peered":"2024-03-28T18:42:58.987661+0000","last_clean":"2024-03-28T18:42:58.987661+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:42:58.987661+0000","last_undegraded":"2024-03-28T18:42:58.987661+0000","last_fullsized":"2024-03-28T18:42:58.987661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168678+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:42:31.168678+0000","last_peered":"2024-03-28T18:42:31.168678+0000","last_clean":"2024-03-28T18:42:31.168678+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:42:31.168678+0000","last_undegraded":"2024-03-28T18:42:31.168678+0000","last_fullsized":"2024-03-28T18:42:31.168678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978619+0000","last_change":"2024-03-28T18:42:31.303127+0000","last_active":"2024-03-28T18:42:58.978619+0000","last_peered":"2024-03-28T18:42:58.978619+0000","last_clean":"2024-03-28T18:42:58.978619+0000","last_became_active":"2024-03-28T18:42:31.301321+0000","last_became_peered":"2024-03-28T18:42:31.301321+0000","last_unstale":"2024-03-28T18:42:58.978619+0000","last_undegraded":"2024-03-28T18:42:58.978619+0000","last_fullsized":"2024-03-28T18:42:58.978619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:08.892803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":351,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.346068+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:43:47.346068+0000","last_peered":"2024-03-28T18:43:47.346068+0000","last_clean":"2024-03-28T18:43:47.346068+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:43:47.346068+0000","last_undegraded":"2024-03-28T18:43:47.346068+0000","last_fullsized":"2024-03-28T18:43:47.346068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.948299+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:42:58.948299+0000","last_peered":"2024-03-28T18:42:58.948299+0000","last_clean":"2024-03-28T18:42:58.948299+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:42:58.948299+0000","last_undegraded":"2024-03-28T18:42:58.948299+0000","last_fullsized":"2024-03-28T18:42:58.948299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317615+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:42:31.317615+0000","last_peered":"2024-03-28T18:42:31.317615+0000","last_clean":"2024-03-28T18:42:31.317615+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:42:31.317615+0000","last_undegraded":"2024-03-28T18:42:31.317615+0000","last_fullsized":"2024-03-28T18:42:31.317615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983586+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:42:58.983586+0000","last_peered":"2024-03-28T18:42:58.983586+0000","last_clean":"2024-03-28T18:42:58.983586+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:42:58.983586+0000","last_undegraded":"2024-03-28T18:42:58.983586+0000","last_fullsized":"2024-03-28T18:42:58.983586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168283+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:42:31.168283+0000","last_peered":"2024-03-28T18:42:31.168283+0000","last_clean":"2024-03-28T18:42:31.168283+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:42:31.168283+0000","last_undegraded":"2024-03-28T18:42:31.168283+0000","last_fullsized":"2024-03-28T18:42:31.168283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982443+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:42:58.982443+0000","last_peered":"2024-03-28T18:42:58.982443+0000","last_clean":"2024-03-28T18:42:58.982443+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:42:58.982443+0000","last_undegraded":"2024-03-28T18:42:58.982443+0000","last_fullsized":"2024-03-28T18:42:58.982443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.419661+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:43:43.419661+0000","last_peered":"2024-03-28T18:43:43.419661+0000","last_clean":"2024-03-28T18:43:43.419661+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:43:43.419661+0000","last_undegraded":"2024-03-28T18:43:43.419661+0000","last_fullsized":"2024-03-28T18:43:43.419661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970949+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:42:58.970949+0000","last_peered":"2024-03-28T18:42:58.970949+0000","last_clean":"2024-03-28T18:42:58.970949+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:42:58.970949+0000","last_undegraded":"2024-03-28T18:42:58.970949+0000","last_fullsized":"2024-03-28T18:42:58.970949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238896+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:42:31.238896+0000","last_peered":"2024-03-28T18:42:31.238896+0000","last_clean":"2024-03-28T18:42:31.238896+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:42:31.238896+0000","last_undegraded":"2024-03-28T18:42:31.238896+0000","last_fullsized":"2024-03-28T18:42:31.238896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957235+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:42:58.957235+0000","last_peered":"2024-03-28T18:42:58.957235+0000","last_clean":"2024-03-28T18:42:58.957235+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:42:58.957235+0000","last_undegraded":"2024-03-28T18:42:58.957235+0000","last_fullsized":"2024-03-28T18:42:58.957235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239439+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:42:31.239439+0000","last_peered":"2024-03-28T18:42:31.239439+0000","last_clean":"2024-03-28T18:42:31.239439+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:42:31.239439+0000","last_undegraded":"2024-03-28T18:42:31.239439+0000","last_fullsized":"2024-03-28T18:42:31.239439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971622+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:42:58.971622+0000","last_peered":"2024-03-28T18:42:58.971622+0000","last_clean":"2024-03-28T18:42:58.971622+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:42:58.971622+0000","last_undegraded":"2024-03-28T18:42:58.971622+0000","last_fullsized":"2024-03-28T18:42:58.971622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.173997+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:42:31.173997+0000","last_peered":"2024-03-28T18:42:31.173997+0000","last_clean":"2024-03-28T18:42:31.173997+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:42:31.173997+0000","last_undegraded":"2024-03-28T18:42:31.173997+0000","last_fullsized":"2024-03-28T18:42:31.173997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976756+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:42:58.976756+0000","last_peered":"2024-03-28T18:42:58.976756+0000","last_clean":"2024-03-28T18:42:58.976756+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:42:58.976756+0000","last_undegraded":"2024-03-28T18:42:58.976756+0000","last_fullsized":"2024-03-28T18:42:58.976756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":169,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.514348+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:43:45.514348+0000","last_peered":"2024-03-28T18:43:45.514348+0000","last_clean":"2024-03-28T18:43:45.514348+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:43:45.514348+0000","last_undegraded":"2024-03-28T18:43:45.514348+0000","last_fullsized":"2024-03-28T18:43:45.514348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988444+0000","last_change":"2024-03-28T18:41:30.219905+0000","last_active":"2024-03-28T18:42:58.988444+0000","last_peered":"2024-03-28T18:42:58.988444+0000","last_clean":"2024-03-28T18:42:58.988444+0000","last_became_active":"2024-03-28T18:41:30.219488+0000","last_became_peered":"2024-03-28T18:41:30.219488+0000","last_unstale":"2024-03-28T18:42:58.988444+0000","last_undegraded":"2024-03-28T18:42:58.988444+0000","last_fullsized":"2024-03-28T18:42:58.988444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:16.108678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975656+0000","last_change":"2024-03-28T18:42:31.320096+0000","last_active":"2024-03-28T18:42:58.975656+0000","last_peered":"2024-03-28T18:42:58.975656+0000","last_clean":"2024-03-28T18:42:58.975656+0000","last_became_active":"2024-03-28T18:42:31.318988+0000","last_became_peered":"2024-03-28T18:42:31.318988+0000","last_unstale":"2024-03-28T18:42:58.975656+0000","last_undegraded":"2024-03-28T18:42:58.975656+0000","last_fullsized":"2024-03-28T18:42:58.975656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:50.019195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.422734+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:43:45.422734+0000","last_peered":"2024-03-28T18:43:45.422734+0000","last_clean":"2024-03-28T18:43:45.422734+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:43:45.422734+0000","last_undegraded":"2024-03-28T18:43:45.422734+0000","last_fullsized":"2024-03-28T18:43:45.422734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975540+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:42:58.975540+0000","last_peered":"2024-03-28T18:42:58.975540+0000","last_clean":"2024-03-28T18:42:58.975540+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:42:58.975540+0000","last_undegraded":"2024-03-28T18:42:58.975540+0000","last_fullsized":"2024-03-28T18:42:58.975540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299771+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:42:31.299771+0000","last_peered":"2024-03-28T18:42:31.299771+0000","last_clean":"2024-03-28T18:42:31.299771+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:42:31.299771+0000","last_undegraded":"2024-03-28T18:42:31.299771+0000","last_fullsized":"2024-03-28T18:42:31.299771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360313+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:42:31.360313+0000","last_peered":"2024-03-28T18:42:31.360313+0000","last_clean":"2024-03-28T18:42:31.360313+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:42:31.360313+0000","last_undegraded":"2024-03-28T18:42:31.360313+0000","last_fullsized":"2024-03-28T18:42:31.360313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964665+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:42:58.964665+0000","last_peered":"2024-03-28T18:42:58.964665+0000","last_clean":"2024-03-28T18:42:58.964665+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:42:58.964665+0000","last_undegraded":"2024-03-28T18:42:58.964665+0000","last_fullsized":"2024-03-28T18:42:58.964665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360379+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:42:31.360379+0000","last_peered":"2024-03-28T18:42:31.360379+0000","last_clean":"2024-03-28T18:42:31.360379+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:42:31.360379+0000","last_undegraded":"2024-03-28T18:42:31.360379+0000","last_fullsized":"2024-03-28T18:42:31.360379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411326+0000","last_change":"2024-03-28T18:41:30.227916+0000","last_active":"2024-03-28T18:42:31.411326+0000","last_peered":"2024-03-28T18:42:31.411326+0000","last_clean":"2024-03-28T18:42:31.411326+0000","last_became_active":"2024-03-28T18:41:30.218947+0000","last_became_peered":"2024-03-28T18:41:30.218947+0000","last_unstale":"2024-03-28T18:42:31.411326+0000","last_undegraded":"2024-03-28T18:42:31.411326+0000","last_fullsized":"2024-03-28T18:42:31.411326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:14.922324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983366+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:42:58.983366+0000","last_peered":"2024-03-28T18:42:58.983366+0000","last_clean":"2024-03-28T18:42:58.983366+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:42:58.983366+0000","last_undegraded":"2024-03-28T18:42:58.983366+0000","last_fullsized":"2024-03-28T18:42:58.983366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:30.238634+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:43:30.238634+0000","last_peered":"2024-03-28T18:43:30.238634+0000","last_clean":"2024-03-28T18:43:30.238634+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:43:30.238634+0000","last_undegraded":"2024-03-28T18:43:30.238634+0000","last_fullsized":"2024-03-28T18:43:30.238634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983095+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:42:58.983095+0000","last_peered":"2024-03-28T18:42:58.983095+0000","last_clean":"2024-03-28T18:42:58.983095+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:42:58.983095+0000","last_undegraded":"2024-03-28T18:42:58.983095+0000","last_fullsized":"2024-03-28T18:42:58.983095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299859+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:42:31.299859+0000","last_peered":"2024-03-28T18:42:31.299859+0000","last_clean":"2024-03-28T18:42:31.299859+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:42:31.299859+0000","last_undegraded":"2024-03-28T18:42:31.299859+0000","last_fullsized":"2024-03-28T18:42:31.299859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988228+0000","last_change":"2024-03-28T18:42:31.302339+0000","last_active":"2024-03-28T18:42:58.988228+0000","last_peered":"2024-03-28T18:42:58.988228+0000","last_clean":"2024-03-28T18:42:58.988228+0000","last_became_active":"2024-03-28T18:42:31.301324+0000","last_became_peered":"2024-03-28T18:42:31.301324+0000","last_unstale":"2024-03-28T18:42:58.988228+0000","last_undegraded":"2024-03-28T18:42:58.988228+0000","last_fullsized":"2024-03-28T18:42:58.988228+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:56.490969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167540+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:42:31.167540+0000","last_peered":"2024-03-28T18:42:31.167540+0000","last_clean":"2024-03-28T18:42:31.167540+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:42:31.167540+0000","last_undegraded":"2024-03-28T18:42:31.167540+0000","last_fullsized":"2024-03-28T18:42:31.167540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.941667+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:42:58.941667+0000","last_peered":"2024-03-28T18:42:58.941667+0000","last_clean":"2024-03-28T18:42:58.941667+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:42:58.941667+0000","last_undegraded":"2024-03-28T18:42:58.941667+0000","last_fullsized":"2024-03-28T18:42:58.941667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403649+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:42:31.403649+0000","last_peered":"2024-03-28T18:42:31.403649+0000","last_clean":"2024-03-28T18:42:31.403649+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:42:31.403649+0000","last_undegraded":"2024-03-28T18:42:31.403649+0000","last_fullsized":"2024-03-28T18:42:31.403649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987979+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:42:58.987979+0000","last_peered":"2024-03-28T18:42:58.987979+0000","last_clean":"2024-03-28T18:42:58.987979+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:42:58.987979+0000","last_undegraded":"2024-03-28T18:42:58.987979+0000","last_fullsized":"2024-03-28T18:42:58.987979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317603+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:42:31.317603+0000","last_peered":"2024-03-28T18:42:31.317603+0000","last_clean":"2024-03-28T18:42:31.317603+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:42:31.317603+0000","last_undegraded":"2024-03-28T18:42:31.317603+0000","last_fullsized":"2024-03-28T18:42:31.317603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989314+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:42:58.989314+0000","last_peered":"2024-03-28T18:42:58.989314+0000","last_clean":"2024-03-28T18:42:58.989314+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:42:58.989314+0000","last_undegraded":"2024-03-28T18:42:58.989314+0000","last_fullsized":"2024-03-28T18:42:58.989314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239033+0000","last_change":"2024-03-28T18:41:30.319270+0000","last_active":"2024-03-28T18:42:31.239033+0000","last_peered":"2024-03-28T18:42:31.239033+0000","last_clean":"2024-03-28T18:42:31.239033+0000","last_became_active":"2024-03-28T18:41:30.311219+0000","last_became_peered":"2024-03-28T18:41:30.311219+0000","last_unstale":"2024-03-28T18:42:31.239033+0000","last_undegraded":"2024-03-28T18:42:31.239033+0000","last_fullsized":"2024-03-28T18:42:31.239033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:23.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982021+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:42:58.982021+0000","last_peered":"2024-03-28T18:42:58.982021+0000","last_clean":"2024-03-28T18:42:58.982021+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:42:58.982021+0000","last_undegraded":"2024-03-28T18:42:58.982021+0000","last_fullsized":"2024-03-28T18:42:58.982021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.436770+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:43:45.436770+0000","last_peered":"2024-03-28T18:43:45.436770+0000","last_clean":"2024-03-28T18:43:45.436770+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:43:45.436770+0000","last_undegraded":"2024-03-28T18:43:45.436770+0000","last_fullsized":"2024-03-28T18:43:45.436770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979050+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:42:58.979050+0000","last_peered":"2024-03-28T18:42:58.979050+0000","last_clean":"2024-03-28T18:42:58.979050+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:42:58.979050+0000","last_undegraded":"2024-03-28T18:42:58.979050+0000","last_fullsized":"2024-03-28T18:42:58.979050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168484+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:42:31.168484+0000","last_peered":"2024-03-28T18:42:31.168484+0000","last_clean":"2024-03-28T18:42:31.168484+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:42:31.168484+0000","last_undegraded":"2024-03-28T18:42:31.168484+0000","last_fullsized":"2024-03-28T18:42:31.168484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.984053+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:42:58.984053+0000","last_peered":"2024-03-28T18:42:58.984053+0000","last_clean":"2024-03-28T18:42:58.984053+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:42:58.984053+0000","last_undegraded":"2024-03-28T18:42:58.984053+0000","last_fullsized":"2024-03-28T18:42:58.984053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.176338+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:42:31.176338+0000","last_peered":"2024-03-28T18:42:31.176338+0000","last_clean":"2024-03-28T18:42:31.176338+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:42:31.176338+0000","last_undegraded":"2024-03-28T18:42:31.176338+0000","last_fullsized":"2024-03-28T18:42:31.176338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978116+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:42:58.978116+0000","last_peered":"2024-03-28T18:42:58.978116+0000","last_clean":"2024-03-28T18:42:58.978116+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:42:58.978116+0000","last_undegraded":"2024-03-28T18:42:58.978116+0000","last_fullsized":"2024-03-28T18:42:58.978116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972976+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:42:58.972976+0000","last_peered":"2024-03-28T18:42:58.972976+0000","last_clean":"2024-03-28T18:42:58.972976+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:42:58.972976+0000","last_undegraded":"2024-03-28T18:42:58.972976+0000","last_fullsized":"2024-03-28T18:42:58.972976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.967314+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:42:58.967314+0000","last_peered":"2024-03-28T18:42:58.967314+0000","last_clean":"2024-03-28T18:42:58.967314+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:42:58.967314+0000","last_undegraded":"2024-03-28T18:42:58.967314+0000","last_fullsized":"2024-03-28T18:42:58.967314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987481+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:42:58.987481+0000","last_peered":"2024-03-28T18:42:58.987481+0000","last_clean":"2024-03-28T18:42:58.987481+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:42:58.987481+0000","last_undegraded":"2024-03-28T18:42:58.987481+0000","last_fullsized":"2024-03-28T18:42:58.987481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988446+0000","last_change":"2024-03-28T18:42:31.362520+0000","last_active":"2024-03-28T18:42:58.988446+0000","last_peered":"2024-03-28T18:42:58.988446+0000","last_clean":"2024-03-28T18:42:58.988446+0000","last_became_active":"2024-03-28T18:42:31.361563+0000","last_became_peered":"2024-03-28T18:42:31.361563+0000","last_unstale":"2024-03-28T18:42:58.988446+0000","last_undegraded":"2024-03-28T18:42:58.988446+0000","last_fullsized":"2024-03-28T18:42:58.988446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:27.960042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988524+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:42:58.988524+0000","last_peered":"2024-03-28T18:42:58.988524+0000","last_clean":"2024-03-28T18:42:58.988524+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:42:58.988524+0000","last_undegraded":"2024-03-28T18:42:58.988524+0000","last_fullsized":"2024-03-28T18:42:58.988524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971656+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:42:58.971656+0000","last_peered":"2024-03-28T18:42:58.971656+0000","last_clean":"2024-03-28T18:42:58.971656+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:42:58.971656+0000","last_undegraded":"2024-03-28T18:42:58.971656+0000","last_fullsized":"2024-03-28T18:42:58.971656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970936+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:42:58.970936+0000","last_peered":"2024-03-28T18:42:58.970936+0000","last_clean":"2024-03-28T18:42:58.970936+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:42:58.970936+0000","last_undegraded":"2024-03-28T18:42:58.970936+0000","last_fullsized":"2024-03-28T18:42:58.970936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.977119+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:42:58.977119+0000","last_peered":"2024-03-28T18:42:58.977119+0000","last_clean":"2024-03-28T18:42:58.977119+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:42:58.977119+0000","last_undegraded":"2024-03-28T18:42:58.977119+0000","last_fullsized":"2024-03-28T18:42:58.977119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.981154+0000","last_change":"2024-03-28T18:41:30.510267+0000","last_active":"2024-03-28T18:42:58.981154+0000","last_peered":"2024-03-28T18:42:58.981154+0000","last_clean":"2024-03-28T18:42:58.981154+0000","last_became_active":"2024-03-28T18:41:30.508658+0000","last_became_peered":"2024-03-28T18:41:30.508658+0000","last_unstale":"2024-03-28T18:42:58.981154+0000","last_undegraded":"2024-03-28T18:42:58.981154+0000","last_fullsized":"2024-03-28T18:42:58.981154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:30.892433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982988+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:42:58.982988+0000","last_peered":"2024-03-28T18:42:58.982988+0000","last_clean":"2024-03-28T18:42:58.982988+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:42:58.982988+0000","last_undegraded":"2024-03-28T18:42:58.982988+0000","last_fullsized":"2024-03-28T18:42:58.982988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.974718+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:42:58.974718+0000","last_peered":"2024-03-28T18:42:58.974718+0000","last_clean":"2024-03-28T18:42:58.974718+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:42:58.974718+0000","last_undegraded":"2024-03-28T18:42:58.974718+0000","last_fullsized":"2024-03-28T18:42:58.974718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988976+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:42:58.988976+0000","last_peered":"2024-03-28T18:42:58.988976+0000","last_clean":"2024-03-28T18:42:58.988976+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:42:58.988976+0000","last_undegraded":"2024-03-28T18:42:58.988976+0000","last_fullsized":"2024-03-28T18:42:58.988976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968063+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:42:58.968063+0000","last_peered":"2024-03-28T18:42:58.968063+0000","last_clean":"2024-03-28T18:42:58.968063+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:42:58.968063+0000","last_undegraded":"2024-03-28T18:42:58.968063+0000","last_fullsized":"2024-03-28T18:42:58.968063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70048,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1429,"num_read_kb":790,"num_write":255,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":215517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477349,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1131054,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477167,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037820,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477433,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071042,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":476884,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":540053,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":478223,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":541111,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":476709,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1069606,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:43:49.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:43:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 -- 172.21.15.119:0/3177266548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141028b0 msgr2=0x7f0a14102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 --2- 172.21.15.119:0/3177266548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141028b0 0x7f0a14102d30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009900 tx=0x7f0a0002bc50 comp rx=0 tx=0).stop 2024-03-28T18:43:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 -- 172.21.15.119:0/3177266548 shutdown_connections 2024-03-28T18:43:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 --2- 172.21.15.119:0/3177266548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 0x7f0a14109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 --2- 172.21.15.119:0/3177266548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141028b0 0x7f0a14102d30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 --2- 172.21.15.119:0/3177266548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141016c0 0x7f0a14101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 -- 172.21.15.119:0/3177266548 >> 172.21.15.119:0/3177266548 conn(0x7f0a140fcc50 msgr2=0x7f0a140ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:49.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 -- 172.21.15.119:0/3177266548 shutdown_connections 2024-03-28T18:43:49.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.957+0000 7f0a1c400640 1 -- 172.21.15.119:0/3177266548 wait complete. 2024-03-28T18:43:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.958+0000 7f0a1c400640 1 Processor -- start 2024-03-28T18:43:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.958+0000 7f0a1c400640 1 -- start start 2024-03-28T18:43:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.958+0000 7f0a1c400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 0x7f0a14197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.958+0000 7f0a1c400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 0x7f0a14197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a1c400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 0x7f0a1419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a1c400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d340 con 0x7f0a14103270 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a1c400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d1c0 con 0x7f0a141028b0 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a1c400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1410d4c0 con 0x7f0a141016c0 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 0x7f0a14197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 0x7f0a14197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 0x7f0a14197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35968/0 (socket says 172.21.15.119:35968) 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19a00640 1 -- 172.21.15.119:0/2810091437 learned_addr learned my addr 172.21.15.119:0/2810091437 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a1a400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 0x7f0a1419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 -- 172.21.15.119:0/2810091437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 msgr2=0x7f0a14197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 0x7f0a14197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 -- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 msgr2=0x7f0a1419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 0x7f0a1419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.959+0000 7f0a19000640 1 -- 172.21.15.119:0/2810091437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1419f670 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a19000640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 0x7f0a14197f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009230 tx=0x7f0a0002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00042020 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a00004450 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1419f900 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00003df0 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.960+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1419fea0 con 0x7f0a141028b0 2024-03-28T18:43:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.961+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a141089e0 con 0x7f0a141028b0 2024-03-28T18:43:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.964+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a00045020 con 0x7f0a141028b0 2024-03-28T18:43:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.965+0000 7f0a12000640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 0x7f09fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.965+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f0a000f0da0 con 0x7f0a141028b0 2024-03-28T18:43:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.965+0000 7f0a19a00640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 0x7f09fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:49.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.966+0000 7f0a19a00640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 0x7f09fc0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a08005f40 tx=0x7f0a08009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:49.968+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a0004c020 con 0x7f0a141028b0 2024-03-28T18:43:50.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.174+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a14061ae0 con 0x7f09fc0a2880 2024-03-28T18:43:50.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.183+0000 7f0a12000640 1 -- 172.21.15.119:0/2810091437 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259126 (secure 0 0 0) 0x7f0a14061ae0 con 0x7f09fc0a2880 2024-03-28T18:43:50.187 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:50.188 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 msgr2=0x7f09fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 0x7f09fc0a4d40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0a08005f40 tx=0x7f0a08009500 comp rx=0 tx=0).stop 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 msgr2=0x7f0a14197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 0x7f0a14197f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009230 tx=0x7f0a0002c670 comp rx=0 tx=0).stop 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 shutdown_connections 2024-03-28T18:43:50.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f09fc0a2880 0x7f09fc0a4d40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a14103270 0x7f0a1419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a141028b0 0x7f0a14197f90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 --2- 172.21.15.119:0/2810091437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a141016c0 0x7f0a14197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 >> 172.21.15.119:0/2810091437 conn(0x7f0a140fcc50 msgr2=0x7f0a14105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 shutdown_connections 2024-03-28T18:43:50.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.189+0000 7f0a1c400640 1 -- 172.21.15.119:0/2810091437 wait complete. 2024-03-28T18:43:50.210 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-03-28T18:43:49.022675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660650,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1565,"num_read_kb":909,"num_write":438,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361260,"kb_used_data":12108,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580564,"statfs":{"total":1151940427776,"available":1151570497536,"internally_reserved":0,"allocated":12398592,"data_stored":7694490,"data_compressed":246444,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":72,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986177+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:42:58.986177+0000","last_peered":"2024-03-28T18:42:58.986177+0000","last_clean":"2024-03-28T18:42:58.986177+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:42:58.986177+0000","last_undegraded":"2024-03-28T18:42:58.986177+0000","last_fullsized":"2024-03-28T18:42:58.986177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988151+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:42:58.988151+0000","last_peered":"2024-03-28T18:42:58.988151+0000","last_clean":"2024-03-28T18:42:58.988151+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:42:58.988151+0000","last_undegraded":"2024-03-28T18:42:58.988151+0000","last_fullsized":"2024-03-28T18:42:58.988151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987731+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:42:58.987731+0000","last_peered":"2024-03-28T18:42:58.987731+0000","last_clean":"2024-03-28T18:42:58.987731+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:42:58.987731+0000","last_undegraded":"2024-03-28T18:42:58.987731+0000","last_fullsized":"2024-03-28T18:42:58.987731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964648+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:42:58.964648+0000","last_peered":"2024-03-28T18:42:58.964648+0000","last_clean":"2024-03-28T18:42:58.964648+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:42:58.964648+0000","last_undegraded":"2024-03-28T18:42:58.964648+0000","last_fullsized":"2024-03-28T18:42:58.964648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.973904+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:42:58.973904+0000","last_peered":"2024-03-28T18:42:58.973904+0000","last_clean":"2024-03-28T18:42:58.973904+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:42:58.973904+0000","last_undegraded":"2024-03-28T18:42:58.973904+0000","last_fullsized":"2024-03-28T18:42:58.973904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978449+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:42:58.978449+0000","last_peered":"2024-03-28T18:42:58.978449+0000","last_clean":"2024-03-28T18:42:58.978449+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:42:58.978449+0000","last_undegraded":"2024-03-28T18:42:58.978449+0000","last_fullsized":"2024-03-28T18:42:58.978449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983496+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:42:58.983496+0000","last_peered":"2024-03-28T18:42:58.983496+0000","last_clean":"2024-03-28T18:42:58.983496+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:42:58.983496+0000","last_undegraded":"2024-03-28T18:42:58.983496+0000","last_fullsized":"2024-03-28T18:42:58.983496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979669+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:42:58.979669+0000","last_peered":"2024-03-28T18:42:58.979669+0000","last_clean":"2024-03-28T18:42:58.979669+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:42:58.979669+0000","last_undegraded":"2024-03-28T18:42:58.979669+0000","last_fullsized":"2024-03-28T18:42:58.979669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972427+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:42:58.972427+0000","last_peered":"2024-03-28T18:42:58.972427+0000","last_clean":"2024-03-28T18:42:58.972427+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:42:58.972427+0000","last_undegraded":"2024-03-28T18:42:58.972427+0000","last_fullsized":"2024-03-28T18:42:58.972427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.961476+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:42:58.961476+0000","last_peered":"2024-03-28T18:42:58.961476+0000","last_clean":"2024-03-28T18:42:58.961476+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:42:58.961476+0000","last_undegraded":"2024-03-28T18:42:58.961476+0000","last_fullsized":"2024-03-28T18:42:58.961476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978753+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:42:58.978753+0000","last_peered":"2024-03-28T18:42:58.978753+0000","last_clean":"2024-03-28T18:42:58.978753+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:42:58.978753+0000","last_undegraded":"2024-03-28T18:42:58.978753+0000","last_fullsized":"2024-03-28T18:42:58.978753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957622+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:42:58.957622+0000","last_peered":"2024-03-28T18:42:58.957622+0000","last_clean":"2024-03-28T18:42:58.957622+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:42:58.957622+0000","last_undegraded":"2024-03-28T18:42:58.957622+0000","last_fullsized":"2024-03-28T18:42:58.957622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972016+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:42:58.972016+0000","last_peered":"2024-03-28T18:42:58.972016+0000","last_clean":"2024-03-28T18:42:58.972016+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:42:58.972016+0000","last_undegraded":"2024-03-28T18:42:58.972016+0000","last_fullsized":"2024-03-28T18:42:58.972016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987551+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:42:58.987551+0000","last_peered":"2024-03-28T18:42:58.987551+0000","last_clean":"2024-03-28T18:42:58.987551+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:42:58.987551+0000","last_undegraded":"2024-03-28T18:42:58.987551+0000","last_fullsized":"2024-03-28T18:42:58.987551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987723+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:42:58.987723+0000","last_peered":"2024-03-28T18:42:58.987723+0000","last_clean":"2024-03-28T18:42:58.987723+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:42:58.987723+0000","last_undegraded":"2024-03-28T18:42:58.987723+0000","last_fullsized":"2024-03-28T18:42:58.987723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989419+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:42:58.989419+0000","last_peered":"2024-03-28T18:42:58.989419+0000","last_clean":"2024-03-28T18:42:58.989419+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:42:58.989419+0000","last_undegraded":"2024-03-28T18:42:58.989419+0000","last_fullsized":"2024-03-28T18:42:58.989419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986304+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:42:58.986304+0000","last_peered":"2024-03-28T18:42:58.986304+0000","last_clean":"2024-03-28T18:42:58.986304+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:42:58.986304+0000","last_undegraded":"2024-03-28T18:42:58.986304+0000","last_fullsized":"2024-03-28T18:42:58.986304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980073+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:42:58.980073+0000","last_peered":"2024-03-28T18:42:58.980073+0000","last_clean":"2024-03-28T18:42:58.980073+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:42:58.980073+0000","last_undegraded":"2024-03-28T18:42:58.980073+0000","last_fullsized":"2024-03-28T18:42:58.980073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983143+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:42:58.983143+0000","last_peered":"2024-03-28T18:42:58.983143+0000","last_clean":"2024-03-28T18:42:58.983143+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:42:58.983143+0000","last_undegraded":"2024-03-28T18:42:58.983143+0000","last_fullsized":"2024-03-28T18:42:58.983143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.509497+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:43:43.509497+0000","last_peered":"2024-03-28T18:43:43.509497+0000","last_clean":"2024-03-28T18:43:43.509497+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:43:43.509497+0000","last_undegraded":"2024-03-28T18:43:43.509497+0000","last_fullsized":"2024-03-28T18:43:43.509497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976163+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:42:58.976163+0000","last_peered":"2024-03-28T18:42:58.976163+0000","last_clean":"2024-03-28T18:42:58.976163+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:42:58.976163+0000","last_undegraded":"2024-03-28T18:42:58.976163+0000","last_fullsized":"2024-03-28T18:42:58.976163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.405108+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:42:31.405108+0000","last_peered":"2024-03-28T18:42:31.405108+0000","last_clean":"2024-03-28T18:42:31.405108+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:42:31.405108+0000","last_undegraded":"2024-03-28T18:42:31.405108+0000","last_fullsized":"2024-03-28T18:42:31.405108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987959+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:42:58.987959+0000","last_peered":"2024-03-28T18:42:58.987959+0000","last_clean":"2024-03-28T18:42:58.987959+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:42:58.987959+0000","last_undegraded":"2024-03-28T18:42:58.987959+0000","last_fullsized":"2024-03-28T18:42:58.987959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239229+0000","last_change":"2024-03-28T18:41:30.335571+0000","last_active":"2024-03-28T18:42:31.239229+0000","last_peered":"2024-03-28T18:42:31.239229+0000","last_clean":"2024-03-28T18:42:31.239229+0000","last_became_active":"2024-03-28T18:41:30.311420+0000","last_became_peered":"2024-03-28T18:41:30.311420+0000","last_unstale":"2024-03-28T18:42:31.239229+0000","last_undegraded":"2024-03-28T18:42:31.239229+0000","last_fullsized":"2024-03-28T18:42:31.239229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:39.796667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972421+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:42:58.972421+0000","last_peered":"2024-03-28T18:42:58.972421+0000","last_clean":"2024-03-28T18:42:58.972421+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:42:58.972421+0000","last_undegraded":"2024-03-28T18:42:58.972421+0000","last_fullsized":"2024-03-28T18:42:58.972421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239193+0000","last_change":"2024-03-28T18:41:30.315981+0000","last_active":"2024-03-28T18:42:31.239193+0000","last_peered":"2024-03-28T18:42:31.239193+0000","last_clean":"2024-03-28T18:42:31.239193+0000","last_became_active":"2024-03-28T18:41:30.311208+0000","last_became_peered":"2024-03-28T18:41:30.311208+0000","last_unstale":"2024-03-28T18:42:31.239193+0000","last_undegraded":"2024-03-28T18:42:31.239193+0000","last_fullsized":"2024-03-28T18:42:31.239193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:10.817446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.962262+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:42:58.962262+0000","last_peered":"2024-03-28T18:42:58.962262+0000","last_clean":"2024-03-28T18:42:58.962262+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:42:58.962262+0000","last_undegraded":"2024-03-28T18:42:58.962262+0000","last_fullsized":"2024-03-28T18:42:58.962262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:46.346077+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:43:46.346077+0000","last_peered":"2024-03-28T18:43:46.346077+0000","last_clean":"2024-03-28T18:43:46.346077+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:43:46.346077+0000","last_undegraded":"2024-03-28T18:43:46.346077+0000","last_fullsized":"2024-03-28T18:43:46.346077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.969774+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:42:58.969774+0000","last_peered":"2024-03-28T18:42:58.969774+0000","last_clean":"2024-03-28T18:42:58.969774+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:42:58.969774+0000","last_undegraded":"2024-03-28T18:42:58.969774+0000","last_fullsized":"2024-03-28T18:42:58.969774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269565+0000","last_change":"2024-03-28T18:41:30.228521+0000","last_active":"2024-03-28T18:42:31.269565+0000","last_peered":"2024-03-28T18:42:31.269565+0000","last_clean":"2024-03-28T18:42:31.269565+0000","last_became_active":"2024-03-28T18:41:30.221867+0000","last_became_peered":"2024-03-28T18:41:30.221867+0000","last_unstale":"2024-03-28T18:42:31.269565+0000","last_undegraded":"2024-03-28T18:42:31.269565+0000","last_fullsized":"2024-03-28T18:42:31.269565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:45.207508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968706+0000","last_change":"2024-03-28T18:41:30.507663+0000","last_active":"2024-03-28T18:42:58.968706+0000","last_peered":"2024-03-28T18:42:58.968706+0000","last_clean":"2024-03-28T18:42:58.968706+0000","last_became_active":"2024-03-28T18:41:30.507284+0000","last_became_peered":"2024-03-28T18:41:30.507284+0000","last_unstale":"2024-03-28T18:42:58.968706+0000","last_undegraded":"2024-03-28T18:42:58.968706+0000","last_fullsized":"2024-03-28T18:42:58.968706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:03.045818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175185+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:42:31.175185+0000","last_peered":"2024-03-28T18:42:31.175185+0000","last_clean":"2024-03-28T18:42:31.175185+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:42:31.175185+0000","last_undegraded":"2024-03-28T18:42:31.175185+0000","last_fullsized":"2024-03-28T18:42:31.175185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982042+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:42:58.982042+0000","last_peered":"2024-03-28T18:42:58.982042+0000","last_clean":"2024-03-28T18:42:58.982042+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:42:58.982042+0000","last_undegraded":"2024-03-28T18:42:58.982042+0000","last_fullsized":"2024-03-28T18:42:58.982042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168096+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:42:31.168096+0000","last_peered":"2024-03-28T18:42:31.168096+0000","last_clean":"2024-03-28T18:42:31.168096+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:42:31.168096+0000","last_undegraded":"2024-03-28T18:42:31.168096+0000","last_fullsized":"2024-03-28T18:42:31.168096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980414+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:42:58.980414+0000","last_peered":"2024-03-28T18:42:58.980414+0000","last_clean":"2024-03-28T18:42:58.980414+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:42:58.980414+0000","last_undegraded":"2024-03-28T18:42:58.980414+0000","last_fullsized":"2024-03-28T18:42:58.980414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.439337+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:43:47.439337+0000","last_peered":"2024-03-28T18:43:47.439337+0000","last_clean":"2024-03-28T18:43:47.439337+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:43:47.439337+0000","last_undegraded":"2024-03-28T18:43:47.439337+0000","last_fullsized":"2024-03-28T18:43:47.439337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":280,"num_read_kb":280,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982861+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:42:58.982861+0000","last_peered":"2024-03-28T18:42:58.982861+0000","last_clean":"2024-03-28T18:42:58.982861+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:42:58.982861+0000","last_undegraded":"2024-03-28T18:42:58.982861+0000","last_fullsized":"2024-03-28T18:42:58.982861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360380+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:42:31.360380+0000","last_peered":"2024-03-28T18:42:31.360380+0000","last_clean":"2024-03-28T18:42:31.360380+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:42:31.360380+0000","last_undegraded":"2024-03-28T18:42:31.360380+0000","last_fullsized":"2024-03-28T18:42:31.360380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987661+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:42:58.987661+0000","last_peered":"2024-03-28T18:42:58.987661+0000","last_clean":"2024-03-28T18:42:58.987661+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:42:58.987661+0000","last_undegraded":"2024-03-28T18:42:58.987661+0000","last_fullsized":"2024-03-28T18:42:58.987661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168678+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:42:31.168678+0000","last_peered":"2024-03-28T18:42:31.168678+0000","last_clean":"2024-03-28T18:42:31.168678+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:42:31.168678+0000","last_undegraded":"2024-03-28T18:42:31.168678+0000","last_fullsized":"2024-03-28T18:42:31.168678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978619+0000","last_change":"2024-03-28T18:42:31.303127+0000","last_active":"2024-03-28T18:42:58.978619+0000","last_peered":"2024-03-28T18:42:58.978619+0000","last_clean":"2024-03-28T18:42:58.978619+0000","last_became_active":"2024-03-28T18:42:31.301321+0000","last_became_peered":"2024-03-28T18:42:31.301321+0000","last_unstale":"2024-03-28T18:42:58.978619+0000","last_undegraded":"2024-03-28T18:42:58.978619+0000","last_fullsized":"2024-03-28T18:42:58.978619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:08.892803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":351,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.346068+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:43:47.346068+0000","last_peered":"2024-03-28T18:43:47.346068+0000","last_clean":"2024-03-28T18:43:47.346068+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:43:47.346068+0000","last_undegraded":"2024-03-28T18:43:47.346068+0000","last_fullsized":"2024-03-28T18:43:47.346068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.948299+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:42:58.948299+0000","last_peered":"2024-03-28T18:42:58.948299+0000","last_clean":"2024-03-28T18:42:58.948299+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:42:58.948299+0000","last_undegraded":"2024-03-28T18:42:58.948299+0000","last_fullsized":"2024-03-28T18:42:58.948299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317615+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:42:31.317615+0000","last_peered":"2024-03-28T18:42:31.317615+0000","last_clean":"2024-03-28T18:42:31.317615+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:42:31.317615+0000","last_undegraded":"2024-03-28T18:42:31.317615+0000","last_fullsized":"2024-03-28T18:42:31.317615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983586+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:42:58.983586+0000","last_peered":"2024-03-28T18:42:58.983586+0000","last_clean":"2024-03-28T18:42:58.983586+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:42:58.983586+0000","last_undegraded":"2024-03-28T18:42:58.983586+0000","last_fullsized":"2024-03-28T18:42:58.983586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168283+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:42:31.168283+0000","last_peered":"2024-03-28T18:42:31.168283+0000","last_clean":"2024-03-28T18:42:31.168283+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:42:31.168283+0000","last_undegraded":"2024-03-28T18:42:31.168283+0000","last_fullsized":"2024-03-28T18:42:31.168283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982443+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:42:58.982443+0000","last_peered":"2024-03-28T18:42:58.982443+0000","last_clean":"2024-03-28T18:42:58.982443+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:42:58.982443+0000","last_undegraded":"2024-03-28T18:42:58.982443+0000","last_fullsized":"2024-03-28T18:42:58.982443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.419661+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:43:43.419661+0000","last_peered":"2024-03-28T18:43:43.419661+0000","last_clean":"2024-03-28T18:43:43.419661+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:43:43.419661+0000","last_undegraded":"2024-03-28T18:43:43.419661+0000","last_fullsized":"2024-03-28T18:43:43.419661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970949+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:42:58.970949+0000","last_peered":"2024-03-28T18:42:58.970949+0000","last_clean":"2024-03-28T18:42:58.970949+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:42:58.970949+0000","last_undegraded":"2024-03-28T18:42:58.970949+0000","last_fullsized":"2024-03-28T18:42:58.970949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238896+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:42:31.238896+0000","last_peered":"2024-03-28T18:42:31.238896+0000","last_clean":"2024-03-28T18:42:31.238896+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:42:31.238896+0000","last_undegraded":"2024-03-28T18:42:31.238896+0000","last_fullsized":"2024-03-28T18:42:31.238896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957235+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:42:58.957235+0000","last_peered":"2024-03-28T18:42:58.957235+0000","last_clean":"2024-03-28T18:42:58.957235+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:42:58.957235+0000","last_undegraded":"2024-03-28T18:42:58.957235+0000","last_fullsized":"2024-03-28T18:42:58.957235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239439+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:42:31.239439+0000","last_peered":"2024-03-28T18:42:31.239439+0000","last_clean":"2024-03-28T18:42:31.239439+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:42:31.239439+0000","last_undegraded":"2024-03-28T18:42:31.239439+0000","last_fullsized":"2024-03-28T18:42:31.239439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971622+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:42:58.971622+0000","last_peered":"2024-03-28T18:42:58.971622+0000","last_clean":"2024-03-28T18:42:58.971622+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:42:58.971622+0000","last_undegraded":"2024-03-28T18:42:58.971622+0000","last_fullsized":"2024-03-28T18:42:58.971622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.173997+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:42:31.173997+0000","last_peered":"2024-03-28T18:42:31.173997+0000","last_clean":"2024-03-28T18:42:31.173997+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:42:31.173997+0000","last_undegraded":"2024-03-28T18:42:31.173997+0000","last_fullsized":"2024-03-28T18:42:31.173997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976756+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:42:58.976756+0000","last_peered":"2024-03-28T18:42:58.976756+0000","last_clean":"2024-03-28T18:42:58.976756+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:42:58.976756+0000","last_undegraded":"2024-03-28T18:42:58.976756+0000","last_fullsized":"2024-03-28T18:42:58.976756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":169,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.514348+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:43:45.514348+0000","last_peered":"2024-03-28T18:43:45.514348+0000","last_clean":"2024-03-28T18:43:45.514348+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:43:45.514348+0000","last_undegraded":"2024-03-28T18:43:45.514348+0000","last_fullsized":"2024-03-28T18:43:45.514348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988444+0000","last_change":"2024-03-28T18:41:30.219905+0000","last_active":"2024-03-28T18:42:58.988444+0000","last_peered":"2024-03-28T18:42:58.988444+0000","last_clean":"2024-03-28T18:42:58.988444+0000","last_became_active":"2024-03-28T18:41:30.219488+0000","last_became_peered":"2024-03-28T18:41:30.219488+0000","last_unstale":"2024-03-28T18:42:58.988444+0000","last_undegraded":"2024-03-28T18:42:58.988444+0000","last_fullsized":"2024-03-28T18:42:58.988444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:16.108678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975656+0000","last_change":"2024-03-28T18:42:31.320096+0000","last_active":"2024-03-28T18:42:58.975656+0000","last_peered":"2024-03-28T18:42:58.975656+0000","last_clean":"2024-03-28T18:42:58.975656+0000","last_became_active":"2024-03-28T18:42:31.318988+0000","last_became_peered":"2024-03-28T18:42:31.318988+0000","last_unstale":"2024-03-28T18:42:58.975656+0000","last_undegraded":"2024-03-28T18:42:58.975656+0000","last_fullsized":"2024-03-28T18:42:58.975656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:50.019195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.422734+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:43:45.422734+0000","last_peered":"2024-03-28T18:43:45.422734+0000","last_clean":"2024-03-28T18:43:45.422734+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:43:45.422734+0000","last_undegraded":"2024-03-28T18:43:45.422734+0000","last_fullsized":"2024-03-28T18:43:45.422734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975540+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:42:58.975540+0000","last_peered":"2024-03-28T18:42:58.975540+0000","last_clean":"2024-03-28T18:42:58.975540+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:42:58.975540+0000","last_undegraded":"2024-03-28T18:42:58.975540+0000","last_fullsized":"2024-03-28T18:42:58.975540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299771+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:42:31.299771+0000","last_peered":"2024-03-28T18:42:31.299771+0000","last_clean":"2024-03-28T18:42:31.299771+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:42:31.299771+0000","last_undegraded":"2024-03-28T18:42:31.299771+0000","last_fullsized":"2024-03-28T18:42:31.299771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360313+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:42:31.360313+0000","last_peered":"2024-03-28T18:42:31.360313+0000","last_clean":"2024-03-28T18:42:31.360313+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:42:31.360313+0000","last_undegraded":"2024-03-28T18:42:31.360313+0000","last_fullsized":"2024-03-28T18:42:31.360313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964665+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:42:58.964665+0000","last_peered":"2024-03-28T18:42:58.964665+0000","last_clean":"2024-03-28T18:42:58.964665+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:42:58.964665+0000","last_undegraded":"2024-03-28T18:42:58.964665+0000","last_fullsized":"2024-03-28T18:42:58.964665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360379+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:42:31.360379+0000","last_peered":"2024-03-28T18:42:31.360379+0000","last_clean":"2024-03-28T18:42:31.360379+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:42:31.360379+0000","last_undegraded":"2024-03-28T18:42:31.360379+0000","last_fullsized":"2024-03-28T18:42:31.360379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411326+0000","last_change":"2024-03-28T18:41:30.227916+0000","last_active":"2024-03-28T18:42:31.411326+0000","last_peered":"2024-03-28T18:42:31.411326+0000","last_clean":"2024-03-28T18:42:31.411326+0000","last_became_active":"2024-03-28T18:41:30.218947+0000","last_became_peered":"2024-03-28T18:41:30.218947+0000","last_unstale":"2024-03-28T18:42:31.411326+0000","last_undegraded":"2024-03-28T18:42:31.411326+0000","last_fullsized":"2024-03-28T18:42:31.411326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:14.922324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983366+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:42:58.983366+0000","last_peered":"2024-03-28T18:42:58.983366+0000","last_clean":"2024-03-28T18:42:58.983366+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:42:58.983366+0000","last_undegraded":"2024-03-28T18:42:58.983366+0000","last_fullsized":"2024-03-28T18:42:58.983366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:30.238634+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:43:30.238634+0000","last_peered":"2024-03-28T18:43:30.238634+0000","last_clean":"2024-03-28T18:43:30.238634+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:43:30.238634+0000","last_undegraded":"2024-03-28T18:43:30.238634+0000","last_fullsized":"2024-03-28T18:43:30.238634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983095+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:42:58.983095+0000","last_peered":"2024-03-28T18:42:58.983095+0000","last_clean":"2024-03-28T18:42:58.983095+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:42:58.983095+0000","last_undegraded":"2024-03-28T18:42:58.983095+0000","last_fullsized":"2024-03-28T18:42:58.983095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299859+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:42:31.299859+0000","last_peered":"2024-03-28T18:42:31.299859+0000","last_clean":"2024-03-28T18:42:31.299859+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:42:31.299859+0000","last_undegraded":"2024-03-28T18:42:31.299859+0000","last_fullsized":"2024-03-28T18:42:31.299859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988228+0000","last_change":"2024-03-28T18:42:31.302339+0000","last_active":"2024-03-28T18:42:58.988228+0000","last_peered":"2024-03-28T18:42:58.988228+0000","last_clean":"2024-03-28T18:42:58.988228+0000","last_became_active":"2024-03-28T18:42:31.301324+0000","last_became_peered":"2024-03-28T18:42:31.301324+0000","last_unstale":"2024-03-28T18:42:58.988228+0000","last_undegraded":"2024-03-28T18:42:58.988228+0000","last_fullsized":"2024-03-28T18:42:58.988228+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:56.490969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167540+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:42:31.167540+0000","last_peered":"2024-03-28T18:42:31.167540+0000","last_clean":"2024-03-28T18:42:31.167540+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:42:31.167540+0000","last_undegraded":"2024-03-28T18:42:31.167540+0000","last_fullsized":"2024-03-28T18:42:31.167540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.941667+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:42:58.941667+0000","last_peered":"2024-03-28T18:42:58.941667+0000","last_clean":"2024-03-28T18:42:58.941667+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:42:58.941667+0000","last_undegraded":"2024-03-28T18:42:58.941667+0000","last_fullsized":"2024-03-28T18:42:58.941667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403649+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:42:31.403649+0000","last_peered":"2024-03-28T18:42:31.403649+0000","last_clean":"2024-03-28T18:42:31.403649+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:42:31.403649+0000","last_undegraded":"2024-03-28T18:42:31.403649+0000","last_fullsized":"2024-03-28T18:42:31.403649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987979+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:42:58.987979+0000","last_peered":"2024-03-28T18:42:58.987979+0000","last_clean":"2024-03-28T18:42:58.987979+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:42:58.987979+0000","last_undegraded":"2024-03-28T18:42:58.987979+0000","last_fullsized":"2024-03-28T18:42:58.987979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317603+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:42:31.317603+0000","last_peered":"2024-03-28T18:42:31.317603+0000","last_clean":"2024-03-28T18:42:31.317603+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:42:31.317603+0000","last_undegraded":"2024-03-28T18:42:31.317603+0000","last_fullsized":"2024-03-28T18:42:31.317603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989314+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:42:58.989314+0000","last_peered":"2024-03-28T18:42:58.989314+0000","last_clean":"2024-03-28T18:42:58.989314+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:42:58.989314+0000","last_undegraded":"2024-03-28T18:42:58.989314+0000","last_fullsized":"2024-03-28T18:42:58.989314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239033+0000","last_change":"2024-03-28T18:41:30.319270+0000","last_active":"2024-03-28T18:42:31.239033+0000","last_peered":"2024-03-28T18:42:31.239033+0000","last_clean":"2024-03-28T18:42:31.239033+0000","last_became_active":"2024-03-28T18:41:30.311219+0000","last_became_peered":"2024-03-28T18:41:30.311219+0000","last_unstale":"2024-03-28T18:42:31.239033+0000","last_undegraded":"2024-03-28T18:42:31.239033+0000","last_fullsized":"2024-03-28T18:42:31.239033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:23.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982021+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:42:58.982021+0000","last_peered":"2024-03-28T18:42:58.982021+0000","last_clean":"2024-03-28T18:42:58.982021+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:42:58.982021+0000","last_undegraded":"2024-03-28T18:42:58.982021+0000","last_fullsized":"2024-03-28T18:42:58.982021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.436770+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:43:45.436770+0000","last_peered":"2024-03-28T18:43:45.436770+0000","last_clean":"2024-03-28T18:43:45.436770+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:43:45.436770+0000","last_undegraded":"2024-03-28T18:43:45.436770+0000","last_fullsized":"2024-03-28T18:43:45.436770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979050+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:42:58.979050+0000","last_peered":"2024-03-28T18:42:58.979050+0000","last_clean":"2024-03-28T18:42:58.979050+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:42:58.979050+0000","last_undegraded":"2024-03-28T18:42:58.979050+0000","last_fullsized":"2024-03-28T18:42:58.979050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168484+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:42:31.168484+0000","last_peered":"2024-03-28T18:42:31.168484+0000","last_clean":"2024-03-28T18:42:31.168484+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:42:31.168484+0000","last_undegraded":"2024-03-28T18:42:31.168484+0000","last_fullsized":"2024-03-28T18:42:31.168484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.984053+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:42:58.984053+0000","last_peered":"2024-03-28T18:42:58.984053+0000","last_clean":"2024-03-28T18:42:58.984053+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:42:58.984053+0000","last_undegraded":"2024-03-28T18:42:58.984053+0000","last_fullsized":"2024-03-28T18:42:58.984053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.176338+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:42:31.176338+0000","last_peered":"2024-03-28T18:42:31.176338+0000","last_clean":"2024-03-28T18:42:31.176338+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:42:31.176338+0000","last_undegraded":"2024-03-28T18:42:31.176338+0000","last_fullsized":"2024-03-28T18:42:31.176338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978116+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:42:58.978116+0000","last_peered":"2024-03-28T18:42:58.978116+0000","last_clean":"2024-03-28T18:42:58.978116+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:42:58.978116+0000","last_undegraded":"2024-03-28T18:42:58.978116+0000","last_fullsized":"2024-03-28T18:42:58.978116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972976+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:42:58.972976+0000","last_peered":"2024-03-28T18:42:58.972976+0000","last_clean":"2024-03-28T18:42:58.972976+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:42:58.972976+0000","last_undegraded":"2024-03-28T18:42:58.972976+0000","last_fullsized":"2024-03-28T18:42:58.972976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.967314+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:42:58.967314+0000","last_peered":"2024-03-28T18:42:58.967314+0000","last_clean":"2024-03-28T18:42:58.967314+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:42:58.967314+0000","last_undegraded":"2024-03-28T18:42:58.967314+0000","last_fullsized":"2024-03-28T18:42:58.967314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987481+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:42:58.987481+0000","last_peered":"2024-03-28T18:42:58.987481+0000","last_clean":"2024-03-28T18:42:58.987481+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:42:58.987481+0000","last_undegraded":"2024-03-28T18:42:58.987481+0000","last_fullsized":"2024-03-28T18:42:58.987481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988446+0000","last_change":"2024-03-28T18:42:31.362520+0000","last_active":"2024-03-28T18:42:58.988446+0000","last_peered":"2024-03-28T18:42:58.988446+0000","last_clean":"2024-03-28T18:42:58.988446+0000","last_became_active":"2024-03-28T18:42:31.361563+0000","last_became_peered":"2024-03-28T18:42:31.361563+0000","last_unstale":"2024-03-28T18:42:58.988446+0000","last_undegraded":"2024-03-28T18:42:58.988446+0000","last_fullsized":"2024-03-28T18:42:58.988446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:27.960042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988524+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:42:58.988524+0000","last_peered":"2024-03-28T18:42:58.988524+0000","last_clean":"2024-03-28T18:42:58.988524+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:42:58.988524+0000","last_undegraded":"2024-03-28T18:42:58.988524+0000","last_fullsized":"2024-03-28T18:42:58.988524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971656+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:42:58.971656+0000","last_peered":"2024-03-28T18:42:58.971656+0000","last_clean":"2024-03-28T18:42:58.971656+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:42:58.971656+0000","last_undegraded":"2024-03-28T18:42:58.971656+0000","last_fullsized":"2024-03-28T18:42:58.971656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970936+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:42:58.970936+0000","last_peered":"2024-03-28T18:42:58.970936+0000","last_clean":"2024-03-28T18:42:58.970936+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:42:58.970936+0000","last_undegraded":"2024-03-28T18:42:58.970936+0000","last_fullsized":"2024-03-28T18:42:58.970936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.977119+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:42:58.977119+0000","last_peered":"2024-03-28T18:42:58.977119+0000","last_clean":"2024-03-28T18:42:58.977119+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:42:58.977119+0000","last_undegraded":"2024-03-28T18:42:58.977119+0000","last_fullsized":"2024-03-28T18:42:58.977119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.981154+0000","last_change":"2024-03-28T18:41:30.510267+0000","last_active":"2024-03-28T18:42:58.981154+0000","last_peered":"2024-03-28T18:42:58.981154+0000","last_clean":"2024-03-28T18:42:58.981154+0000","last_became_active":"2024-03-28T18:41:30.508658+0000","last_became_peered":"2024-03-28T18:41:30.508658+0000","last_unstale":"2024-03-28T18:42:58.981154+0000","last_undegraded":"2024-03-28T18:42:58.981154+0000","last_fullsized":"2024-03-28T18:42:58.981154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:30.892433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982988+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:42:58.982988+0000","last_peered":"2024-03-28T18:42:58.982988+0000","last_clean":"2024-03-28T18:42:58.982988+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:42:58.982988+0000","last_undegraded":"2024-03-28T18:42:58.982988+0000","last_fullsized":"2024-03-28T18:42:58.982988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.974718+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:42:58.974718+0000","last_peered":"2024-03-28T18:42:58.974718+0000","last_clean":"2024-03-28T18:42:58.974718+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:42:58.974718+0000","last_undegraded":"2024-03-28T18:42:58.974718+0000","last_fullsized":"2024-03-28T18:42:58.974718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988976+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:42:58.988976+0000","last_peered":"2024-03-28T18:42:58.988976+0000","last_clean":"2024-03-28T18:42:58.988976+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:42:58.988976+0000","last_undegraded":"2024-03-28T18:42:58.988976+0000","last_fullsized":"2024-03-28T18:42:58.988976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968063+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:42:58.968063+0000","last_peered":"2024-03-28T18:42:58.968063+0000","last_clean":"2024-03-28T18:42:58.968063+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:42:58.968063+0000","last_undegraded":"2024-03-28T18:42:58.968063+0000","last_fullsized":"2024-03-28T18:42:58.968063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70048,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1429,"num_read_kb":790,"num_write":255,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":215517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477349,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1131054,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477167,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037820,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477433,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071042,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":476884,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":540053,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":478223,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":541111,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":476709,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1069606,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:43:50.215 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-28T18:43:50.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.450+0000 7f958ba00640 1 -- 172.21.15.119:0/3408077138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841016d0 msgr2=0x7f9584101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.450+0000 7f958ba00640 1 --2- 172.21.15.119:0/3408077138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841016d0 0x7f9584101af0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9578009900 tx=0x7f957802bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 -- 172.21.15.119:0/3408077138 shutdown_connections 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 --2- 172.21.15.119:0/3408077138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 --2- 172.21.15.119:0/3408077138 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841028c0 0x7f9584102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 --2- 172.21.15.119:0/3408077138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841016d0 0x7f9584101af0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 -- 172.21.15.119:0/3408077138 >> 172.21.15.119:0/3408077138 conn(0x7f95840fcc40 msgr2=0x7f95840ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 -- 172.21.15.119:0/3408077138 shutdown_connections 2024-03-28T18:43:50.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.451+0000 7f958ba00640 1 -- 172.21.15.119:0/3408077138 wait complete. 2024-03-28T18:43:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 Processor -- start 2024-03-28T18:43:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 -- start start 2024-03-28T18:43:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 0x7f9584075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 0x7f9584073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958410d3a0 con 0x7f95841028c0 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958410d220 con 0x7f95841016d0 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f958ba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958410d520 con 0x7f9584103280 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f9589a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f9589000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 0x7f9584075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f9589a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35986/0 (socket says 172.21.15.119:35986) 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f9589a00640 1 -- 172.21.15.119:0/1839862751 learned_addr learned my addr 172.21.15.119:0/1839862751 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.452+0000 7f9583e00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 0x7f9584073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 -- 172.21.15.119:0/1839862751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 msgr2=0x7f9584075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 0x7f9584075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 -- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 msgr2=0x7f9584073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 0x7f9584073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 -- 172.21.15.119:0/1839862751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9584074860 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9583e00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 0x7f9584073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589000640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 0x7f9584075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9589a00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f957c00eb10 tx=0x7f957c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c004b20 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95841a1b30 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f957c004cc0 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c010760 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.453+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95841a2080 con 0x7f9584103280 2024-03-28T18:43:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.455+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95841089f0 con 0x7f9584103280 2024-03-28T18:43:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.457+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f957c0459b0 con 0x7f9584103280 2024-03-28T18:43:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.458+0000 7f9581600640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 0x7f956c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.458+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f957c0cf2d0 con 0x7f9584103280 2024-03-28T18:43:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.458+0000 7f9589000640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 0x7f956c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.459+0000 7f9589000640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 0x7f956c0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f957802c180 tx=0x7f957802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.460+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f957c09c290 con 0x7f9584103280 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2636552283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:50 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.657+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9584061ae0 con 0x7f956c0a2800 2024-03-28T18:43:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T18:43:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2636552283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:50 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.687+0000 7f9581600640 1 -- 172.21.15.119:0/1839862751 <== mgr.14152 v2:172.21.15.119:6800/2493078203 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259126 (secure 0 0 0) 0x7f9584061ae0 con 0x7f956c0a2800 2024-03-28T18:43:50.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:50.694 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 msgr2=0x7f956c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 0x7f956c0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f957802c180 tx=0x7f957802cf90 comp rx=0 tx=0).stop 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 msgr2=0x7f9584073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f957c00eb10 tx=0x7f957c00c2d0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 shutdown_connections 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f956c0a2800 0x7f956c0a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9584103280 0x7f9584073fd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95841028c0 0x7f9584073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 --2- 172.21.15.119:0/1839862751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95841016d0 0x7f9584075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 >> 172.21.15.119:0/1839862751 conn(0x7f95840fcc40 msgr2=0x7f9584105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:50.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 shutdown_connections 2024-03-28T18:43:50.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.692+0000 7f958ba00640 1 -- 172.21.15.119:0/1839862751 wait complete. 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2636552283' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:50 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:50.713 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-03-28T18:43:49.022675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660650,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1565,"num_read_kb":909,"num_write":438,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361260,"kb_used_data":12108,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580564,"statfs":{"total":1151940427776,"available":1151570497536,"internally_reserved":0,"allocated":12398592,"data_stored":7694490,"data_compressed":246444,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":612,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":72,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004738"},"pg_stats":[{"pgid":"3.3f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986177+0000","last_change":"2024-03-28T18:41:21.114318+0000","last_active":"2024-03-28T18:42:58.986177+0000","last_peered":"2024-03-28T18:42:58.986177+0000","last_clean":"2024-03-28T18:42:58.986177+0000","last_became_active":"2024-03-28T18:41:21.113883+0000","last_became_peered":"2024-03-28T18:41:21.113883+0000","last_unstale":"2024-03-28T18:42:58.986177+0000","last_undegraded":"2024-03-28T18:42:58.986177+0000","last_fullsized":"2024-03-28T18:42:58.986177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:14:51.255635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988151+0000","last_change":"2024-03-28T18:41:21.118616+0000","last_active":"2024-03-28T18:42:58.988151+0000","last_peered":"2024-03-28T18:42:58.988151+0000","last_clean":"2024-03-28T18:42:58.988151+0000","last_became_active":"2024-03-28T18:41:21.117948+0000","last_became_peered":"2024-03-28T18:41:21.117948+0000","last_unstale":"2024-03-28T18:42:58.988151+0000","last_undegraded":"2024-03-28T18:42:58.988151+0000","last_fullsized":"2024-03-28T18:42:58.988151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:08.921544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987731+0000","last_change":"2024-03-28T18:41:21.115958+0000","last_active":"2024-03-28T18:42:58.987731+0000","last_peered":"2024-03-28T18:42:58.987731+0000","last_clean":"2024-03-28T18:42:58.987731+0000","last_became_active":"2024-03-28T18:41:21.112971+0000","last_became_peered":"2024-03-28T18:41:21.112971+0000","last_unstale":"2024-03-28T18:42:58.987731+0000","last_undegraded":"2024-03-28T18:42:58.987731+0000","last_fullsized":"2024-03-28T18:42:58.987731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:46:59.931913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964648+0000","last_change":"2024-03-28T18:41:21.094778+0000","last_active":"2024-03-28T18:42:58.964648+0000","last_peered":"2024-03-28T18:42:58.964648+0000","last_clean":"2024-03-28T18:42:58.964648+0000","last_became_active":"2024-03-28T18:41:21.093667+0000","last_became_peered":"2024-03-28T18:41:21.093667+0000","last_unstale":"2024-03-28T18:42:58.964648+0000","last_undegraded":"2024-03-28T18:42:58.964648+0000","last_fullsized":"2024-03-28T18:42:58.964648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:16.895705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.973904+0000","last_change":"2024-03-28T18:41:21.113004+0000","last_active":"2024-03-28T18:42:58.973904+0000","last_peered":"2024-03-28T18:42:58.973904+0000","last_clean":"2024-03-28T18:42:58.973904+0000","last_became_active":"2024-03-28T18:41:21.112584+0000","last_became_peered":"2024-03-28T18:41:21.112584+0000","last_unstale":"2024-03-28T18:42:58.973904+0000","last_undegraded":"2024-03-28T18:42:58.973904+0000","last_fullsized":"2024-03-28T18:42:58.973904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:06:04.708108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978449+0000","last_change":"2024-03-28T18:41:21.103599+0000","last_active":"2024-03-28T18:42:58.978449+0000","last_peered":"2024-03-28T18:42:58.978449+0000","last_clean":"2024-03-28T18:42:58.978449+0000","last_became_active":"2024-03-28T18:41:21.103270+0000","last_became_peered":"2024-03-28T18:41:21.103270+0000","last_unstale":"2024-03-28T18:42:58.978449+0000","last_undegraded":"2024-03-28T18:42:58.978449+0000","last_fullsized":"2024-03-28T18:42:58.978449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:11.165044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983496+0000","last_change":"2024-03-28T18:41:21.102165+0000","last_active":"2024-03-28T18:42:58.983496+0000","last_peered":"2024-03-28T18:42:58.983496+0000","last_clean":"2024-03-28T18:42:58.983496+0000","last_became_active":"2024-03-28T18:41:21.101801+0000","last_became_peered":"2024-03-28T18:41:21.101801+0000","last_unstale":"2024-03-28T18:42:58.983496+0000","last_undegraded":"2024-03-28T18:42:58.983496+0000","last_fullsized":"2024-03-28T18:42:58.983496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:47.409725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979669+0000","last_change":"2024-03-28T18:41:21.117456+0000","last_active":"2024-03-28T18:42:58.979669+0000","last_peered":"2024-03-28T18:42:58.979669+0000","last_clean":"2024-03-28T18:42:58.979669+0000","last_became_active":"2024-03-28T18:41:21.116977+0000","last_became_peered":"2024-03-28T18:41:21.116977+0000","last_unstale":"2024-03-28T18:42:58.979669+0000","last_undegraded":"2024-03-28T18:42:58.979669+0000","last_fullsized":"2024-03-28T18:42:58.979669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:42.213947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972427+0000","last_change":"2024-03-28T18:41:21.091869+0000","last_active":"2024-03-28T18:42:58.972427+0000","last_peered":"2024-03-28T18:42:58.972427+0000","last_clean":"2024-03-28T18:42:58.972427+0000","last_became_active":"2024-03-28T18:41:21.091164+0000","last_became_peered":"2024-03-28T18:41:21.091164+0000","last_unstale":"2024-03-28T18:42:58.972427+0000","last_undegraded":"2024-03-28T18:42:58.972427+0000","last_fullsized":"2024-03-28T18:42:58.972427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:19:18.201709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.961476+0000","last_change":"2024-03-28T18:41:21.097053+0000","last_active":"2024-03-28T18:42:58.961476+0000","last_peered":"2024-03-28T18:42:58.961476+0000","last_clean":"2024-03-28T18:42:58.961476+0000","last_became_active":"2024-03-28T18:41:21.096642+0000","last_became_peered":"2024-03-28T18:41:21.096642+0000","last_unstale":"2024-03-28T18:42:58.961476+0000","last_undegraded":"2024-03-28T18:42:58.961476+0000","last_fullsized":"2024-03-28T18:42:58.961476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:01:01.862814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978753+0000","last_change":"2024-03-28T18:41:21.096769+0000","last_active":"2024-03-28T18:42:58.978753+0000","last_peered":"2024-03-28T18:42:58.978753+0000","last_clean":"2024-03-28T18:42:58.978753+0000","last_became_active":"2024-03-28T18:41:21.096471+0000","last_became_peered":"2024-03-28T18:41:21.096471+0000","last_unstale":"2024-03-28T18:42:58.978753+0000","last_undegraded":"2024-03-28T18:42:58.978753+0000","last_fullsized":"2024-03-28T18:42:58.978753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:41:24.090946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957622+0000","last_change":"2024-03-28T18:41:21.101241+0000","last_active":"2024-03-28T18:42:58.957622+0000","last_peered":"2024-03-28T18:42:58.957622+0000","last_clean":"2024-03-28T18:42:58.957622+0000","last_became_active":"2024-03-28T18:41:21.100821+0000","last_became_peered":"2024-03-28T18:41:21.100821+0000","last_unstale":"2024-03-28T18:42:58.957622+0000","last_undegraded":"2024-03-28T18:42:58.957622+0000","last_fullsized":"2024-03-28T18:42:58.957622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:42:44.672788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972016+0000","last_change":"2024-03-28T18:41:21.098423+0000","last_active":"2024-03-28T18:42:58.972016+0000","last_peered":"2024-03-28T18:42:58.972016+0000","last_clean":"2024-03-28T18:42:58.972016+0000","last_became_active":"2024-03-28T18:41:21.097408+0000","last_became_peered":"2024-03-28T18:41:21.097408+0000","last_unstale":"2024-03-28T18:42:58.972016+0000","last_undegraded":"2024-03-28T18:42:58.972016+0000","last_fullsized":"2024-03-28T18:42:58.972016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:04:34.407126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987551+0000","last_change":"2024-03-28T18:41:21.115799+0000","last_active":"2024-03-28T18:42:58.987551+0000","last_peered":"2024-03-28T18:42:58.987551+0000","last_clean":"2024-03-28T18:42:58.987551+0000","last_became_active":"2024-03-28T18:41:21.112368+0000","last_became_peered":"2024-03-28T18:41:21.112368+0000","last_unstale":"2024-03-28T18:42:58.987551+0000","last_undegraded":"2024-03-28T18:42:58.987551+0000","last_fullsized":"2024-03-28T18:42:58.987551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:10:59.720380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987723+0000","last_change":"2024-03-28T18:41:21.124383+0000","last_active":"2024-03-28T18:42:58.987723+0000","last_peered":"2024-03-28T18:42:58.987723+0000","last_clean":"2024-03-28T18:42:58.987723+0000","last_became_active":"2024-03-28T18:41:21.121297+0000","last_became_peered":"2024-03-28T18:41:21.121297+0000","last_unstale":"2024-03-28T18:42:58.987723+0000","last_undegraded":"2024-03-28T18:42:58.987723+0000","last_fullsized":"2024-03-28T18:42:58.987723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:44:27.372693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989419+0000","last_change":"2024-03-28T18:41:21.101129+0000","last_active":"2024-03-28T18:42:58.989419+0000","last_peered":"2024-03-28T18:42:58.989419+0000","last_clean":"2024-03-28T18:42:58.989419+0000","last_became_active":"2024-03-28T18:41:21.100793+0000","last_became_peered":"2024-03-28T18:41:21.100793+0000","last_unstale":"2024-03-28T18:42:58.989419+0000","last_undegraded":"2024-03-28T18:42:58.989419+0000","last_fullsized":"2024-03-28T18:42:58.989419+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:15:27.605131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.986304+0000","last_change":"2024-03-28T18:41:21.114190+0000","last_active":"2024-03-28T18:42:58.986304+0000","last_peered":"2024-03-28T18:42:58.986304+0000","last_clean":"2024-03-28T18:42:58.986304+0000","last_became_active":"2024-03-28T18:41:21.113832+0000","last_became_peered":"2024-03-28T18:41:21.113832+0000","last_unstale":"2024-03-28T18:42:58.986304+0000","last_undegraded":"2024-03-28T18:42:58.986304+0000","last_fullsized":"2024-03-28T18:42:58.986304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:29:45.655345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980073+0000","last_change":"2024-03-28T18:41:21.104096+0000","last_active":"2024-03-28T18:42:58.980073+0000","last_peered":"2024-03-28T18:42:58.980073+0000","last_clean":"2024-03-28T18:42:58.980073+0000","last_became_active":"2024-03-28T18:41:21.102660+0000","last_became_peered":"2024-03-28T18:41:21.102660+0000","last_unstale":"2024-03-28T18:42:58.980073+0000","last_undegraded":"2024-03-28T18:42:58.980073+0000","last_fullsized":"2024-03-28T18:42:58.980073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:40:11.226664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983143+0000","last_change":"2024-03-28T18:41:21.121922+0000","last_active":"2024-03-28T18:42:58.983143+0000","last_peered":"2024-03-28T18:42:58.983143+0000","last_clean":"2024-03-28T18:42:58.983143+0000","last_became_active":"2024-03-28T18:41:21.121548+0000","last_became_peered":"2024-03-28T18:41:21.121548+0000","last_unstale":"2024-03-28T18:42:58.983143+0000","last_undegraded":"2024-03-28T18:42:58.983143+0000","last_fullsized":"2024-03-28T18:42:58.983143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:59:04.880906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.509497+0000","last_change":"2024-03-28T18:41:20.078458+0000","last_active":"2024-03-28T18:43:43.509497+0000","last_peered":"2024-03-28T18:43:43.509497+0000","last_clean":"2024-03-28T18:43:43.509497+0000","last_became_active":"2024-03-28T18:41:20.078087+0000","last_became_peered":"2024-03-28T18:41:20.078087+0000","last_unstale":"2024-03-28T18:43:43.509497+0000","last_undegraded":"2024-03-28T18:43:43.509497+0000","last_fullsized":"2024-03-28T18:43:43.509497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:13:42.008001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976163+0000","last_change":"2024-03-28T18:41:21.125911+0000","last_active":"2024-03-28T18:42:58.976163+0000","last_peered":"2024-03-28T18:42:58.976163+0000","last_clean":"2024-03-28T18:42:58.976163+0000","last_became_active":"2024-03-28T18:41:21.125548+0000","last_became_peered":"2024-03-28T18:41:21.125548+0000","last_unstale":"2024-03-28T18:42:58.976163+0000","last_undegraded":"2024-03-28T18:42:58.976163+0000","last_fullsized":"2024-03-28T18:42:58.976163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:48.541926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.405108+0000","last_change":"2024-03-28T18:41:20.084364+0000","last_active":"2024-03-28T18:42:31.405108+0000","last_peered":"2024-03-28T18:42:31.405108+0000","last_clean":"2024-03-28T18:42:31.405108+0000","last_became_active":"2024-03-28T18:41:20.083942+0000","last_became_peered":"2024-03-28T18:41:20.083942+0000","last_unstale":"2024-03-28T18:42:31.405108+0000","last_undegraded":"2024-03-28T18:42:31.405108+0000","last_fullsized":"2024-03-28T18:42:31.405108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:44:34.335574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987959+0000","last_change":"2024-03-28T18:41:21.121306+0000","last_active":"2024-03-28T18:42:58.987959+0000","last_peered":"2024-03-28T18:42:58.987959+0000","last_clean":"2024-03-28T18:42:58.987959+0000","last_became_active":"2024-03-28T18:41:21.121000+0000","last_became_peered":"2024-03-28T18:41:21.121000+0000","last_unstale":"2024-03-28T18:42:58.987959+0000","last_undegraded":"2024-03-28T18:42:58.987959+0000","last_fullsized":"2024-03-28T18:42:58.987959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:35:49.297581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239229+0000","last_change":"2024-03-28T18:41:30.335571+0000","last_active":"2024-03-28T18:42:31.239229+0000","last_peered":"2024-03-28T18:42:31.239229+0000","last_clean":"2024-03-28T18:42:31.239229+0000","last_became_active":"2024-03-28T18:41:30.311420+0000","last_became_peered":"2024-03-28T18:41:30.311420+0000","last_unstale":"2024-03-28T18:42:31.239229+0000","last_undegraded":"2024-03-28T18:42:31.239229+0000","last_fullsized":"2024-03-28T18:42:31.239229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:48:39.796667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972421+0000","last_change":"2024-03-28T18:41:21.092540+0000","last_active":"2024-03-28T18:42:58.972421+0000","last_peered":"2024-03-28T18:42:58.972421+0000","last_clean":"2024-03-28T18:42:58.972421+0000","last_became_active":"2024-03-28T18:41:21.091850+0000","last_became_peered":"2024-03-28T18:41:21.091850+0000","last_unstale":"2024-03-28T18:42:58.972421+0000","last_undegraded":"2024-03-28T18:42:58.972421+0000","last_fullsized":"2024-03-28T18:42:58.972421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:48:03.690373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239193+0000","last_change":"2024-03-28T18:41:30.315981+0000","last_active":"2024-03-28T18:42:31.239193+0000","last_peered":"2024-03-28T18:42:31.239193+0000","last_clean":"2024-03-28T18:42:31.239193+0000","last_became_active":"2024-03-28T18:41:30.311208+0000","last_became_peered":"2024-03-28T18:41:30.311208+0000","last_unstale":"2024-03-28T18:42:31.239193+0000","last_undegraded":"2024-03-28T18:42:31.239193+0000","last_fullsized":"2024-03-28T18:42:31.239193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:33:10.817446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.962262+0000","last_change":"2024-03-28T18:41:21.097340+0000","last_active":"2024-03-28T18:42:58.962262+0000","last_peered":"2024-03-28T18:42:58.962262+0000","last_clean":"2024-03-28T18:42:58.962262+0000","last_became_active":"2024-03-28T18:41:21.096952+0000","last_became_peered":"2024-03-28T18:41:21.096952+0000","last_unstale":"2024-03-28T18:42:58.962262+0000","last_undegraded":"2024-03-28T18:42:58.962262+0000","last_fullsized":"2024-03-28T18:42:58.962262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:02:06.751200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:46.346077+0000","last_change":"2024-03-28T18:41:20.067031+0000","last_active":"2024-03-28T18:43:46.346077+0000","last_peered":"2024-03-28T18:43:46.346077+0000","last_clean":"2024-03-28T18:43:46.346077+0000","last_became_active":"2024-03-28T18:41:20.066698+0000","last_became_peered":"2024-03-28T18:41:20.066698+0000","last_unstale":"2024-03-28T18:43:46.346077+0000","last_undegraded":"2024-03-28T18:43:46.346077+0000","last_fullsized":"2024-03-28T18:43:46.346077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:27:51.877731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1460,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.969774+0000","last_change":"2024-03-28T18:41:21.103746+0000","last_active":"2024-03-28T18:42:58.969774+0000","last_peered":"2024-03-28T18:42:58.969774+0000","last_clean":"2024-03-28T18:42:58.969774+0000","last_became_active":"2024-03-28T18:41:21.103395+0000","last_became_peered":"2024-03-28T18:41:21.103395+0000","last_unstale":"2024-03-28T18:42:58.969774+0000","last_undegraded":"2024-03-28T18:42:58.969774+0000","last_fullsized":"2024-03-28T18:42:58.969774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:20:54.554928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.269565+0000","last_change":"2024-03-28T18:41:30.228521+0000","last_active":"2024-03-28T18:42:31.269565+0000","last_peered":"2024-03-28T18:42:31.269565+0000","last_clean":"2024-03-28T18:42:31.269565+0000","last_became_active":"2024-03-28T18:41:30.221867+0000","last_became_peered":"2024-03-28T18:41:30.221867+0000","last_unstale":"2024-03-28T18:42:31.269565+0000","last_undegraded":"2024-03-28T18:42:31.269565+0000","last_fullsized":"2024-03-28T18:42:31.269565+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:02:45.207508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968706+0000","last_change":"2024-03-28T18:41:30.507663+0000","last_active":"2024-03-28T18:42:58.968706+0000","last_peered":"2024-03-28T18:42:58.968706+0000","last_clean":"2024-03-28T18:42:58.968706+0000","last_became_active":"2024-03-28T18:41:30.507284+0000","last_became_peered":"2024-03-28T18:41:30.507284+0000","last_unstale":"2024-03-28T18:42:58.968706+0000","last_undegraded":"2024-03-28T18:42:58.968706+0000","last_fullsized":"2024-03-28T18:42:58.968706+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:02:03.045818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.175185+0000","last_change":"2024-03-28T18:41:20.079709+0000","last_active":"2024-03-28T18:42:31.175185+0000","last_peered":"2024-03-28T18:42:31.175185+0000","last_clean":"2024-03-28T18:42:31.175185+0000","last_became_active":"2024-03-28T18:41:20.079011+0000","last_became_peered":"2024-03-28T18:41:20.079011+0000","last_unstale":"2024-03-28T18:42:31.175185+0000","last_undegraded":"2024-03-28T18:42:31.175185+0000","last_fullsized":"2024-03-28T18:42:31.175185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:13:38.787593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982042+0000","last_change":"2024-03-28T18:41:21.122100+0000","last_active":"2024-03-28T18:42:58.982042+0000","last_peered":"2024-03-28T18:42:58.982042+0000","last_clean":"2024-03-28T18:42:58.982042+0000","last_became_active":"2024-03-28T18:41:21.121894+0000","last_became_peered":"2024-03-28T18:41:21.121894+0000","last_unstale":"2024-03-28T18:42:58.982042+0000","last_undegraded":"2024-03-28T18:42:58.982042+0000","last_fullsized":"2024-03-28T18:42:58.982042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:39:45.600888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168096+0000","last_change":"2024-03-28T18:41:20.064910+0000","last_active":"2024-03-28T18:42:31.168096+0000","last_peered":"2024-03-28T18:42:31.168096+0000","last_clean":"2024-03-28T18:42:31.168096+0000","last_became_active":"2024-03-28T18:41:20.064687+0000","last_became_peered":"2024-03-28T18:41:20.064687+0000","last_unstale":"2024-03-28T18:42:31.168096+0000","last_undegraded":"2024-03-28T18:42:31.168096+0000","last_fullsized":"2024-03-28T18:42:31.168096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:23:17.147867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.980414+0000","last_change":"2024-03-28T18:41:21.119148+0000","last_active":"2024-03-28T18:42:58.980414+0000","last_peered":"2024-03-28T18:42:58.980414+0000","last_clean":"2024-03-28T18:42:58.980414+0000","last_became_active":"2024-03-28T18:41:21.118786+0000","last_became_peered":"2024-03-28T18:41:21.118786+0000","last_unstale":"2024-03-28T18:42:58.980414+0000","last_undegraded":"2024-03-28T18:42:58.980414+0000","last_fullsized":"2024-03-28T18:42:58.980414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:08.804154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.439337+0000","last_change":"2024-03-28T18:41:20.079355+0000","last_active":"2024-03-28T18:43:47.439337+0000","last_peered":"2024-03-28T18:43:47.439337+0000","last_clean":"2024-03-28T18:43:47.439337+0000","last_became_active":"2024-03-28T18:41:20.078987+0000","last_became_peered":"2024-03-28T18:41:20.078987+0000","last_unstale":"2024-03-28T18:43:47.439337+0000","last_undegraded":"2024-03-28T18:43:47.439337+0000","last_fullsized":"2024-03-28T18:43:47.439337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:51:36.632808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":280,"num_read_kb":280,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982861+0000","last_change":"2024-03-28T18:41:21.100962+0000","last_active":"2024-03-28T18:42:58.982861+0000","last_peered":"2024-03-28T18:42:58.982861+0000","last_clean":"2024-03-28T18:42:58.982861+0000","last_became_active":"2024-03-28T18:41:21.100313+0000","last_became_peered":"2024-03-28T18:41:21.100313+0000","last_unstale":"2024-03-28T18:42:58.982861+0000","last_undegraded":"2024-03-28T18:42:58.982861+0000","last_fullsized":"2024-03-28T18:42:58.982861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:32:36.705216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360380+0000","last_change":"2024-03-28T18:41:20.067292+0000","last_active":"2024-03-28T18:42:31.360380+0000","last_peered":"2024-03-28T18:42:31.360380+0000","last_clean":"2024-03-28T18:42:31.360380+0000","last_became_active":"2024-03-28T18:41:20.066961+0000","last_became_peered":"2024-03-28T18:41:20.066961+0000","last_unstale":"2024-03-28T18:42:31.360380+0000","last_undegraded":"2024-03-28T18:42:31.360380+0000","last_fullsized":"2024-03-28T18:42:31.360380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:02:55.060532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987661+0000","last_change":"2024-03-28T18:41:21.120308+0000","last_active":"2024-03-28T18:42:58.987661+0000","last_peered":"2024-03-28T18:42:58.987661+0000","last_clean":"2024-03-28T18:42:58.987661+0000","last_became_active":"2024-03-28T18:41:21.118521+0000","last_became_peered":"2024-03-28T18:41:21.118521+0000","last_unstale":"2024-03-28T18:42:58.987661+0000","last_undegraded":"2024-03-28T18:42:58.987661+0000","last_fullsized":"2024-03-28T18:42:58.987661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:55.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168678+0000","last_change":"2024-03-28T18:41:20.072783+0000","last_active":"2024-03-28T18:42:31.168678+0000","last_peered":"2024-03-28T18:42:31.168678+0000","last_clean":"2024-03-28T18:42:31.168678+0000","last_became_active":"2024-03-28T18:41:20.072549+0000","last_became_peered":"2024-03-28T18:41:20.072549+0000","last_unstale":"2024-03-28T18:42:31.168678+0000","last_undegraded":"2024-03-28T18:42:31.168678+0000","last_fullsized":"2024-03-28T18:42:31.168678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:17:11.877955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978619+0000","last_change":"2024-03-28T18:42:31.303127+0000","last_active":"2024-03-28T18:42:58.978619+0000","last_peered":"2024-03-28T18:42:58.978619+0000","last_clean":"2024-03-28T18:42:58.978619+0000","last_became_active":"2024-03-28T18:42:31.301321+0000","last_became_peered":"2024-03-28T18:42:31.301321+0000","last_unstale":"2024-03-28T18:42:58.978619+0000","last_undegraded":"2024-03-28T18:42:58.978619+0000","last_fullsized":"2024-03-28T18:42:58.978619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:08.892803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'38","reported_seq":351,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:47.346068+0000","last_change":"2024-03-28T18:41:20.064049+0000","last_active":"2024-03-28T18:43:47.346068+0000","last_peered":"2024-03-28T18:43:47.346068+0000","last_clean":"2024-03-28T18:43:47.346068+0000","last_became_active":"2024-03-28T18:41:20.063574+0000","last_became_peered":"2024-03-28T18:41:20.063574+0000","last_unstale":"2024-03-28T18:43:47.346068+0000","last_undegraded":"2024-03-28T18:43:47.346068+0000","last_fullsized":"2024-03-28T18:43:47.346068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:24:14.320841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":205,"num_write":40,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.948299+0000","last_change":"2024-03-28T18:41:21.098087+0000","last_active":"2024-03-28T18:42:58.948299+0000","last_peered":"2024-03-28T18:42:58.948299+0000","last_clean":"2024-03-28T18:42:58.948299+0000","last_became_active":"2024-03-28T18:41:21.097730+0000","last_became_peered":"2024-03-28T18:41:21.097730+0000","last_unstale":"2024-03-28T18:42:58.948299+0000","last_undegraded":"2024-03-28T18:42:58.948299+0000","last_fullsized":"2024-03-28T18:42:58.948299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:15:17.476007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317615+0000","last_change":"2024-03-28T18:41:20.086814+0000","last_active":"2024-03-28T18:42:31.317615+0000","last_peered":"2024-03-28T18:42:31.317615+0000","last_clean":"2024-03-28T18:42:31.317615+0000","last_became_active":"2024-03-28T18:41:20.086466+0000","last_became_peered":"2024-03-28T18:41:20.086466+0000","last_unstale":"2024-03-28T18:42:31.317615+0000","last_undegraded":"2024-03-28T18:42:31.317615+0000","last_fullsized":"2024-03-28T18:42:31.317615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:29:27.253766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983586+0000","last_change":"2024-03-28T18:41:21.119997+0000","last_active":"2024-03-28T18:42:58.983586+0000","last_peered":"2024-03-28T18:42:58.983586+0000","last_clean":"2024-03-28T18:42:58.983586+0000","last_became_active":"2024-03-28T18:41:21.119642+0000","last_became_peered":"2024-03-28T18:41:21.119642+0000","last_unstale":"2024-03-28T18:42:58.983586+0000","last_undegraded":"2024-03-28T18:42:58.983586+0000","last_fullsized":"2024-03-28T18:42:58.983586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:28:34.990391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168283+0000","last_change":"2024-03-28T18:41:20.085976+0000","last_active":"2024-03-28T18:42:31.168283+0000","last_peered":"2024-03-28T18:42:31.168283+0000","last_clean":"2024-03-28T18:42:31.168283+0000","last_became_active":"2024-03-28T18:41:20.085492+0000","last_became_peered":"2024-03-28T18:41:20.085492+0000","last_unstale":"2024-03-28T18:42:31.168283+0000","last_undegraded":"2024-03-28T18:42:31.168283+0000","last_fullsized":"2024-03-28T18:42:31.168283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:38:30.924967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982443+0000","last_change":"2024-03-28T18:41:21.105636+0000","last_active":"2024-03-28T18:42:58.982443+0000","last_peered":"2024-03-28T18:42:58.982443+0000","last_clean":"2024-03-28T18:42:58.982443+0000","last_became_active":"2024-03-28T18:41:21.105252+0000","last_became_peered":"2024-03-28T18:41:21.105252+0000","last_unstale":"2024-03-28T18:42:58.982443+0000","last_undegraded":"2024-03-28T18:42:58.982443+0000","last_fullsized":"2024-03-28T18:42:58.982443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:01:29.454056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":171,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:43.419661+0000","last_change":"2024-03-28T18:41:20.067189+0000","last_active":"2024-03-28T18:43:43.419661+0000","last_peered":"2024-03-28T18:43:43.419661+0000","last_clean":"2024-03-28T18:43:43.419661+0000","last_became_active":"2024-03-28T18:41:20.066789+0000","last_became_peered":"2024-03-28T18:41:20.066789+0000","last_unstale":"2024-03-28T18:43:43.419661+0000","last_undegraded":"2024-03-28T18:43:43.419661+0000","last_fullsized":"2024-03-28T18:43:43.419661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:26.920863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970949+0000","last_change":"2024-03-28T18:41:21.091336+0000","last_active":"2024-03-28T18:42:58.970949+0000","last_peered":"2024-03-28T18:42:58.970949+0000","last_clean":"2024-03-28T18:42:58.970949+0000","last_became_active":"2024-03-28T18:41:21.090999+0000","last_became_peered":"2024-03-28T18:41:21.090999+0000","last_unstale":"2024-03-28T18:42:58.970949+0000","last_undegraded":"2024-03-28T18:42:58.970949+0000","last_fullsized":"2024-03-28T18:42:58.970949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:17:25.004323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.238896+0000","last_change":"2024-03-28T18:41:20.076502+0000","last_active":"2024-03-28T18:42:31.238896+0000","last_peered":"2024-03-28T18:42:31.238896+0000","last_clean":"2024-03-28T18:42:31.238896+0000","last_became_active":"2024-03-28T18:41:20.076185+0000","last_became_peered":"2024-03-28T18:41:20.076185+0000","last_unstale":"2024-03-28T18:42:31.238896+0000","last_undegraded":"2024-03-28T18:42:31.238896+0000","last_fullsized":"2024-03-28T18:42:31.238896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:55:37.683090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.957235+0000","last_change":"2024-03-28T18:41:21.097749+0000","last_active":"2024-03-28T18:42:58.957235+0000","last_peered":"2024-03-28T18:42:58.957235+0000","last_clean":"2024-03-28T18:42:58.957235+0000","last_became_active":"2024-03-28T18:41:21.097287+0000","last_became_peered":"2024-03-28T18:41:21.097287+0000","last_unstale":"2024-03-28T18:42:58.957235+0000","last_undegraded":"2024-03-28T18:42:58.957235+0000","last_fullsized":"2024-03-28T18:42:58.957235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:45:28.571446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239439+0000","last_change":"2024-03-28T18:41:20.063007+0000","last_active":"2024-03-28T18:42:31.239439+0000","last_peered":"2024-03-28T18:42:31.239439+0000","last_clean":"2024-03-28T18:42:31.239439+0000","last_became_active":"2024-03-28T18:41:20.062681+0000","last_became_peered":"2024-03-28T18:41:20.062681+0000","last_unstale":"2024-03-28T18:42:31.239439+0000","last_undegraded":"2024-03-28T18:42:31.239439+0000","last_fullsized":"2024-03-28T18:42:31.239439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:11:22.790077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971622+0000","last_change":"2024-03-28T18:41:21.100798+0000","last_active":"2024-03-28T18:42:58.971622+0000","last_peered":"2024-03-28T18:42:58.971622+0000","last_clean":"2024-03-28T18:42:58.971622+0000","last_became_active":"2024-03-28T18:41:21.100456+0000","last_became_peered":"2024-03-28T18:41:21.100456+0000","last_unstale":"2024-03-28T18:42:58.971622+0000","last_undegraded":"2024-03-28T18:42:58.971622+0000","last_fullsized":"2024-03-28T18:42:58.971622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:47:39.730490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.173997+0000","last_change":"2024-03-28T18:41:30.220354+0000","last_active":"2024-03-28T18:42:31.173997+0000","last_peered":"2024-03-28T18:42:31.173997+0000","last_clean":"2024-03-28T18:42:31.173997+0000","last_became_active":"2024-03-28T18:41:30.218579+0000","last_became_peered":"2024-03-28T18:41:30.218579+0000","last_unstale":"2024-03-28T18:42:31.173997+0000","last_undegraded":"2024-03-28T18:42:31.173997+0000","last_fullsized":"2024-03-28T18:42:31.173997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.973056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.976756+0000","last_change":"2024-03-28T18:41:21.121958+0000","last_active":"2024-03-28T18:42:58.976756+0000","last_peered":"2024-03-28T18:42:58.976756+0000","last_clean":"2024-03-28T18:42:58.976756+0000","last_became_active":"2024-03-28T18:41:21.121564+0000","last_became_peered":"2024-03-28T18:41:21.121564+0000","last_unstale":"2024-03-28T18:42:58.976756+0000","last_undegraded":"2024-03-28T18:42:58.976756+0000","last_fullsized":"2024-03-28T18:42:58.976756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:46:23.336796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":169,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.514348+0000","last_change":"2024-03-28T18:41:20.077970+0000","last_active":"2024-03-28T18:43:45.514348+0000","last_peered":"2024-03-28T18:43:45.514348+0000","last_clean":"2024-03-28T18:43:45.514348+0000","last_became_active":"2024-03-28T18:41:20.077544+0000","last_became_peered":"2024-03-28T18:41:20.077544+0000","last_unstale":"2024-03-28T18:43:45.514348+0000","last_undegraded":"2024-03-28T18:43:45.514348+0000","last_fullsized":"2024-03-28T18:43:45.514348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:34.911454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":140,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988444+0000","last_change":"2024-03-28T18:41:30.219905+0000","last_active":"2024-03-28T18:42:58.988444+0000","last_peered":"2024-03-28T18:42:58.988444+0000","last_clean":"2024-03-28T18:42:58.988444+0000","last_became_active":"2024-03-28T18:41:30.219488+0000","last_became_peered":"2024-03-28T18:41:30.219488+0000","last_unstale":"2024-03-28T18:42:58.988444+0000","last_undegraded":"2024-03-28T18:42:58.988444+0000","last_fullsized":"2024-03-28T18:42:58.988444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:24:16.108678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975656+0000","last_change":"2024-03-28T18:42:31.320096+0000","last_active":"2024-03-28T18:42:58.975656+0000","last_peered":"2024-03-28T18:42:58.975656+0000","last_clean":"2024-03-28T18:42:58.975656+0000","last_became_active":"2024-03-28T18:42:31.318988+0000","last_became_peered":"2024-03-28T18:42:31.318988+0000","last_unstale":"2024-03-28T18:42:58.975656+0000","last_undegraded":"2024-03-28T18:42:58.975656+0000","last_fullsized":"2024-03-28T18:42:58.975656+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:23:50.019195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.422734+0000","last_change":"2024-03-28T18:41:20.083520+0000","last_active":"2024-03-28T18:43:45.422734+0000","last_peered":"2024-03-28T18:43:45.422734+0000","last_clean":"2024-03-28T18:43:45.422734+0000","last_became_active":"2024-03-28T18:41:20.083198+0000","last_became_peered":"2024-03-28T18:41:20.083198+0000","last_unstale":"2024-03-28T18:43:45.422734+0000","last_undegraded":"2024-03-28T18:43:45.422734+0000","last_fullsized":"2024-03-28T18:43:45.422734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:46:23.923619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":140,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.975540+0000","last_change":"2024-03-28T18:41:21.115918+0000","last_active":"2024-03-28T18:42:58.975540+0000","last_peered":"2024-03-28T18:42:58.975540+0000","last_clean":"2024-03-28T18:42:58.975540+0000","last_became_active":"2024-03-28T18:41:21.113175+0000","last_became_peered":"2024-03-28T18:41:21.113175+0000","last_unstale":"2024-03-28T18:42:58.975540+0000","last_undegraded":"2024-03-28T18:42:58.975540+0000","last_fullsized":"2024-03-28T18:42:58.975540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:34:22.163079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299771+0000","last_change":"2024-03-28T18:41:20.075790+0000","last_active":"2024-03-28T18:42:31.299771+0000","last_peered":"2024-03-28T18:42:31.299771+0000","last_clean":"2024-03-28T18:42:31.299771+0000","last_became_active":"2024-03-28T18:41:20.075374+0000","last_became_peered":"2024-03-28T18:41:20.075374+0000","last_unstale":"2024-03-28T18:42:31.299771+0000","last_undegraded":"2024-03-28T18:42:31.299771+0000","last_fullsized":"2024-03-28T18:42:31.299771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:00:12.281193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360313+0000","last_change":"2024-03-28T18:40:01.582122+0000","last_active":"2024-03-28T18:42:31.360313+0000","last_peered":"2024-03-28T18:42:31.360313+0000","last_clean":"2024-03-28T18:42:31.360313+0000","last_became_active":"2024-03-28T18:40:01.581615+0000","last_became_peered":"2024-03-28T18:40:01.581615+0000","last_unstale":"2024-03-28T18:42:31.360313+0000","last_undegraded":"2024-03-28T18:42:31.360313+0000","last_fullsized":"2024-03-28T18:42:31.360313+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:36:13.568716+0000","last_clean_scrub_stamp":"2024-03-28T18:36:13.568716+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:26.587402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.964665+0000","last_change":"2024-03-28T18:41:21.093278+0000","last_active":"2024-03-28T18:42:58.964665+0000","last_peered":"2024-03-28T18:42:58.964665+0000","last_clean":"2024-03-28T18:42:58.964665+0000","last_became_active":"2024-03-28T18:41:21.092750+0000","last_became_peered":"2024-03-28T18:41:21.092750+0000","last_unstale":"2024-03-28T18:42:58.964665+0000","last_undegraded":"2024-03-28T18:42:58.964665+0000","last_fullsized":"2024-03-28T18:42:58.964665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:22:43.460064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.360379+0000","last_change":"2024-03-28T18:41:20.073123+0000","last_active":"2024-03-28T18:42:31.360379+0000","last_peered":"2024-03-28T18:42:31.360379+0000","last_clean":"2024-03-28T18:42:31.360379+0000","last_became_active":"2024-03-28T18:41:20.072671+0000","last_became_peered":"2024-03-28T18:41:20.072671+0000","last_unstale":"2024-03-28T18:42:31.360379+0000","last_undegraded":"2024-03-28T18:42:31.360379+0000","last_fullsized":"2024-03-28T18:42:31.360379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:13.566291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.411326+0000","last_change":"2024-03-28T18:41:30.227916+0000","last_active":"2024-03-28T18:42:31.411326+0000","last_peered":"2024-03-28T18:42:31.411326+0000","last_clean":"2024-03-28T18:42:31.411326+0000","last_became_active":"2024-03-28T18:41:30.218947+0000","last_became_peered":"2024-03-28T18:41:30.218947+0000","last_unstale":"2024-03-28T18:42:31.411326+0000","last_undegraded":"2024-03-28T18:42:31.411326+0000","last_fullsized":"2024-03-28T18:42:31.411326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:09:14.922324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983366+0000","last_change":"2024-03-28T18:41:21.104753+0000","last_active":"2024-03-28T18:42:58.983366+0000","last_peered":"2024-03-28T18:42:58.983366+0000","last_clean":"2024-03-28T18:42:58.983366+0000","last_became_active":"2024-03-28T18:41:21.104378+0000","last_became_peered":"2024-03-28T18:41:21.104378+0000","last_unstale":"2024-03-28T18:42:58.983366+0000","last_undegraded":"2024-03-28T18:42:58.983366+0000","last_fullsized":"2024-03-28T18:42:58.983366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:34:42.066051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:30.238634+0000","last_change":"2024-03-28T18:41:20.086186+0000","last_active":"2024-03-28T18:43:30.238634+0000","last_peered":"2024-03-28T18:43:30.238634+0000","last_clean":"2024-03-28T18:43:30.238634+0000","last_became_active":"2024-03-28T18:41:20.085851+0000","last_became_peered":"2024-03-28T18:41:20.085851+0000","last_unstale":"2024-03-28T18:43:30.238634+0000","last_undegraded":"2024-03-28T18:43:30.238634+0000","last_fullsized":"2024-03-28T18:43:30.238634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:03:43.127548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.983095+0000","last_change":"2024-03-28T18:41:21.124613+0000","last_active":"2024-03-28T18:42:58.983095+0000","last_peered":"2024-03-28T18:42:58.983095+0000","last_clean":"2024-03-28T18:42:58.983095+0000","last_became_active":"2024-03-28T18:41:21.124258+0000","last_became_peered":"2024-03-28T18:41:21.124258+0000","last_unstale":"2024-03-28T18:42:58.983095+0000","last_undegraded":"2024-03-28T18:42:58.983095+0000","last_fullsized":"2024-03-28T18:42:58.983095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:52:25.914071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.299859+0000","last_change":"2024-03-28T18:41:20.069562+0000","last_active":"2024-03-28T18:42:31.299859+0000","last_peered":"2024-03-28T18:42:31.299859+0000","last_clean":"2024-03-28T18:42:31.299859+0000","last_became_active":"2024-03-28T18:41:20.068971+0000","last_became_peered":"2024-03-28T18:41:20.068971+0000","last_unstale":"2024-03-28T18:42:31.299859+0000","last_undegraded":"2024-03-28T18:42:31.299859+0000","last_fullsized":"2024-03-28T18:42:31.299859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:47.713261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988228+0000","last_change":"2024-03-28T18:42:31.302339+0000","last_active":"2024-03-28T18:42:58.988228+0000","last_peered":"2024-03-28T18:42:58.988228+0000","last_clean":"2024-03-28T18:42:58.988228+0000","last_became_active":"2024-03-28T18:42:31.301324+0000","last_became_peered":"2024-03-28T18:42:31.301324+0000","last_unstale":"2024-03-28T18:42:58.988228+0000","last_undegraded":"2024-03-28T18:42:58.988228+0000","last_fullsized":"2024-03-28T18:42:58.988228+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:37:56.490969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.167540+0000","last_change":"2024-03-28T18:41:20.077420+0000","last_active":"2024-03-28T18:42:31.167540+0000","last_peered":"2024-03-28T18:42:31.167540+0000","last_clean":"2024-03-28T18:42:31.167540+0000","last_became_active":"2024-03-28T18:41:20.077048+0000","last_became_peered":"2024-03-28T18:41:20.077048+0000","last_unstale":"2024-03-28T18:42:31.167540+0000","last_undegraded":"2024-03-28T18:42:31.167540+0000","last_fullsized":"2024-03-28T18:42:31.167540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:07:00.496836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.941667+0000","last_change":"2024-03-28T18:41:21.097033+0000","last_active":"2024-03-28T18:42:58.941667+0000","last_peered":"2024-03-28T18:42:58.941667+0000","last_clean":"2024-03-28T18:42:58.941667+0000","last_became_active":"2024-03-28T18:41:21.095814+0000","last_became_peered":"2024-03-28T18:41:21.095814+0000","last_unstale":"2024-03-28T18:42:58.941667+0000","last_undegraded":"2024-03-28T18:42:58.941667+0000","last_fullsized":"2024-03-28T18:42:58.941667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:00.355512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.403649+0000","last_change":"2024-03-28T18:41:20.085057+0000","last_active":"2024-03-28T18:42:31.403649+0000","last_peered":"2024-03-28T18:42:31.403649+0000","last_clean":"2024-03-28T18:42:31.403649+0000","last_became_active":"2024-03-28T18:41:20.084594+0000","last_became_peered":"2024-03-28T18:41:20.084594+0000","last_unstale":"2024-03-28T18:42:31.403649+0000","last_undegraded":"2024-03-28T18:42:31.403649+0000","last_fullsized":"2024-03-28T18:42:31.403649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:31:21.969346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987979+0000","last_change":"2024-03-28T18:41:21.103489+0000","last_active":"2024-03-28T18:42:58.987979+0000","last_peered":"2024-03-28T18:42:58.987979+0000","last_clean":"2024-03-28T18:42:58.987979+0000","last_became_active":"2024-03-28T18:41:21.103130+0000","last_became_peered":"2024-03-28T18:41:21.103130+0000","last_unstale":"2024-03-28T18:42:58.987979+0000","last_undegraded":"2024-03-28T18:42:58.987979+0000","last_fullsized":"2024-03-28T18:42:58.987979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:29.015256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.317603+0000","last_change":"2024-03-28T18:41:20.086793+0000","last_active":"2024-03-28T18:42:31.317603+0000","last_peered":"2024-03-28T18:42:31.317603+0000","last_clean":"2024-03-28T18:42:31.317603+0000","last_became_active":"2024-03-28T18:41:20.086403+0000","last_became_peered":"2024-03-28T18:41:20.086403+0000","last_unstale":"2024-03-28T18:42:31.317603+0000","last_undegraded":"2024-03-28T18:42:31.317603+0000","last_fullsized":"2024-03-28T18:42:31.317603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:33:59.008625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.989314+0000","last_change":"2024-03-28T18:41:21.128580+0000","last_active":"2024-03-28T18:42:58.989314+0000","last_peered":"2024-03-28T18:42:58.989314+0000","last_clean":"2024-03-28T18:42:58.989314+0000","last_became_active":"2024-03-28T18:41:21.128130+0000","last_became_peered":"2024-03-28T18:41:21.128130+0000","last_unstale":"2024-03-28T18:42:58.989314+0000","last_undegraded":"2024-03-28T18:42:58.989314+0000","last_fullsized":"2024-03-28T18:42:58.989314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:16:34.560263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.239033+0000","last_change":"2024-03-28T18:41:30.319270+0000","last_active":"2024-03-28T18:42:31.239033+0000","last_peered":"2024-03-28T18:42:31.239033+0000","last_clean":"2024-03-28T18:42:31.239033+0000","last_became_active":"2024-03-28T18:41:30.311219+0000","last_became_peered":"2024-03-28T18:41:30.311219+0000","last_unstale":"2024-03-28T18:42:31.239033+0000","last_undegraded":"2024-03-28T18:42:31.239033+0000","last_fullsized":"2024-03-28T18:42:31.239033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T06:24:23.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982021+0000","last_change":"2024-03-28T18:41:21.104563+0000","last_active":"2024-03-28T18:42:58.982021+0000","last_peered":"2024-03-28T18:42:58.982021+0000","last_clean":"2024-03-28T18:42:58.982021+0000","last_became_active":"2024-03-28T18:41:21.104218+0000","last_became_peered":"2024-03-28T18:41:21.104218+0000","last_unstale":"2024-03-28T18:42:58.982021+0000","last_undegraded":"2024-03-28T18:42:58.982021+0000","last_fullsized":"2024-03-28T18:42:58.982021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:07.227245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:43:45.436770+0000","last_change":"2024-03-28T18:41:20.066711+0000","last_active":"2024-03-28T18:43:45.436770+0000","last_peered":"2024-03-28T18:43:45.436770+0000","last_clean":"2024-03-28T18:43:45.436770+0000","last_became_active":"2024-03-28T18:41:20.066375+0000","last_became_peered":"2024-03-28T18:41:20.066375+0000","last_unstale":"2024-03-28T18:43:45.436770+0000","last_undegraded":"2024-03-28T18:43:45.436770+0000","last_fullsized":"2024-03-28T18:43:45.436770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:39:11.415568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.979050+0000","last_change":"2024-03-28T18:41:21.092575+0000","last_active":"2024-03-28T18:42:58.979050+0000","last_peered":"2024-03-28T18:42:58.979050+0000","last_clean":"2024-03-28T18:42:58.979050+0000","last_became_active":"2024-03-28T18:41:21.092262+0000","last_became_peered":"2024-03-28T18:41:21.092262+0000","last_unstale":"2024-03-28T18:42:58.979050+0000","last_undegraded":"2024-03-28T18:42:58.979050+0000","last_fullsized":"2024-03-28T18:42:58.979050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:05.993850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.168484+0000","last_change":"2024-03-28T18:41:20.085769+0000","last_active":"2024-03-28T18:42:31.168484+0000","last_peered":"2024-03-28T18:42:31.168484+0000","last_clean":"2024-03-28T18:42:31.168484+0000","last_became_active":"2024-03-28T18:41:20.085411+0000","last_became_peered":"2024-03-28T18:41:20.085411+0000","last_unstale":"2024-03-28T18:42:31.168484+0000","last_undegraded":"2024-03-28T18:42:31.168484+0000","last_fullsized":"2024-03-28T18:42:31.168484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:04:58.863663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.984053+0000","last_change":"2024-03-28T18:41:21.122149+0000","last_active":"2024-03-28T18:42:58.984053+0000","last_peered":"2024-03-28T18:42:58.984053+0000","last_clean":"2024-03-28T18:42:58.984053+0000","last_became_active":"2024-03-28T18:41:21.121795+0000","last_became_peered":"2024-03-28T18:41:21.121795+0000","last_unstale":"2024-03-28T18:42:58.984053+0000","last_undegraded":"2024-03-28T18:42:58.984053+0000","last_fullsized":"2024-03-28T18:42:58.984053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:22:08.447274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:31.176338+0000","last_change":"2024-03-28T18:41:20.085725+0000","last_active":"2024-03-28T18:42:31.176338+0000","last_peered":"2024-03-28T18:42:31.176338+0000","last_clean":"2024-03-28T18:42:31.176338+0000","last_became_active":"2024-03-28T18:41:20.085154+0000","last_became_peered":"2024-03-28T18:41:20.085154+0000","last_unstale":"2024-03-28T18:42:31.176338+0000","last_undegraded":"2024-03-28T18:42:31.176338+0000","last_fullsized":"2024-03-28T18:42:31.176338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:19.036858+0000","last_clean_scrub_stamp":"2024-03-28T18:41:19.036858+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:37:31.156548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.978116+0000","last_change":"2024-03-28T18:41:21.095273+0000","last_active":"2024-03-28T18:42:58.978116+0000","last_peered":"2024-03-28T18:42:58.978116+0000","last_clean":"2024-03-28T18:42:58.978116+0000","last_became_active":"2024-03-28T18:41:21.094911+0000","last_became_peered":"2024-03-28T18:41:21.094911+0000","last_unstale":"2024-03-28T18:42:58.978116+0000","last_undegraded":"2024-03-28T18:42:58.978116+0000","last_fullsized":"2024-03-28T18:42:58.978116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:58:40.771848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.972976+0000","last_change":"2024-03-28T18:41:21.095303+0000","last_active":"2024-03-28T18:42:58.972976+0000","last_peered":"2024-03-28T18:42:58.972976+0000","last_clean":"2024-03-28T18:42:58.972976+0000","last_became_active":"2024-03-28T18:41:21.094916+0000","last_became_peered":"2024-03-28T18:41:21.094916+0000","last_unstale":"2024-03-28T18:42:58.972976+0000","last_undegraded":"2024-03-28T18:42:58.972976+0000","last_fullsized":"2024-03-28T18:42:58.972976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:26:54.783529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.967314+0000","last_change":"2024-03-28T18:41:21.092780+0000","last_active":"2024-03-28T18:42:58.967314+0000","last_peered":"2024-03-28T18:42:58.967314+0000","last_clean":"2024-03-28T18:42:58.967314+0000","last_became_active":"2024-03-28T18:41:21.092439+0000","last_became_peered":"2024-03-28T18:41:21.092439+0000","last_unstale":"2024-03-28T18:42:58.967314+0000","last_undegraded":"2024-03-28T18:42:58.967314+0000","last_fullsized":"2024-03-28T18:42:58.967314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:32.710421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.987481+0000","last_change":"2024-03-28T18:41:21.121998+0000","last_active":"2024-03-28T18:42:58.987481+0000","last_peered":"2024-03-28T18:42:58.987481+0000","last_clean":"2024-03-28T18:42:58.987481+0000","last_became_active":"2024-03-28T18:41:21.121646+0000","last_became_peered":"2024-03-28T18:41:21.121646+0000","last_unstale":"2024-03-28T18:42:58.987481+0000","last_undegraded":"2024-03-28T18:42:58.987481+0000","last_fullsized":"2024-03-28T18:42:58.987481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:07:51.081772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988446+0000","last_change":"2024-03-28T18:42:31.362520+0000","last_active":"2024-03-28T18:42:58.988446+0000","last_peered":"2024-03-28T18:42:58.988446+0000","last_clean":"2024-03-28T18:42:58.988446+0000","last_became_active":"2024-03-28T18:42:31.361563+0000","last_became_peered":"2024-03-28T18:42:31.361563+0000","last_unstale":"2024-03-28T18:42:58.988446+0000","last_undegraded":"2024-03-28T18:42:58.988446+0000","last_fullsized":"2024-03-28T18:42:58.988446+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:02:27.960042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988524+0000","last_change":"2024-03-28T18:41:21.122302+0000","last_active":"2024-03-28T18:42:58.988524+0000","last_peered":"2024-03-28T18:42:58.988524+0000","last_clean":"2024-03-28T18:42:58.988524+0000","last_became_active":"2024-03-28T18:41:21.121924+0000","last_became_peered":"2024-03-28T18:41:21.121924+0000","last_unstale":"2024-03-28T18:42:58.988524+0000","last_undegraded":"2024-03-28T18:42:58.988524+0000","last_fullsized":"2024-03-28T18:42:58.988524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:26:53.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.971656+0000","last_change":"2024-03-28T18:41:21.095203+0000","last_active":"2024-03-28T18:42:58.971656+0000","last_peered":"2024-03-28T18:42:58.971656+0000","last_clean":"2024-03-28T18:42:58.971656+0000","last_became_active":"2024-03-28T18:41:21.094803+0000","last_became_peered":"2024-03-28T18:41:21.094803+0000","last_unstale":"2024-03-28T18:42:58.971656+0000","last_undegraded":"2024-03-28T18:42:58.971656+0000","last_fullsized":"2024-03-28T18:42:58.971656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T05:09:02.915953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.970936+0000","last_change":"2024-03-28T18:41:21.090716+0000","last_active":"2024-03-28T18:42:58.970936+0000","last_peered":"2024-03-28T18:42:58.970936+0000","last_clean":"2024-03-28T18:42:58.970936+0000","last_became_active":"2024-03-28T18:41:21.089997+0000","last_became_peered":"2024-03-28T18:41:21.089997+0000","last_unstale":"2024-03-28T18:42:58.970936+0000","last_undegraded":"2024-03-28T18:42:58.970936+0000","last_fullsized":"2024-03-28T18:42:58.970936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T04:53:13.095858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.977119+0000","last_change":"2024-03-28T18:41:21.119899+0000","last_active":"2024-03-28T18:42:58.977119+0000","last_peered":"2024-03-28T18:42:58.977119+0000","last_clean":"2024-03-28T18:42:58.977119+0000","last_became_active":"2024-03-28T18:41:21.119546+0000","last_became_peered":"2024-03-28T18:41:21.119546+0000","last_unstale":"2024-03-28T18:42:58.977119+0000","last_undegraded":"2024-03-28T18:42:58.977119+0000","last_fullsized":"2024-03-28T18:42:58.977119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:25.900179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.981154+0000","last_change":"2024-03-28T18:41:30.510267+0000","last_active":"2024-03-28T18:42:58.981154+0000","last_peered":"2024-03-28T18:42:58.981154+0000","last_clean":"2024-03-28T18:42:58.981154+0000","last_became_active":"2024-03-28T18:41:30.508658+0000","last_became_peered":"2024-03-28T18:41:30.508658+0000","last_unstale":"2024-03-28T18:42:58.981154+0000","last_undegraded":"2024-03-28T18:42:58.981154+0000","last_fullsized":"2024-03-28T18:42:58.981154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:25:30.892433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.982988+0000","last_change":"2024-03-28T18:41:21.123194+0000","last_active":"2024-03-28T18:42:58.982988+0000","last_peered":"2024-03-28T18:42:58.982988+0000","last_clean":"2024-03-28T18:42:58.982988+0000","last_became_active":"2024-03-28T18:41:21.122747+0000","last_became_peered":"2024-03-28T18:41:21.122747+0000","last_unstale":"2024-03-28T18:42:58.982988+0000","last_undegraded":"2024-03-28T18:42:58.982988+0000","last_fullsized":"2024-03-28T18:42:58.982988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:50:04.688483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.974718+0000","last_change":"2024-03-28T18:41:21.114973+0000","last_active":"2024-03-28T18:42:58.974718+0000","last_peered":"2024-03-28T18:42:58.974718+0000","last_clean":"2024-03-28T18:42:58.974718+0000","last_became_active":"2024-03-28T18:41:21.114628+0000","last_became_peered":"2024-03-28T18:41:21.114628+0000","last_unstale":"2024-03-28T18:42:58.974718+0000","last_undegraded":"2024-03-28T18:42:58.974718+0000","last_fullsized":"2024-03-28T18:42:58.974718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:07:57.139590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.988976+0000","last_change":"2024-03-28T18:41:21.120766+0000","last_active":"2024-03-28T18:42:58.988976+0000","last_peered":"2024-03-28T18:42:58.988976+0000","last_clean":"2024-03-28T18:42:58.988976+0000","last_became_active":"2024-03-28T18:41:21.120303+0000","last_became_peered":"2024-03-28T18:41:21.120303+0000","last_unstale":"2024-03-28T18:42:58.988976+0000","last_undegraded":"2024-03-28T18:42:58.988976+0000","last_fullsized":"2024-03-28T18:42:58.988976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:23:15.892295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T18:42:58.968063+0000","last_change":"2024-03-28T18:41:21.100886+0000","last_active":"2024-03-28T18:42:58.968063+0000","last_peered":"2024-03-28T18:42:58.968063+0000","last_clean":"2024-03-28T18:42:58.968063+0000","last_became_active":"2024-03-28T18:41:21.100563+0000","last_became_peered":"2024-03-28T18:41:21.100563+0000","last_unstale":"2024-03-28T18:42:58.968063+0000","last_undegraded":"2024-03-28T18:42:58.968063+0000","last_fullsized":"2024-03-28T18:42:58.968063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T18:41:20.045165+0000","last_clean_scrub_stamp":"2024-03-28T18:41:20.045165+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:22:15.037733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":211580,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70048,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1429,"num_read_kb":790,"num_write":255,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":215517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477349,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1131054,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776182,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477167,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037820,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477433,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299457,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":1071042,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":476884,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":540053,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":478223,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":541111,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":476709,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888147,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":11584,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1069606,"data_compressed":47396,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34166,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35812,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T18:43:50.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 -- 172.21.15.119:0/3753736047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19200ffd60 msgr2=0x7f1920100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 --2- 172.21.15.119:0/3753736047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19200ffd60 0x7f1920100180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1910009900 tx=0x7f191002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 -- 172.21.15.119:0/3753736047 shutdown_connections 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 --2- 172.21.15.119:0/3753736047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1920101060 0x7f19201014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 --2- 172.21.15.119:0/3753736047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19201006c0 0x7f1920100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 --2- 172.21.15.119:0/3753736047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19200ffd60 0x7f1920100180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.976+0000 7f1925800640 1 -- 172.21.15.119:0/3753736047 >> 172.21.15.119:0/3753736047 conn(0x7f19200fb980 msgr2=0x7f19200fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.977+0000 7f1925800640 1 -- 172.21.15.119:0/3753736047 shutdown_connections 2024-03-28T18:43:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.977+0000 7f1925800640 1 -- 172.21.15.119:0/3753736047 wait complete. 2024-03-28T18:43:50.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.978+0000 7f1925800640 1 Processor -- start 2024-03-28T18:43:50.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.978+0000 7f1925800640 1 -- start start 2024-03-28T18:43:50.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.978+0000 7f1925800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 0x7f19201999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.978+0000 7f1925800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 0x7f1920199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1925800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1925800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192010f3a0 con 0x7f19200ffd60 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1925800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192010f220 con 0x7f19201006c0 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1925800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192010f520 con 0x7f1920101060 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f191fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 0x7f19201999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f191f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 0x7f1920199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36008/0 (socket says 172.21.15.119:36008) 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 -- 172.21.15.119:0/1968999516 learned_addr learned my addr 172.21.15.119:0/1968999516 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f191fe00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 0x7f19201999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38802/0 (socket says 172.21.15.119:38802) 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 -- 172.21.15.119:0/1968999516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 msgr2=0x7f1920199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 0x7f1920199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 -- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 msgr2=0x7f19201999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 0x7f19201999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f1924e00640 1 -- 172.21.15.119:0/1968999516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19201a15f0 con 0x7f1920101060 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.979+0000 7f191f400640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 0x7f1920199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f1924e00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f191800eb10 tx=0x7f191800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918004b20 con 0x7f1920101060 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1918004cc0 con 0x7f1920101060 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918010760 con 0x7f1920101060 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19201a18e0 con 0x7f1920101060 2024-03-28T18:43:50.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.980+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19201a1de0 con 0x7f1920101060 2024-03-28T18:43:50.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.981+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192010aab0 con 0x7f1920101060 2024-03-28T18:43:50.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.983+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1918005020 con 0x7f1920101060 2024-03-28T18:43:50.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.984+0000 7f191cc00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 0x7f19040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:50.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.985+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f19180cefe0 con 0x7f1920101060 2024-03-28T18:43:50.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.985+0000 7f191fe00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 0x7f19040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:50.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.985+0000 7f191fe00640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 0x7f19040a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f191002c240 tx=0x7f191002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:50.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:50.988+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f191809bfe0 con 0x7f1920101060 2024-03-28T18:43:51.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.214+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1920062340 con 0x7f1920101060 2024-03-28T18:43:51.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.216+0000 7f191cc00640 1 -- 172.21.15.119:0/1968999516 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7f191809b8e0 con 0x7f1920101060 2024-03-28T18:43:51.216 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:51.216 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:51.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 msgr2=0x7f19040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 0x7f19040a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f191002c240 tx=0x7f191002df90 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 msgr2=0x7f19201a0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f191800eb10 tx=0x7f191800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 shutdown_connections 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f19040a2880 0x7f19040a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1920101060 0x7f19201a0f80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f19201006c0 0x7f1920199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 --2- 172.21.15.119:0/1968999516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19200ffd60 0x7f19201999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.218+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 >> 172.21.15.119:0/1968999516 conn(0x7f19200fb980 msgr2=0x7f19200fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.219+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 shutdown_connections 2024-03-28T18:43:51.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.219+0000 7f1925800640 1 -- 172.21.15.119:0/1968999516 wait complete. 2024-03-28T18:43:51.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:43:51.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:43:51.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 -- 172.21.15.119:0/2151242929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac0fff20 msgr2=0x7f72ac102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 --2- 172.21.15.119:0/2151242929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac0fff20 0x7f72ac102350 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009900 tx=0x7f72a002bc40 comp rx=0 tx=0).stop 2024-03-28T18:43:51.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 -- 172.21.15.119:0/2151242929 shutdown_connections 2024-03-28T18:43:51.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 --2- 172.21.15.119:0/2151242929 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 --2- 172.21.15.119:0/2151242929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac102890 0x7f72ac104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 --2- 172.21.15.119:0/2151242929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac0fff20 0x7f72ac102350 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 -- 172.21.15.119:0/2151242929 >> 172.21.15.119:0/2151242929 conn(0x7f72ac0fbb40 msgr2=0x7f72ac0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 -- 172.21.15.119:0/2151242929 shutdown_connections 2024-03-28T18:43:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.444+0000 7f72b3e00640 1 -- 172.21.15.119:0/2151242929 wait complete. 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.445+0000 7f72b3e00640 1 Processor -- start 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.445+0000 7f72b3e00640 1 -- start start 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 0x7f72ac19e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac113970 con 0x7f72ac0fff20 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac1137f0 con 0x7f72ac105240 2024-03-28T18:43:51.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac113af0 con 0x7f72ac102890 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 0x7f72ac19e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36032/0 (socket says 172.21.15.119:36032) 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 -- 172.21.15.119:0/4146315839 learned_addr learned my addr 172.21.15.119:0/4146315839 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 -- 172.21.15.119:0/4146315839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 msgr2=0x7f72ac1a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 -- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 msgr2=0x7f72ac19e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 0x7f72ac19e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b0a00640 1 -- 172.21.15.119:0/4146315839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac1a5c30 con 0x7f72ac102890 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b1e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.446+0000 7f72b1400640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 0x7f72ac19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72b0a00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9f0 tx=0x7f729800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729800cd80 con 0x7f72ac102890 2024-03-28T18:43:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7298010910 con 0x7f72ac102890 2024-03-28T18:43:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac1a5f20 con 0x7f72ac102890 2024-03-28T18:43:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980188f0 con 0x7f72ac102890 2024-03-28T18:43:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.447+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac1a6430 con 0x7f72ac102890 2024-03-28T18:43:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.448+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac0fd630 con 0x7f72ac102890 2024-03-28T18:43:51.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.450+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7298010ac0 con 0x7f72ac102890 2024-03-28T18:43:51.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.452+0000 7f72aa000640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 0x7f72940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.452+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f72980cefb0 con 0x7f72ac102890 2024-03-28T18:43:51.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.452+0000 7f72b1400640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 0x7f72940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.453+0000 7f72b1400640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 0x7f72940a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f72a002c250 tx=0x7f72a002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.453+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f729809b920 con 0x7f72ac102890 2024-03-28T18:43:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:51 smithi119 ceph-mon[31813]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:51 smithi119 ceph-mon[31813]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:51 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1968999516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:51.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.682+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72ac02d110 con 0x7f72ac102890 2024-03-28T18:43:51.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.682+0000 7f72aa000640 1 -- 172.21.15.119:0/4146315839 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f729809f9c0 con 0x7f72ac102890 2024-03-28T18:43:51.682 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:43:51.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:51 smithi196 ceph-mon[36300]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:51 smithi196 ceph-mon[36300]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:51 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1968999516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:51.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.684+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 msgr2=0x7f72940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.684+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 0x7f72940a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f72a002c250 tx=0x7f72a002cf90 comp rx=0 tx=0).stop 2024-03-28T18:43:51.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.684+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 msgr2=0x7f72ac19e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.684+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9f0 tx=0x7f729800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 shutdown_connections 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f72940a2880 0x7f72940a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f72ac105240 0x7f72ac1a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f72ac102890 0x7f72ac19e550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 --2- 172.21.15.119:0/4146315839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72ac0fff20 0x7f72ac19e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 >> 172.21.15.119:0/4146315839 conn(0x7f72ac0fbb40 msgr2=0x7f72ac103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 shutdown_connections 2024-03-28T18:43:51.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.685+0000 7f72b3e00640 1 -- 172.21.15.119:0/4146315839 wait complete. 2024-03-28T18:43:51.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:51 smithi131 ceph-mon[36065]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:51 smithi131 ceph-mon[36065]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:51.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:51 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1968999516' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:51.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.894+0000 7fd764800640 1 -- 172.21.15.119:0/1535576535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c103280 msgr2=0x7fd75c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.894+0000 7fd764800640 1 --2- 172.21.15.119:0/1535576535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c103280 0x7fd75c109b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd758009ca0 tx=0x7fd75802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 -- 172.21.15.119:0/1535576535 shutdown_connections 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 --2- 172.21.15.119:0/1535576535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c103280 0x7fd75c109b10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 --2- 172.21.15.119:0/1535576535 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c1028c0 0x7fd75c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 --2- 172.21.15.119:0/1535576535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1016d0 0x7fd75c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 -- 172.21.15.119:0/1535576535 >> 172.21.15.119:0/1535576535 conn(0x7fd75c0fcc40 msgr2=0x7fd75c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 -- 172.21.15.119:0/1535576535 shutdown_connections 2024-03-28T18:43:51.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.895+0000 7fd764800640 1 -- 172.21.15.119:0/1535576535 wait complete. 2024-03-28T18:43:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 Processor -- start 2024-03-28T18:43:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 -- start start 2024-03-28T18:43:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1028c0 0x7fd75c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 0x7fd75c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c10d350 con 0x7fd75c1028c0 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c10d1d0 con 0x7fd75c103280 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd764800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75c10d4d0 con 0x7fd75c1016d0 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd761e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd761400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1028c0 0x7fd75c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd762800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 0x7fd75c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd761e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36046/0 (socket says 172.21.15.119:36046) 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.896+0000 7fd761e00640 1 -- 172.21.15.119:0/2088636312 learned_addr learned my addr 172.21.15.119:0/2088636312 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 -- 172.21.15.119:0/2088636312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 msgr2=0x7fd75c199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 -- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1028c0 msgr2=0x7fd75c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1028c0 0x7fd75c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 -- 172.21.15.119:0/2088636312 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c1a1870 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd761e00640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd762800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 0x7fd75c1a11d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd758009bf0 tx=0x7fd758032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758041020 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd758034cd0 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75c1a1b00 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758039040 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.897+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75c1a1fc0 con 0x7fd75c103280 2024-03-28T18:43:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.899+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75c1a2190 con 0x7fd75c103280 2024-03-28T18:43:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.900+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd758044020 con 0x7fd75c103280 2024-03-28T18:43:51.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.901+0000 7fd756a00640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 0x7fd7440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:51.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.901+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fd7580f0980 con 0x7fd75c103280 2024-03-28T18:43:51.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.902+0000 7fd761e00640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 0x7fd7440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:51.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.902+0000 7fd761e00640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 0x7fd7440a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd75c19b130 tx=0x7fd750005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:51.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:51.905+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd75804b020 con 0x7fd75c103280 2024-03-28T18:43:52.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.105+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd75c1a2360 con 0x7fd75c103280 2024-03-28T18:43:52.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.105+0000 7fd756a00640 1 -- 172.21.15.119:0/2088636312 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd7580bd940 con 0x7fd75c103280 2024-03-28T18:43:52.105 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 msgr2=0x7fd7440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 0x7fd7440a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd75c19b130 tx=0x7fd750005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 msgr2=0x7fd75c1a11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 0x7fd75c1a11d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd758009bf0 tx=0x7fd758032ea0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 shutdown_connections 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fd7440a2800 0x7fd7440a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd75c103280 0x7fd75c1a11d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd75c1028c0 0x7fd75c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 --2- 172.21.15.119:0/2088636312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd75c1016d0 0x7fd75c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 >> 172.21.15.119:0/2088636312 conn(0x7fd75c0fcc40 msgr2=0x7fd75c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:52.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 shutdown_connections 2024-03-28T18:43:52.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.108+0000 7fd764800640 1 -- 172.21.15.119:0/2088636312 wait complete. 2024-03-28T18:43:52.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:43:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 -- 172.21.15.119:0/986975134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac104ad0 msgr2=0x7f48ac104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 --2- 172.21.15.119:0/986975134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac104ad0 0x7f48ac104f50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4898009900 tx=0x7f489802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 -- 172.21.15.119:0/986975134 shutdown_connections 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 --2- 172.21.15.119:0/986975134 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 --2- 172.21.15.119:0/986975134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac104ad0 0x7f48ac104f50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 --2- 172.21.15.119:0/986975134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac1038e0 0x7f48ac103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 -- 172.21.15.119:0/986975134 >> 172.21.15.119:0/986975134 conn(0x7f48ac0fee50 msgr2=0x7f48ac101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 -- 172.21.15.119:0/986975134 shutdown_connections 2024-03-28T18:43:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.347+0000 7f48b1000640 1 -- 172.21.15.119:0/986975134 wait complete. 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.348+0000 7f48b1000640 1 Processor -- start 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.348+0000 7f48b1000640 1 -- start start 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac104ad0 0x7f48ac19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac10f590 con 0x7f48ac104ad0 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac10f410 con 0x7f48ac105490 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48b1000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac10f710 con 0x7f48ac1038e0 2024-03-28T18:43:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36062/0 (socket says 172.21.15.119:36062) 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48a9e00640 1 -- 172.21.15.119:0/1220850631 learned_addr learned my addr 172.21.15.119:0/1220850631 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48a9400640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac104ad0 0x7f48ac19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 -- 172.21.15.119:0/1220850631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 msgr2=0x7f48ac199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 -- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac104ad0 msgr2=0x7f48ac19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac104ad0 0x7f48ac19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48aa800640 1 -- 172.21.15.119:0/1220850631 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac1a1800 con 0x7f48ac105490 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.349+0000 7f48a9e00640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48aa800640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac1a1190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f48a400ec50 tx=0x7f48a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4003b70 con 0x7f48ac105490 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48a4003d10 con 0x7f48ac105490 2024-03-28T18:43:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4010890 con 0x7f48ac105490 2024-03-28T18:43:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac1a1af0 con 0x7f48ac105490 2024-03-28T18:43:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.350+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac1a1fc0 con 0x7f48ac105490 2024-03-28T18:43:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.351+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac10ac00 con 0x7f48ac105490 2024-03-28T18:43:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.353+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f48a4003fc0 con 0x7f48ac105490 2024-03-28T18:43:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.354+0000 7f48a2a00640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 0x7f48900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.355+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f48a40cf0f0 con 0x7f48ac105490 2024-03-28T18:43:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.355+0000 7f48a9e00640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 0x7f48900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.355+0000 7f48a9e00640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 0x7f48900a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f489c0058c0 tx=0x7f489c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:52.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.358+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f48a409c0b0 con 0x7f48ac105490 2024-03-28T18:43:52.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.564+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48ac0622a0 con 0x7f48ac105490 2024-03-28T18:43:52.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.564+0000 7f48a2a00640 1 -- 172.21.15.119:0/1220850631 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48a409b9b0 con 0x7f48ac105490 2024-03-28T18:43:52.564 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 msgr2=0x7f48900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 0x7f48900a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f489c0058c0 tx=0x7f489c005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 msgr2=0x7f48ac1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac1a1190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f48a400ec50 tx=0x7f48a400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 shutdown_connections 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f48900a2800 0x7f48900a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f48ac105490 0x7f48ac1a1190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ac104ad0 0x7f48ac19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 --2- 172.21.15.119:0/1220850631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48ac1038e0 0x7f48ac199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 >> 172.21.15.119:0/1220850631 conn(0x7f48ac0fee50 msgr2=0x7f48ac107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4146315839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2088636312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 shutdown_connections 2024-03-28T18:43:52.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.568+0000 7f48b1000640 1 -- 172.21.15.119:0/1220850631 wait complete. 2024-03-28T18:43:52.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:43:52.595 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:52.595 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:43:52.596 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4146315839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.596 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.596 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.596 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2088636312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T18:43:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T18:43:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4146315839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:52.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2088636312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:43:52.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 -- 172.21.15.119:0/2847049192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c076060 msgr2=0x7f454c076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 --2- 172.21.15.119:0/2847049192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c076060 0x7f454c076480 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f453c009900 tx=0x7f453c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:52.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 -- 172.21.15.119:0/2847049192 shutdown_connections 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 --2- 172.21.15.119:0/2847049192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 --2- 172.21.15.119:0/2847049192 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c0772a0 0x7f454c077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 --2- 172.21.15.119:0/2847049192 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c076060 0x7f454c076480 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 -- 172.21.15.119:0/2847049192 >> 172.21.15.119:0/2847049192 conn(0x7f454c0fee90 msgr2=0x7f454c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 -- 172.21.15.119:0/2847049192 shutdown_connections 2024-03-28T18:43:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.786+0000 7f4551400640 1 -- 172.21.15.119:0/2847049192 wait complete. 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.787+0000 7f4551400640 1 Processor -- start 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.787+0000 7f4551400640 1 -- start start 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.787+0000 7f4551400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4551400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 0x7f454c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4551400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c1a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4551400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c113970 con 0x7f454c077c40 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4551400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c1137f0 con 0x7f454c076060 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4551400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c113af0 con 0x7f454c0772a0 2024-03-28T18:43:52.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59094/0 (socket says 172.21.15.119:59094) 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 -- 172.21.15.119:0/3682097956 learned_addr learned my addr 172.21.15.119:0/3682097956 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4549e00640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 0x7f454c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454b200640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c1a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 -- 172.21.15.119:0/3682097956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 msgr2=0x7f454c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 0x7f454c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 -- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 msgr2=0x7f454c1a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c1a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 -- 172.21.15.119:0/3682097956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c1a5bf0 con 0x7f454c076060 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454b200640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c1a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f4549e00640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 0x7f454c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.788+0000 7f454a800640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f453c009230 tx=0x7f453c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.789+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c041070 con 0x7f454c076060 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.789+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f453c004450 con 0x7f454c076060 2024-03-28T18:43:52.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.789+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c003e20 con 0x7f454c076060 2024-03-28T18:43:52.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.789+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c1a5e80 con 0x7f454c076060 2024-03-28T18:43:52.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.789+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c1a62e0 con 0x7f454c076060 2024-03-28T18:43:52.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.790+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c110400 con 0x7f454c076060 2024-03-28T18:43:52.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.792+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f453c044070 con 0x7f454c076060 2024-03-28T18:43:52.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.793+0000 7f4543400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 0x7f45300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:52.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.793+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f453c0f0a80 con 0x7f454c076060 2024-03-28T18:43:52.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.794+0000 7f4549e00640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 0x7f45300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:52.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.794+0000 7f4549e00640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 0x7f45300a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f454c19f4e0 tx=0x7f4538005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:52.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:52.795+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f453c0bda40 con 0x7f454c076060 2024-03-28T18:43:53.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.040+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f454c02d190 con 0x7f454c076060 2024-03-28T18:43:53.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.041+0000 7f4543400640 1 -- 172.21.15.119:0/3682097956 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f453c0bd340 con 0x7f454c076060 2024-03-28T18:43:53.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:53.041 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:43:53.042 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.043+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 msgr2=0x7f45300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.043+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 0x7f45300a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f454c19f4e0 tx=0x7f4538005f90 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.043+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 msgr2=0x7f454c19dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.043+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f453c009230 tx=0x7f453c02c650 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 shutdown_connections 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f45300a2880 0x7f45300a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f454c077c40 0x7f454c1a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f454c0772a0 0x7f454c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 --2- 172.21.15.119:0/3682097956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f454c076060 0x7f454c19dfd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 >> 172.21.15.119:0/3682097956 conn(0x7f454c0fee90 msgr2=0x7f454c1005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 shutdown_connections 2024-03-28T18:43:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.044+0000 7f4551400640 1 -- 172.21.15.119:0/3682097956 wait complete. 2024-03-28T18:43:53.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:43:53.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 -- 172.21.15.119:0/777229468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa1081038e0 msgr2=0x7fa108103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 --2- 172.21.15.119:0/777229468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa1081038e0 0x7fa108103d00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa100009ca0 tx=0x7fa10002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 -- 172.21.15.119:0/777229468 shutdown_connections 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 --2- 172.21.15.119:0/777229468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108105490 0x7fa10810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 --2- 172.21.15.119:0/777229468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108104ad0 0x7fa108104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 --2- 172.21.15.119:0/777229468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa1081038e0 0x7fa108103d00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 -- 172.21.15.119:0/777229468 >> 172.21.15.119:0/777229468 conn(0x7fa1080fee50 msgr2=0x7fa108101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 -- 172.21.15.119:0/777229468 shutdown_connections 2024-03-28T18:43:53.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.249+0000 7fa10d800640 1 -- 172.21.15.119:0/777229468 wait complete. 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.250+0000 7fa10d800640 1 Processor -- start 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.250+0000 7fa10d800640 1 -- start start 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 0x7fa108199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10810f530 con 0x7fa1081038e0 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10810f3b0 con 0x7fa108105490 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa10d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10810f6b0 con 0x7fa108104ad0 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa107400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 0x7fa108199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa107400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59116/0 (socket says 172.21.15.119:59116) 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa107400640 1 -- 172.21.15.119:0/1236340469 learned_addr learned my addr 172.21.15.119:0/1236340469 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36110/0 (socket says 172.21.15.119:36110) 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 -- 172.21.15.119:0/1236340469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 msgr2=0x7fa1081a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 -- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 msgr2=0x7fa108199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 0x7fa108199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106000640 1 -- 172.21.15.119:0/1236340469 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1081a1850 con 0x7fa108104ad0 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa107400640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.251+0000 7fa106a00640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 0x7fa108199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa106000640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d6d0 tx=0x7fa0f000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0017070 con 0x7fa108104ad0 2024-03-28T18:43:53.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1081a1b40 con 0x7fa108104ad0 2024-03-28T18:43:53.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f0009820 con 0x7fa108104ad0 2024-03-28T18:43:53.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0004cd0 con 0x7fa108104ad0 2024-03-28T18:43:53.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.252+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1081a2090 con 0x7fa108104ad0 2024-03-28T18:43:53.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.253+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10810ad20 con 0x7fa108104ad0 2024-03-28T18:43:53.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.255+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa0f00099d0 con 0x7fa108104ad0 2024-03-28T18:43:53.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.256+0000 7fa0ff400640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.257+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fa0f00ce640 con 0x7fa108104ad0 2024-03-28T18:43:53.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.257+0000 7fa106a00640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.257+0000 7fa106a00640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa10819b110 tx=0x7fa100032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.263+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa0f009b010 con 0x7fa108104ad0 2024-03-28T18:43:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1220850631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:53 smithi119 ceph-mon[31813]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:53.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3682097956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:53.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.465+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1080622a0 con 0x7fa108104ad0 2024-03-28T18:43:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.467+0000 7fa0ff400640 1 -- 172.21.15.119:0/1236340469 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22962 (secure 0 0 0) 0x7fa0f009f0b0 con 0x7fa108104ad0 2024-03-28T18:43:53.468 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:43:53.468 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":90,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:42:31.162785+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 msgr2=0x7fa0ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa10819b110 tx=0x7fa100032590 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 msgr2=0x7fa10819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000d6d0 tx=0x7fa0f000db90 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 shutdown_connections 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa108105490 0x7fa1081a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa108104ad0 0x7fa10819a180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.470+0000 7fa10d800640 1 --2- 172.21.15.119:0/1236340469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1081038e0 0x7fa108199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.471+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 >> 172.21.15.119:0/1236340469 conn(0x7fa1080fee50 msgr2=0x7fa108107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.471+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 shutdown_connections 2024-03-28T18:43:53.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.471+0000 7fa10d800640 1 -- 172.21.15.119:0/1236340469 wait complete. 2024-03-28T18:43:53.490 DEBUG:tasks.fwd_scrub:Yielding 2024-03-28T18:43:53.490 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T18:43:53.500 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:43:53.500 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:43:53.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T18:43:53.501 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-28T18:43:53.501 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:43:53.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:43:53.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:43:53.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 -- 172.21.15.119:0/4147774510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434102700 msgr2=0x7fc434102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4147774510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434102700 0x7fc434102b80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc420009900 tx=0x7fc42002bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 -- 172.21.15.119:0/4147774510 shutdown_connections 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4147774510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 0x7fc434109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4147774510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434102700 0x7fc434102b80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4147774510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434101510 0x7fc434101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 -- 172.21.15.119:0/4147774510 >> 172.21.15.119:0/4147774510 conn(0x7fc4340fca80 msgr2=0x7fc4340feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 -- 172.21.15.119:0/4147774510 shutdown_connections 2024-03-28T18:43:53.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.684+0000 7fc43ae00640 1 -- 172.21.15.119:0/4147774510 wait complete. 2024-03-28T18:43:53.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1220850631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:53.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:53.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:53.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:53 smithi196 ceph-mon[36300]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:53.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3682097956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.685+0000 7fc43ae00640 1 Processor -- start 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.685+0000 7fc43ae00640 1 -- start start 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 0x7fc434199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 0x7fc4341a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d1f0 con 0x7fc434102700 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d070 con 0x7fc434101510 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d370 con 0x7fc4341030c0 2024-03-28T18:43:53.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc439a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc439000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 0x7fc434199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc439a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59140/0 (socket says 172.21.15.119:59140) 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc439a00640 1 -- 172.21.15.119:0/4195393209 learned_addr learned my addr 172.21.15.119:0/4195393209 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.686+0000 7fc43a400640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 0x7fc4341a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 -- 172.21.15.119:0/4195393209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 msgr2=0x7fc4341a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 0x7fc4341a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 -- 172.21.15.119:0/4195393209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 msgr2=0x7fc4341999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 -- 172.21.15.119:0/4195393209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4341a1610 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439a00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc439000640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 0x7fc434199f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc420036040 tx=0x7fc42002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420043020 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4200054e0 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420003be0 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.687+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4341a18a0 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.688+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4341a1e10 con 0x7fc434102700 2024-03-28T18:43:53.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.689+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4341a1fe0 con 0x7fc434102700 2024-03-28T18:43:53.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.693+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc420046020 con 0x7fc434102700 2024-03-28T18:43:53.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1220850631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:43:53.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:43:53.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:43:53.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:53 smithi131 ceph-mon[36065]: from='mgr.14152 172.21.15.119:0/3879142519' entity='mgr.x' 2024-03-28T18:43:53.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3682097956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:43:53.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.695+0000 7fc432000640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 0x7fc4180a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.695+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fc4200f1090 con 0x7fc434102700 2024-03-28T18:43:53.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.695+0000 7fc439a00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 0x7fc4180a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 -- 172.21.15.119:0/4078847213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668102130 msgr2=0x7fe668104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 --2- 172.21.15.119:0/4078847213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668102130 0x7fe668104520 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe658009900 tx=0x7fe65802bc60 comp rx=0 tx=0).stop 2024-03-28T18:43:53.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 -- 172.21.15.119:0/4078847213 shutdown_connections 2024-03-28T18:43:53.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 --2- 172.21.15.119:0/4078847213 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668107390 0x7fe6681097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 --2- 172.21.15.119:0/4078847213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668104a60 0x7fe668106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 --2- 172.21.15.119:0/4078847213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668102130 0x7fe668104520 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 -- 172.21.15.119:0/4078847213 >> 172.21.15.119:0/4078847213 conn(0x7fe6680fbb40 msgr2=0x7fe6680fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 -- 172.21.15.119:0/4078847213 shutdown_connections 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fe66d200640 1 -- 172.21.15.119:0/4078847213 wait complete. 2024-03-28T18:43:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.696+0000 7fc439a00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 0x7fc4180a4d30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc424011750 tx=0x7fc4240096d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.697+0000 7fe66d200640 1 Processor -- start 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.697+0000 7fe66d200640 1 -- start start 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 0x7fe66819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668107390 0x7fe6681a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810d3a0 con 0x7fe668102130 2024-03-28T18:43:53.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810d220 con 0x7fe668104a60 2024-03-28T18:43:53.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe66d200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66810d520 con 0x7fe668107390 2024-03-28T18:43:53.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc42004d020 con 0x7fc434102700 2024-03-28T18:43:53.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 0x7fe66819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38922/0 (socket says 172.21.15.119:38922) 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 -- 172.21.15.119:0/636433312 learned_addr learned my addr 172.21.15.119:0/636433312 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe666800640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668107390 0x7fe6681a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 -- 172.21.15.119:0/636433312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668107390 msgr2=0x7fe6681a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668107390 0x7fe6681a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 -- 172.21.15.119:0/636433312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 msgr2=0x7fe66819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 0x7fe66819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665e00640 1 -- 172.21.15.119:0/636433312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6681a1810 con 0x7fe668102130 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.698+0000 7fe665400640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 0x7fe66819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe665e00640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe65802c270 tx=0x7fe658004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe65ea00640 1 -- 172.21.15.119:0/636433312 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658043020 con 0x7fe668102130 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6681a1aa0 con 0x7fe668102130 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe65ea00640 1 -- 172.21.15.119:0/636433312 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6580043d0 con 0x7fe668102130 2024-03-28T18:43:53.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe65ea00640 1 -- 172.21.15.119:0/636433312 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe658003d30 con 0x7fe668102130 2024-03-28T18:43:53.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.699+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6681a1ff0 con 0x7fe668102130 2024-03-28T18:43:53.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.703+0000 7fe66fe82740 1 Processor -- start 2024-03-28T18:43:53.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.703+0000 7fe66fe82740 1 -- start start 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502bf7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bfa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502b57eb0 con 0x55d502bf7c00 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502b00a60 con 0x55d502bf20a0 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe66fe82740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502b53f50 con 0x55d502bf2a90 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59164/0 (socket says 172.21.15.119:59164) 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 -- 172.21.15.119:0/2390919219 learned_addr learned my addr 172.21.15.119:0/2390919219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe665400640 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bfa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 -- 172.21.15.119:0/2390919219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 msgr2=0x55d502bf7590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:43:53.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502bf7590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 -- 172.21.15.119:0/2390919219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 msgr2=0x55d502bfa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bfa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.704+0000 7fe666800640 1 -- 172.21.15.119:0/2390919219 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5029347e0 con 0x55d502bf20a0 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.705+0000 7fe65ea00640 1 -- 172.21.15.119:0/636433312 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe658046020 con 0x7fe668102130 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.705+0000 7fe666800640 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe660009fd0 tx=0x7fe6600118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbfc217a5c062aed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.705+0000 7fe653400640 1 -- 172.21.15.119:0/2390919219 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660004d20 con 0x55d502bf20a0 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.705+0000 7fe653400640 1 -- 172.21.15.119:0/2390919219 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe660004050 con 0x55d502bf20a0 2024-03-28T18:43:53.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.705+0000 7fe653400640 1 -- 172.21.15.119:0/2390919219 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660005260 con 0x55d502bf20a0 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- 172.21.15.119:0/2390919219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 msgr2=0x55d502bf24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe660009fd0 tx=0x7fe6600118d0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- 172.21.15.119:0/2390919219 shutdown_connections 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bfa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502bf7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2390919219 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf24c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- 172.21.15.119:0/2390919219 >> 172.21.15.119:0/2390919219 conn(0x55d502bed610 msgr2=0x55d502beda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- 172.21.15.119:0/2390919219 shutdown_connections 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- 172.21.15.119:0/2390919219 wait complete. 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.706+0000 7fe66fe82740 1 -- start start 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 Processor -- start 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502bfc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502bfc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502bfabb0 con 0x55d502bf7c00 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502b13eb0 con 0x55d502bf2a90 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d502bfad30 con 0x55d502bf20a0 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe65ea00640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 0x7fe64c0a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.707+0000 7fe66fe82740 10 client.0 _ll_register_callbacks cb 0x55d502b08150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe65ea00640 1 -- 172.21.15.119:0/636433312 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe6580f0cb0 con 0x7fe668102130 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38952/0 (socket says 172.21.15.119:38952) 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 -- 172.21.15.119:0/1080471327 learned_addr learned my addr 172.21.15.119:0/1080471327 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe666800640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502bfc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 -- 172.21.15.119:0/1080471327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 msgr2=0x55d502bfc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502bfc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665e00640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502bfc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 -- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 msgr2=0x55d502bfc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502bfc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 -- 172.21.15.119:0/1080471327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d502994f70 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe666800640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 0x7fe64c0a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe665400640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe654011760 tx=0x7fe654011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b473ac04da790ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.708+0000 7fe647e00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654010c00 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe660004050 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe654010da0 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe654004020 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe666800640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 0x7fe64c0a4c50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe660009de0 tx=0x7fe6600118d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe66fe82740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d50282c170 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.709+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe660004050 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe654004960 con 0x55d502bf7c00 2024-03-28T18:43:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 10 client.15273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 20 client.15273 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c081f0 0x55d502c0a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 4 client.15273 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d502b653f0 con 0x55d502c081f0 2024-03-28T18:43:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.710+0000 7fe665e00640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c081f0 0x55d502c0a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.711+0000 7fe665e00640 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c081f0 0x55d502c0a610 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ddd7061f2860935 server_cookie=21f37a7e6810888b in_seq=0 out_seq=0 2024-03-28T18:43:53.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.711+0000 7fe647e00640 10 client.15273 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.711+0000 7fe647e00640 1 -- 172.21.15.119:0/1080471327 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d502b653f0 con 0x55d502c081f0 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.711+0000 7fe647e00640 10 client.15273 handle_command_reply: tid=0 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.711+0000 7fe66fe82740 1 client.15273 shutdown 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 msgr2=0x55d502c03860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe654011760 tx=0x7fe654011c20 comp rx=0 tx=0).stop 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 shutdown_connections 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c081f0 0x55d502c0a610 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c03860 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502bfc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1080471327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502bfc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 >> 172.21.15.119:0/1080471327 conn(0x55d502bed610 msgr2=0x55d502bec830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe647e00640 0 client.15273 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 shutdown_connections 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 1 -- 172.21.15.119:0/1080471327 wait complete. 2024-03-28T18:43:53.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.712+0000 7fe66fe82740 20 client.15273 trim_cache size 0 max 16384 2024-03-28T18:43:53.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.729+0000 7fe66fe82740 1 Processor -- start 2024-03-28T18:43:53.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.729+0000 7fe66fe82740 1 -- start start 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bf3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681aad70 con 0x55d502bf7c00 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681ac6f0 con 0x55d502bf20a0 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe66fe82740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681ab3d0 con 0x55d502bf2a90 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe665400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe665e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe666800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bf3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe665400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36156/0 (socket says 172.21.15.119:36156) 2024-03-28T18:43:53.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe665e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59188/0 (socket says 172.21.15.119:59188) 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.730+0000 7fe665400640 1 -- 172.21.15.119:0/2920324811 learned_addr learned my addr 172.21.15.119:0/2920324811 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 -- 172.21.15.119:0/2920324811 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 msgr2=0x55d502bf3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 -- 172.21.15.119:0/2920324811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 msgr2=0x55d502bf3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bf3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 -- 172.21.15.119:0/2920324811 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d502be8c20 con 0x55d502bf2a90 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665e00640 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe665400640 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe654000c00 tx=0x7fe65400b4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26970883f104ad0a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe652a00640 1 -- 172.21.15.119:0/2920324811 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654032070 con 0x55d502bf2a90 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe652a00640 1 -- 172.21.15.119:0/2920324811 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe654029040 con 0x55d502bf2a90 2024-03-28T18:43:53.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.731+0000 7fe652a00640 1 -- 172.21.15.119:0/2920324811 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654028070 con 0x55d502bf2a90 2024-03-28T18:43:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- 172.21.15.119:0/2920324811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 msgr2=0x55d502c03780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe654000c00 tx=0x7fe65400b4a0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- 172.21.15.119:0/2920324811 shutdown_connections 2024-03-28T18:43:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502bf3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf2a90 0x55d502c03780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 --2- 172.21.15.119:0/2920324811 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf20a0 0x55d502bf3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- 172.21.15.119:0/2920324811 >> 172.21.15.119:0/2920324811 conn(0x55d502bed610 msgr2=0x55d502bf1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- 172.21.15.119:0/2920324811 shutdown_connections 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- 172.21.15.119:0/2920324811 wait complete. 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.732+0000 7fe66fe82740 1 -- start start 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 Processor -- start 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502c025a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c02ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681ac090 con 0x55d502bf7c00 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681aba30 con 0x55d502bf2a90 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6681a8f30 con 0x55d502bf20a0 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe66fe82740 10 client.0 _ll_register_callbacks cb 0x7fe65400bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe665e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe665e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36168/0 (socket says 172.21.15.119:36168) 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe665e00640 1 -- 172.21.15.119:0/1374645866 learned_addr learned my addr 172.21.15.119:0/1374645866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe665400640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502c025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.733+0000 7fe666800640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c02ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 -- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 msgr2=0x55d502c025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502c025a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 -- 172.21.15.119:0/1374645866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 msgr2=0x55d502c02ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c02ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 -- 172.21.15.119:0/1374645866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d502c63bd0 con 0x55d502bf20a0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe666800640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c02ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe665e00640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6580b99d0 tx=0x7fe6580a8040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26cd725b58e84a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe647e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6580a6070 con 0x55d502bf20a0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fe6580bb870 con 0x55d502bf20a0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe654029040 con 0x55d502bf20a0 2024-03-28T18:43:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6580d6020 con 0x55d502bf20a0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe66fe82740 10 client.25411 fetch_fsmap learned FSMap version 28 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.734+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d502c64fe0 con 0x55d502bf20a0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe654029040 con 0x55d502bf20a0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe6580b9be0 con 0x55d502bf20a0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 10 client.25411 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 10 client.25411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 20 client.25411 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c65960 0x55d502c67d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 4 client.25411 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d502b5f740 con 0x55d502c65960 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.735+0000 7fe665400640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c65960 0x55d502c67d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.736+0000 7fe665400640 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c65960 0x55d502c67d80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad104ca95e15bf8 server_cookie=ca9a866ca377c12c in_seq=0 out_seq=0 2024-03-28T18:43:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.736+0000 7fe647e00640 10 client.25411 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe647e00640 1 -- 172.21.15.119:0/1374645866 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d502b5f740 con 0x55d502c65960 2024-03-28T18:43:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe647e00640 10 client.25411 handle_command_reply: tid=0 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 client.25411 shutdown 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 msgr2=0x55d502c02060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe6580b99d0 tx=0x7fe6580a8040 comp rx=0 tx=0).stop 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 shutdown_connections 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d502c65960 0x55d502c67d80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d502bf7c00 0x55d502c02ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d502bf2a90 0x55d502c025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 --2- 172.21.15.119:0/1374645866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d502bf20a0 0x55d502c02060 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe647e00640 0 client.25411 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.737+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 >> 172.21.15.119:0/1374645866 conn(0x55d502bed610 msgr2=0x55d502bebfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 shutdown_connections 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66fe82740 1 -- 172.21.15.119:0/1374645866 wait complete. 2024-03-28T18:43:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66fe82740 20 client.25411 trim_cache size 0 max 16384 2024-03-28T18:43:53.741 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:53.741 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:43:53.741 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "1d7fc57e-342c-4373-bc08-a7feb6a2bbd3", 2024-03-28T18:43:53.741 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:43:53.741 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 msgr2=0x7fe64c0a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 0x7fe64c0a4c50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe660009de0 tx=0x7fe6600118d0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 msgr2=0x7fe668199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe65802c270 tx=0x7fe658004eb0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 shutdown_connections 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fe64c0a2790 0x7fe64c0a4c50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe668107390 0x7fe6681a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe668104a60 0x7fe66819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 --2- 172.21.15.119:0/636433312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe668102130 0x7fe668199bf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.738+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 >> 172.21.15.119:0/636433312 conn(0x7fe6680fbb40 msgr2=0x7fe668101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.739+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 shutdown_connections 2024-03-28T18:43:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.739+0000 7fe66d200640 1 -- 172.21.15.119:0/636433312 wait complete. 2024-03-28T18:43:53.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:43:53.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:43:53.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 -- 172.21.15.119:0/4191243814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141028a0 msgr2=0x7f2014102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 --2- 172.21.15.119:0/4191243814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141028a0 0x7f2014102d20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009900 tx=0x7f1ffc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 -- 172.21.15.119:0/4191243814 shutdown_connections 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 --2- 172.21.15.119:0/4191243814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f2014109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 --2- 172.21.15.119:0/4191243814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141028a0 0x7f2014102d20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 --2- 172.21.15.119:0/4191243814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141016b0 0x7f2014101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.955+0000 7f201b200640 1 -- 172.21.15.119:0/4191243814 >> 172.21.15.119:0/4191243814 conn(0x7f20140fcc40 msgr2=0x7f20140ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.956+0000 7f201b200640 1 -- 172.21.15.119:0/4191243814 shutdown_connections 2024-03-28T18:43:53.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.956+0000 7f201b200640 1 -- 172.21.15.119:0/4191243814 wait complete. 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 Processor -- start 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 -- start start 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141028a0 0x7f2014197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f201419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410d360 con 0x7f20141028a0 2024-03-28T18:43:53.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410d1e0 con 0x7f20141016b0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f201b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410d4e0 con 0x7f2014103260 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f2013e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f2013e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59232/0 (socket says 172.21.15.119:59232) 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f2013e00640 1 -- 172.21.15.119:0/303504043 learned_addr learned my addr 172.21.15.119:0/303504043 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f2019200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f201419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.957+0000 7f200b400640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141028a0 0x7f2014197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 -- 172.21.15.119:0/303504043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 msgr2=0x7f201419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f201419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 -- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141028a0 msgr2=0x7f2014197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141028a0 0x7f2014197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 -- 172.21.15.119:0/303504043 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201419f620 con 0x7f20141016b0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2019200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f201419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2013e00640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f200400e9c0 tx=0x7f200400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2011600640 1 -- 172.21.15.119:0/303504043 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400cd80 con 0x7f20141016b0 2024-03-28T18:43:53.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2011600640 1 -- 172.21.15.119:0/303504043 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20040035f0 con 0x7f20141016b0 2024-03-28T18:43:53.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201419f910 con 0x7f20141016b0 2024-03-28T18:43:53.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f2011600640 1 -- 172.21.15.119:0/303504043 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004010690 con 0x7f20141016b0 2024-03-28T18:43:53.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.958+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201419fe30 con 0x7f20141016b0 2024-03-28T18:43:53.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.960+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc4341a2320 con 0x7fc434102700 2024-03-28T18:43:53.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.961+0000 7f2011600640 1 -- 172.21.15.119:0/303504043 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f20040037a0 con 0x7f20141016b0 2024-03-28T18:43:53.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 Processor -- start 2024-03-28T18:43:53.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 -- start start 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c64f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c652280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c5ab250 con 0x56421c64ac80 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c553a30 con 0x56421c64fdf0 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f201dfc7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c5a72f0 con 0x56421c64a290 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f2019200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f2019200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36196/0 (socket says 172.21.15.119:36196) 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f2019200640 1 -- 172.21.15.119:0/510702103 learned_addr learned my addr 172.21.15.119:0/510702103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f2013e00640 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c64f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.962+0000 7f200b400640 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c652280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 -- 172.21.15.119:0/510702103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 msgr2=0x56421c652280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c652280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 -- 172.21.15.119:0/510702103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 msgr2=0x56421c64f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c64f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 -- 172.21.15.119:0/510702103 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56421c53e690 con 0x56421c64a290 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2011600640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 0x7f1ff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2019200640 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f200c0117e0 tx=0x7f200c011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ff4ae6551f9b274 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2011600640 1 -- 172.21.15.119:0/303504043 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f20040ceef0 con 0x7f20141016b0 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2013e00640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 0x7f1ff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2009600640 1 -- 172.21.15.119:0/510702103 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c011310 con 0x56421c64a290 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2009600640 1 -- 172.21.15.119:0/510702103 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f200c003e50 con 0x56421c64a290 2024-03-28T18:43:53.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.963+0000 7f2009600640 1 -- 172.21.15.119:0/510702103 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c004620 con 0x56421c64a290 2024-03-28T18:43:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 -- 172.21.15.119:0/510702103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 msgr2=0x56421c64a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f2013e00640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 0x7f1ff80a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2004097c10 tx=0x7f20040d2930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f200c0117e0 tx=0x7f200c011ca0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 -- 172.21.15.119:0/510702103 shutdown_connections 2024-03-28T18:43:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c652280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c64f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 --2- 172.21.15.119:0/510702103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64a290 0x56421c64a6b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 -- 172.21.15.119:0/510702103 >> 172.21.15.119:0/510702103 conn(0x56421c60e200 msgr2=0x56421c6417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 -- 172.21.15.119:0/510702103 shutdown_connections 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.964+0000 7f201dfc7740 1 -- 172.21.15.119:0/510702103 wait complete. 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 -- start start 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 Processor -- start 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c654230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c652da0 con 0x56421c64a290 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c566e80 con 0x56421c64fdf0 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.965+0000 7f201dfc7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56421c652f20 con 0x56421c64ac80 2024-03-28T18:43:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f201dfc7740 10 client.0 _ll_register_callbacks cb 0x56421c579130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f2019200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c654230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f2019200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c654230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39000/0 (socket says 172.21.15.119:39000) 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f2019200640 1 -- 172.21.15.119:0/1179750457 learned_addr learned my addr 172.21.15.119:0/1179750457 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36208/0 (socket says 172.21.15.119:36208) 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f2019200640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 -- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 msgr2=0x56421c65b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 -- 172.21.15.119:0/1179750457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 msgr2=0x56421c654230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c654230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 -- 172.21.15.119:0/1179750457 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56421c3897d0 con 0x56421c64ac80 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f200b400640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc036040 tx=0x7f1ffc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd73ffd79f764bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.966+0000 7f2002000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc005d50 con 0x56421c64ac80 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f200c003e50 con 0x56421c64ac80 2024-03-28T18:43:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f1ffc0046c0 con 0x56421c64ac80 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ffc050020 con 0x56421c64ac80 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f201dfc7740 10 client.25423 fetch_fsmap learned FSMap version 28 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56421c2bf3f0 con 0x56421c64ac80 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.967+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f200c003e50 con 0x56421c64ac80 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1ffc0035f0 con 0x56421c64ac80 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 10 client.25423 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 10 client.25423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 20 client.25423 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c65ff50 0x56421c6623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 4 client.25423 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56421c5b8790 con 0x56421c65ff50 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.968+0000 7f2019200640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c65ff50 0x56421c6623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f2019200640 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c65ff50 0x56421c6623b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e09d72c5ff4237c5 server_cookie=5389578c6d819066 in_seq=0 out_seq=0 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f2002000640 10 client.25423 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f2002000640 1 -- 172.21.15.119:0/1179750457 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56421c5b8790 con 0x56421c65ff50 2024-03-28T18:43:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f2002000640 10 client.25423 handle_command_reply: tid=0 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f201dfc7740 1 client.25423 shutdown 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 msgr2=0x56421c654770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.969+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc036040 tx=0x7f1ffc004270 comp rx=0 tx=0).stop 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 shutdown_connections 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c65ff50 0x56421c6623b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c654770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1179750457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c654230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f2002000640 0 client.25423 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 >> 172.21.15.119:0/1179750457 conn(0x56421c60e200 msgr2=0x56421c640a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 shutdown_connections 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 1 -- 172.21.15.119:0/1179750457 wait complete. 2024-03-28T18:43:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.970+0000 7f201dfc7740 20 client.25423 trim_cache size 0 max 16384 2024-03-28T18:43:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.987+0000 7f201dfc7740 1 Processor -- start 2024-03-28T18:43:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.987+0000 7f201dfc7740 1 -- start start 2024-03-28T18:43:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c65b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64fdf0 0x56421c640740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a69f0 con 0x56421c64ac80 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a8370 con 0x56421c64a290 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f201dfc7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a7050 con 0x56421c64fdf0 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59264/0 (socket says 172.21.15.119:59264) 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 -- 172.21.15.119:0/1321066393 learned_addr learned my addr 172.21.15.119:0/1321066393 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64fdf0 0x56421c640740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f200b400640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c65b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 -- 172.21.15.119:0/1321066393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64fdf0 msgr2=0x56421c640740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64fdf0 0x56421c640740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 -- 172.21.15.119:0/1321066393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 msgr2=0x56421c65b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c65b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f2019200640 1 -- 172.21.15.119:0/1321066393 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56421c27f0f0 con 0x56421c64a290 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.988+0000 7f200b400640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c65b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.989+0000 7f2019200640 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f200c00a010 tx=0x7f200c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e076ebafbd01382 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.989+0000 7f2008c00640 1 -- 172.21.15.119:0/1321066393 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c023040 con 0x56421c64a290 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.989+0000 7f2008c00640 1 -- 172.21.15.119:0/1321066393 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f200c00f9a0 con 0x56421c64a290 2024-03-28T18:43:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.989+0000 7f2008c00640 1 -- 172.21.15.119:0/1321066393 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c02fbe0 con 0x56421c64a290 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- 172.21.15.119:0/1321066393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 msgr2=0x56421c640200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f200c00a010 tx=0x7f200c005bd0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- 172.21.15.119:0/1321066393 shutdown_connections 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64fdf0 0x56421c640740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64ac80 0x56421c65b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 --2- 172.21.15.119:0/1321066393 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64a290 0x56421c640200 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- 172.21.15.119:0/1321066393 >> 172.21.15.119:0/1321066393 conn(0x56421c60e200 msgr2=0x56421c63c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- 172.21.15.119:0/1321066393 shutdown_connections 2024-03-28T18:43:53.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- 172.21.15.119:0/1321066393 wait complete. 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 -- start start 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.990+0000 7f201dfc7740 1 Processor -- start 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c658c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a7d10 con 0x56421c64a290 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a76b0 con 0x56421c64fdf0 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a4bb0 con 0x56421c64ac80 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f201dfc7740 10 client.0 _ll_register_callbacks cb 0x7f1ffc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f2019200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c658c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36222/0 (socket says 172.21.15.119:36222) 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f2019200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c658c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39012/0 (socket says 172.21.15.119:39012) 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 -- 172.21.15.119:0/2030061127 learned_addr learned my addr 172.21.15.119:0/2030061127 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:53.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f2019200640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 -- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 msgr2=0x56421c65a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 -- 172.21.15.119:0/2030061127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 msgr2=0x56421c658c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c658c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.991+0000 7f200b400640 1 -- 172.21.15.119:0/2030061127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56421c65a750 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f200b400640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc000c00 tx=0x7f1ffc009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c324c113b77022cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc005d50 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f200c00f9a0 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f1ffc004af0 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ffc003bb0 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f201dfc7740 10 client.25429 fetch_fsmap learned FSMap version 28 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56421c65bad0 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f200c00f9a0 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.992+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1ffc04b480 con 0x56421c64ac80 2024-03-28T18:43:53.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 10 client.25429 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 10 client.25429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 20 client.25429 populate_metadata read hostname 'smithi119' 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c6bbfb0 0x56421c6be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 4 client.25429 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56421c5b2ae0 con 0x56421c6bbfb0 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f2019200640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c6bbfb0 0x56421c6be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f2019200640 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c6bbfb0 0x56421c6be3d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab12695c5f3f86e8 server_cookie=9c3b569a867a83f5 in_seq=0 out_seq=0 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f2002000640 10 client.25429 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f2002000640 1 -- 172.21.15.119:0/2030061127 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56421c5b2ae0 con 0x56421c6bbfb0 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f2002000640 10 client.25429 handle_command_reply: tid=0 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.993+0000 7f201dfc7740 1 client.25429 shutdown 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 msgr2=0x56421c659150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc000c00 tx=0x7f1ffc009300 comp rx=0 tx=0).stop 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 shutdown_connections 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56421c6bbfb0 0x56421c6be3d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56421c64fdf0 0x56421c65a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56421c64ac80 0x56421c659150 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f2002000640 0 client.25429 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 --2- 172.21.15.119:0/2030061127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56421c64a290 0x56421c658c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 >> 172.21.15.119:0/2030061127 conn(0x56421c60e200 msgr2=0x56421c64b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 shutdown_connections 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 1 -- 172.21.15.119:0/2030061127 wait complete. 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.994+0000 7f201dfc7740 20 client.25429 trim_cache size 0 max 16384 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:43:53.998 INFO:teuthology.orchestra.run.smithi119.stdout: "1d7fc57e-342c-4373-bc08-a7feb6a2bbd3": { 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "1d7fc57e-342c-4373-bc08-a7feb6a2bbd3", 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:43:53.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 msgr2=0x7f1ff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 0x7f1ff80a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2004097c10 tx=0x7f20040d2930 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 msgr2=0x7f2014197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f200400e9c0 tx=0x7f200400ee80 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 shutdown_connections 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f1ff80a2800 0x7f1ff80a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2014103260 0x7f201419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20141028a0 0x7f2014197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 --2- 172.21.15.119:0/303504043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f20141016b0 0x7f2014197a00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 >> 172.21.15.119:0/303504043 conn(0x7f20140fcc40 msgr2=0x7f2014105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 shutdown_connections 2024-03-28T18:43:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:53.995+0000 7f201b200640 1 -- 172.21.15.119:0/303504043 wait complete. 2024-03-28T18:43:54.013 INFO:tasks.cephfs.filesystem:scrub status for tag:1d7fc57e-342c-4373-bc08-a7feb6a2bbd3 - {'path': '/', 'tag': '1d7fc57e-342c-4373-bc08-a7feb6a2bbd3', 'options': 'recursive,force'} 2024-03-28T18:43:54.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.272+0000 7fc432000640 1 -- 172.21.15.119:0/4195393209 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc4200be0b0 con 0x7fc434102700 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.276+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 msgr2=0x7fc4180a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.276+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 0x7fc4180a4d30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc424011750 tx=0x7fc4240096d0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.276+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 msgr2=0x7fc434199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.276+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 0x7fc434199f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc420036040 tx=0x7fc42002c6c0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 shutdown_connections 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fc4180a2870 0x7fc4180a4d30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4341030c0 0x7fc4341a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc434102700 0x7fc434199f30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 --2- 172.21.15.119:0/4195393209 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc434101510 0x7fc4341999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 >> 172.21.15.119:0/4195393209 conn(0x7fc4340fca80 msgr2=0x7fc4340738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 shutdown_connections 2024-03-28T18:43:54.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.277+0000 7fc43ae00640 1 -- 172.21.15.119:0/4195393209 wait complete. 2024-03-28T18:43:54.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 -- 172.21.15.119:0/2760950405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 msgr2=0x7fee48104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 --2- 172.21.15.119:0/2760950405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee48104f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee34009900 tx=0x7fee3402bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 -- 172.21.15.119:0/2760950405 shutdown_connections 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 --2- 172.21.15.119:0/2760950405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee4810bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 --2- 172.21.15.119:0/2760950405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee48104f70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 --2- 172.21.15.119:0/2760950405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 0x7fee48103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 -- 172.21.15.119:0/2760950405 >> 172.21.15.119:0/2760950405 conn(0x7fee480fee90 msgr2=0x7fee481012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:54.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 -- 172.21.15.119:0/2760950405 shutdown_connections 2024-03-28T18:43:54.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.502+0000 7fee4de00640 1 -- 172.21.15.119:0/2760950405 wait complete. 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 Processor -- start 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 -- start start 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 0x7fee48199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4810f580 con 0x7fee48103900 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4810f400 con 0x7fee481054b0 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.503+0000 7fee4de00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4810f700 con 0x7fee48104af0 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee46000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59290/0 (socket says 172.21.15.119:59290) 2024-03-28T18:43:54.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee46000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36248/0 (socket says 172.21.15.119:36248) 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 -- 172.21.15.119:0/2240727724 learned_addr learned my addr 172.21.15.119:0/2240727724 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee46a00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 0x7fee48199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 msgr2=0x7fee4819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 msgr2=0x7fee48199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 0x7fee48199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 -- 172.21.15.119:0/2240727724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee481a1730 con 0x7fee481054b0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee46000640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.504+0000 7fee47400640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee4000eb10 tx=0x7fee4000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.505+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40004b20 con 0x7fee481054b0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.505+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee481a1a20 con 0x7fee481054b0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.505+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee40004cc0 con 0x7fee481054b0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.505+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee40010760 con 0x7fee481054b0 2024-03-28T18:43:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.505+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee481a1f90 con 0x7fee481054b0 2024-03-28T18:43:54.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.506+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4810ab50 con 0x7fee481054b0 2024-03-28T18:43:54.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.508+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7fee400459c0 con 0x7fee481054b0 2024-03-28T18:43:54.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.509+0000 7fee4ca00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 0x7fee2c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.509+0000 7fee46a00640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 msgr2=0x7fee2c0a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:54.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.509+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7fee400cf220 con 0x7fee481054b0 2024-03-28T18:43:54.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.509+0000 7fee46a00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 0x7fee2c0a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:43:54.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.511+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee4009c1e0 con 0x7fee481054b0 2024-03-28T18:43:54.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setuser ceph since I am not root 2024-03-28T18:43:54.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: ignoring --setgroup ceph since I am not root 2024-03-28T18:43:54.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:54 smithi119 ceph-mon[31813]: pgmap v392: 97 pgs: 97 active+clean; 650 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-28T18:43:54.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:54 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1236340469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:54.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:54 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:43:54.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-mon[36300]: pgmap v392: 97 pgs: 97 active+clean; 650 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-28T18:43:54.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1236340469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:54.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:43:54.685 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: ignoring --setuser ceph since I am not root 2024-03-28T18:43:54.685 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: ignoring --setgroup ceph since I am not root 2024-03-28T18:43:54.685 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-mgr[38041]: -- 172.21.15.196:0/460219815 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5614d0c385a0 con 0x5614d0c0c800 2024-03-28T18:43:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-mon[36065]: pgmap v392: 97 pgs: 97 active+clean; 650 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-28T18:43:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1236340469' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:43:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T18:43:54.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: ignoring --setuser ceph since I am not root 2024-03-28T18:43:54.695 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: ignoring --setgroup ceph since I am not root 2024-03-28T18:43:54.695 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-mgr[37458]: -- 172.21.15.131:0/2341399288 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605c6a325a0 con 0x5605c6a07000 2024-03-28T18:43:54.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.710+0000 7fee46a00640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 msgr2=0x7fee2c0a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:54.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.710+0000 7fee46a00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 0x7fee2c0a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:43:54.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.713+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fee4802d070 con 0x7fee481054b0 2024-03-28T18:43:54.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.721+0000 7fee4ca00640 1 -- 172.21.15.119:0/2240727724 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7fee4009bae0 con 0x7fee481054b0 2024-03-28T18:43:54.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 msgr2=0x7fee2c0a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:43:54.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 0x7fee2c0a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 msgr2=0x7fee481a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fee4000eb10 tx=0x7fee4000c2d0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 shutdown_connections 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7fee2c0a2970 0x7fee2c0a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fee481054b0 0x7fee481a10c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee48104af0 0x7fee4819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 --2- 172.21.15.119:0/2240727724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee48103900 0x7fee48199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.724+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 >> 172.21.15.119:0/2240727724 conn(0x7fee480fee90 msgr2=0x7fee48107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.725+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 shutdown_connections 2024-03-28T18:43:54.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.725+0000 7fee4de00640 1 -- 172.21.15.119:0/2240727724 wait complete. 2024-03-28T18:43:54.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T18:43:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 -- 172.21.15.119:0/3748344196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad41073f0 msgr2=0x7f4ad4109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/3748344196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad41073f0 0x7f4ad4109840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009ca0 tx=0x7f4acc02c100 comp rx=0 tx=0).stop 2024-03-28T18:43:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 -- 172.21.15.119:0/3748344196 shutdown_connections 2024-03-28T18:43:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/3748344196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad41073f0 0x7f4ad4109840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/3748344196 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/3748344196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad4102190 0x7f4ad4104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 -- 172.21.15.119:0/3748344196 >> 172.21.15.119:0/3748344196 conn(0x7f4ad40fbb80 msgr2=0x7f4ad40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 -- 172.21.15.119:0/3748344196 shutdown_connections 2024-03-28T18:43:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.931+0000 7f4ad9c00640 1 -- 172.21.15.119:0/3748344196 wait complete. 2024-03-28T18:43:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.932+0000 7f4ad9c00640 1 Processor -- start 2024-03-28T18:43:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 -- start start 2024-03-28T18:43:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4102190 0x7f4ad4101b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad41073f0 0x7f4ad4100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad410d390 con 0x7f4ad41073f0 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad410d210 con 0x7f4ad4102190 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad9c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad410d510 con 0x7f4ad4104ac0 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4acbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4acbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36264/0 (socket says 172.21.15.119:36264) 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4102190 0x7f4ad4101b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4acbe00640 1 -- 172.21.15.119:0/2237226162 learned_addr learned my addr 172.21.15.119:0/2237226162 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.933+0000 7f4ad3400640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad41073f0 0x7f4ad4100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4102190 msgr2=0x7f4ad4101b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4102190 0x7f4ad4101b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad41073f0 msgr2=0x7f4ad4100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad41073f0 0x7f4ad4100680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:54.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 -- 172.21.15.119:0/2237226162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4100f10 con 0x7f4ad4104ac0 2024-03-28T18:43:54.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acbe00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e9a0 tx=0x7f4abc00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:54.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00cd80 con 0x7f4ad4104ac0 2024-03-28T18:43:54.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad41a1a00 con 0x7f4ad4104ac0 2024-03-28T18:43:54.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4abc0035f0 con 0x7f4ad4104ac0 2024-03-28T18:43:54.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc010670 con 0x7f4ad4104ac0 2024-03-28T18:43:54.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.934+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad41a1fa0 con 0x7f4ad4104ac0 2024-03-28T18:43:54.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.935+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad40fd670 con 0x7f4ad4104ac0 2024-03-28T18:43:54.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.937+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f4abc0037a0 con 0x7f4ad4104ac0 2024-03-28T18:43:54.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.939+0000 7f4acb400640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 0x7f4ab80a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:54.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.939+0000 7f4ad2a00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 msgr2=0x7f4ab80a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:54.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.939+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f4abc0cef20 con 0x7f4ad4104ac0 2024-03-28T18:43:54.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.939+0000 7f4ad2a00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 0x7f4ab80a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:43:54.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:54.943+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4abc09bf70 con 0x7f4ad4104ac0 2024-03-28T18:43:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.140+0000 7f4ad2a00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 msgr2=0x7f4ab80a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.140+0000 7f4ad2a00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 0x7f4ab80a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:43:55.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.143+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f4ad402d070 con 0x7f4ad4104ac0 2024-03-28T18:43:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.151+0000 7f4acb400640 1 -- 172.21.15.119:0/2237226162 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4abc09b870 con 0x7f4ad4104ac0 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 msgr2=0x7f4ab80a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 0x7f4ab80a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 msgr2=0x7f4ad4100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e9a0 tx=0x7f4abc00ee60 comp rx=0 tx=0).stop 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 shutdown_connections 2024-03-28T18:43:55.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f4ab80a28f0 0x7f4ab80a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad41073f0 0x7f4ad4100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ad4104ac0 0x7f4ad4100140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 --2- 172.21.15.119:0/2237226162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ad4102190 0x7f4ad4101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 >> 172.21.15.119:0/2237226162 conn(0x7f4ad40fbb80 msgr2=0x7f4ad4102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 shutdown_connections 2024-03-28T18:43:55.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.154+0000 7f4ad9c00640 1 -- 172.21.15.119:0/2237226162 wait complete. 2024-03-28T18:43:55.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T18:43:55.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:54.925+0000 7f56acc69200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:43:55.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:55.020+0000 7f56acc69200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:43:55.298 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:54.952+0000 7f95cb3f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:43:55.298 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:55.056+0000 7f95cb3f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 -- 172.21.15.119:0/780466667 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c102700 msgr2=0x7f551c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 --2- 172.21.15.119:0/780466667 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c102700 0x7f551c102b80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5508009900 tx=0x7f550802bc30 comp rx=0 tx=0).stop 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 -- 172.21.15.119:0/780466667 shutdown_connections 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 --2- 172.21.15.119:0/780466667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 0x7f551c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 --2- 172.21.15.119:0/780466667 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c102700 0x7f551c102b80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 --2- 172.21.15.119:0/780466667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551c101510 0x7f551c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 -- 172.21.15.119:0/780466667 >> 172.21.15.119:0/780466667 conn(0x7f551c0fca80 msgr2=0x7f551c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:43:55.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 -- 172.21.15.119:0/780466667 shutdown_connections 2024-03-28T18:43:55.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.365+0000 7f5520c00640 1 -- 172.21.15.119:0/780466667 wait complete. 2024-03-28T18:43:55.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 Processor -- start 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 -- start start 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c101510 0x7f551c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551c102700 0x7f551c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 0x7f551c1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c10d1a0 con 0x7f551c102700 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c10d020 con 0x7f551c101510 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f5520c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c10d320 con 0x7f551c1030c0 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f551b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c101510 0x7f551c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f551aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551c102700 0x7f551c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f551b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c101510 0x7f551c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59334/0 (socket says 172.21.15.119:59334) 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.366+0000 7f551be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 0x7f551c1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 learned_addr learned my addr 172.21.15.119:0/1198616977 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c101510 msgr2=0x7f551c199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f551c101510 0x7f551c199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551c102700 msgr2=0x7f551c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551c102700 0x7f551c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c1a1690 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f551be00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 0x7f551c1a1020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f551400ec50 tx=0x7f551400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514003b70 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5514003d10 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c1a1980 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.367+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514010890 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.368+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c1a1e50 con 0x7f551c1030c0 2024-03-28T18:43:55.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.369+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c1a2140 con 0x7f551c1030c0 2024-03-28T18:43:55.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.371+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 19) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f5514004b40 con 0x7f551c1030c0 2024-03-28T18:43:55.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.372+0000 7f5513e00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:43:55.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.372+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:55.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.372+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8364+0+0 (secure 0 0 0) 0x7f55140cf210 con 0x7f551c1030c0 2024-03-28T18:43:55.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.372+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T18:43:55.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.375+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f551409c260 con 0x7f551c1030c0 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:55.143+0000 7f0a839a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:55.268+0000 7f0a839a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:43:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T18:43:55.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:43:55 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.573+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.573+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T18:43:55.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.598+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f551c1a2310 con 0x7f55000a2840 2024-03-28T18:43:55.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:43:55.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:43:55.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:43:55.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T18:43:55.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:43:55.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:43:55.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:43:55.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4195393209' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T18:43:55.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-28T18:43:55.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' 2024-03-28T18:43:55.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.974+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:55.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:55.974+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T18:43:56.029 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:55.753+0000 7f56acc69200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:43:56.029 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:55.859+0000 7f56acc69200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:43:56.029 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.029+0000 7f56acc69200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:43:56.236 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:55.955+0000 7f95cb3f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:43:56.236 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.061+0000 7f95cb3f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:43:56.236 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.236+0000 7f95cb3f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:43:56.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.052+0000 7f0a839a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T18:43:56.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.172+0000 7f0a839a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T18:43:56.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.146+0000 7f56acc69200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:43:56.445 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.274+0000 7f56acc69200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:43:56.684 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.376+0000 7f95cb3f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:43:56.684 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.466+0000 7f95cb3f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:43:56.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:56.775+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:56.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:56.775+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T18:43:56.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.349+0000 7f0a839a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T18:43:56.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.472+0000 7f0a839a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T18:43:56.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.580+0000 7f0a839a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T18:43:56.869 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.594+0000 7f56acc69200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:43:56.870 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.737+0000 7f56acc69200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:43:57.160 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.869+0000 7f56acc69200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:43:57.161 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:56.965+0000 7f56acc69200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:43:57.161 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:57.160+0000 7f56acc69200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:43:57.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.801+0000 7f95cb3f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:43:57.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:56.940+0000 7f95cb3f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:43:57.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:57.049+0000 7f95cb3f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:43:57.282 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:56.911+0000 7f0a839a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T18:43:57.282 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:57.050+0000 7f0a839a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T18:43:57.283 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:57.160+0000 7f0a839a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T18:43:57.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:57.282+0000 7f0a839a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:43:57.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:57.500+0000 7f0a839a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:43:57.684 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:57.191+0000 7f95cb3f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T18:43:57.684 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:57.399+0000 7f95cb3f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T18:43:58.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:58.376+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:43:58.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:43:58.376+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T18:43:58.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:58.206+0000 7f56acc69200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:43:58.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:58.339+0000 7f56acc69200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:43:58.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:58.616+0000 7f95cb3f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:43:58.935 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:58.759+0000 7f95cb3f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:43:58.985 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:58.697+0000 7f56acc69200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:43:58.985 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:58.788+0000 7f56acc69200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:43:58.985 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:58.985+0000 7f56acc69200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:43:59.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:58.764+0000 7f0a839a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T18:43:59.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:58.933+0000 7f0a839a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T18:43:59.434 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:59.183+0000 7f95cb3f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:43:59.435 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:59.299+0000 7f95cb3f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:43:59.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:59.171+0000 7f56acc69200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:43:59.729 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:59.350+0000 7f0a839a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T18:43:59.729 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:59.471+0000 7f0a839a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T18:43:59.729 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:59.728+0000 7f0a839a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:43:59.763 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:59.491+0000 7f56acc69200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:43:59.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:59.525+0000 7f95cb3f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:43:59.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:43:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:43:59.717+0000 7f95cb3f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:44:00.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:43:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:43:59.972+0000 7f0a839a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T18:44:00.351 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:00.040+0000 7f95cb3f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:44:00.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:43:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:43:59.763+0000 7f56acc69200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:44:00.445 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:44:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:44:00.061+0000 7f56acc69200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:44:00.673 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:00.352+0000 7f0a839a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T18:44:00.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:00.351+0000 7f95cb3f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:44:00.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:00.651+0000 7f95cb3f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:44:00.977 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:00.673+0000 7f0a839a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T18:44:01.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:00.977+0000 7f0a839a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T18:44:01.444 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:44:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:44:00.983+0000 7f56acc69200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:44:01.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:01.579+0000 7f551b400640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2493078203 2024-03-28T18:44:01.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:01.580+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T18:44:01.905 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:44:01.576+0000 7f56acc69200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:44:01.905 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:44:01.689+0000 7f56acc69200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:44:01.934 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:01.656+0000 7f95cb3f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:44:01.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:01.991+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 20) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f55140981d0 con 0x7f551c1030c0 2024-03-28T18:44:02.194 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y[37454]: 2024-03-28T18:44:01.905+0000 7f56acc69200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: Standby manager daemon y restarted 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: Standby manager daemon y started 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:44:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:01 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:02.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:02.022+0000 7f0a839a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T18:44:02.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: Standby manager daemon y restarted 2024-03-28T18:44:02.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: Standby manager daemon y started 2024-03-28T18:44:02.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:44:02.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:02.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:44:02.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:01 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: Standby manager daemon y restarted 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: Standby manager daemon y started 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T18:44:02.398 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:01 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.131:0/230262100' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:02.684 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:02.397+0000 7f95cb3f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:44:02.685 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:02.502+0000 7f95cb3f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:44:02.994 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:02.721+0000 7f0a839a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T18:44:02.995 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:02.852+0000 7f0a839a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T18:44:03.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:03.014+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 8 ==== mgrmap(e 21) v1 ==== 137308+0+0 (secure 0 0 0) 0x7f55140cccb0 con 0x7f551c1030c0 2024-03-28T18:44:03.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:03.172+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 9 ==== mgrmap(e 22) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f5514091530 con 0x7f551c1030c0 2024-03-28T18:44:03.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:03.174+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 msgr2=0x7f55000a4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:44:03.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:03.174+0000 7f5513e00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/2493078203,v1:172.21.15.119:6801/2493078203] conn(0x7f55000a2840 0x7f55000a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:03.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z[37985]: 2024-03-28T18:44:02.776+0000 7f95cb3f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:44:03.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: Standby manager daemon z restarted 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: Standby manager daemon z started 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:44:03.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:02 smithi196 ceph-mon[36300]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.147+0000 7f0a839a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: Standby manager daemon z restarted 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: Standby manager daemon z started 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:44:03.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:02 smithi119 ceph-mon[31813]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:03.330 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: Standby manager daemon z restarted 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: Standby manager daemon z started 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T18:44:03.331 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:02 smithi131 ceph-mon[36065]: from='mgr.? 172.21.15.196:0/1799609044' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T18:44:03.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.459+0000 7f09d5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.459+0000 7f09d5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.459+0000 7f09d5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.459+0000 7f09d5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.459+0000 7f09d5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.460+0000 7f09d8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.460+0000 7f09d8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.460+0000 7f09d8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.460+0000 7f09d8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:03.819 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:44:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:44:03.460+0000 7f09d8000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:44:04.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.183+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mon.1 v2:172.21.15.196:3300/0 10 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f5514098aa0 con 0x7f551c1030c0 2024-03-28T18:44:04.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.185+0000 7f5513e00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f550004d420 0x7f550004d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:04.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.185+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f551c1a2310 con 0x7f550004d420 2024-03-28T18:44:04.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.196+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f550004d420 0x7f550004d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:04.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.200+0000 7f551b400640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f550004d420 0x7f550004d800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f550c0058c0 tx=0x7f550c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:04.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.288+0000 7f5513e00640 1 -- 172.21.15.119:0/1198616977 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f551c1a2310 con 0x7f550004d420 2024-03-28T18:44:04.289 INFO:teuthology.orchestra.run.smithi119.stdout:Schedule set for path / 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f550004d420 msgr2=0x7f550004d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f550004d420 0x7f550004d800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f550c0058c0 tx=0x7f550c005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 msgr2=0x7f551c1a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 --2- 172.21.15.119:0/1198616977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f551c1030c0 0x7f551c1a1020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f551400ec50 tx=0x7f551400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f551be00640 1 -- 172.21.15.119:0/1198616977 reap_dead start 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 shutdown_connections 2024-03-28T18:44:04.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 >> 172.21.15.119:0/1198616977 conn(0x7f551c0fca80 msgr2=0x7f551c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 shutdown_connections 2024-03-28T18:44:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.292+0000 7f5520c00640 1 -- 172.21.15.119:0/1198616977 wait complete. 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: Active manager daemon x restarted 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: Activating manager daemon x 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: mgrmap e22: x(active, starting, since 0.0163396s), standbys: y, z 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:44:04.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:44:04.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:44:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:44:04.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: Manager daemon x is now available 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:04 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T18:44:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:44:04.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: Active manager daemon x restarted 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: Activating manager daemon x 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: mgrmap e22: x(active, starting, since 0.0163396s), standbys: y, z 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:44:04.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:44:04.437 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:44:04.438 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: Manager daemon x is now available 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.439 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:04 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: Active manager daemon x restarted 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: Activating manager daemon x 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: mgrmap e22: x(active, starting, since 0.0163396s), standbys: y, z 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T18:44:04.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:44:04.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T18:44:04.447 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: Manager daemon x is now available 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.448 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:04 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 -- 172.21.15.119:0/203126708 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204076060 msgr2=0x7f3204076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 --2- 172.21.15.119:0/203126708 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204076060 0x7f3204076480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f31f4009900 tx=0x7f31f402bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 -- 172.21.15.119:0/203126708 shutdown_connections 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 --2- 172.21.15.119:0/203126708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3204077c40 0x7f32040784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 --2- 172.21.15.119:0/203126708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32040772a0 0x7f3204077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 --2- 172.21.15.119:0/203126708 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204076060 0x7f3204076480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.500+0000 7f320a800640 1 -- 172.21.15.119:0/203126708 >> 172.21.15.119:0/203126708 conn(0x7f32040fee90 msgr2=0x7f32041012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.501+0000 7f320a800640 1 -- 172.21.15.119:0/203126708 shutdown_connections 2024-03-28T18:44:04.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.501+0000 7f320a800640 1 -- 172.21.15.119:0/203126708 wait complete. 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 Processor -- start 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 -- start start 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32040772a0 0x7f320419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204077c40 0x7f32041a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32041139d0 con 0x7f32040772a0 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3204113850 con 0x7f3204077c40 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f320a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3204113b50 con 0x7f3204076060 2024-03-28T18:44:04.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f3203400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f3203400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46968/0 (socket says 172.21.15.119:46968) 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f3203400640 1 -- 172.21.15.119:0/1813138775 learned_addr learned my addr 172.21.15.119:0/1813138775 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f3202a00640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32040772a0 0x7f320419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.502+0000 7f3203e00640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204077c40 0x7f32041a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 -- 172.21.15.119:0/1813138775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204077c40 msgr2=0x7f32041a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204077c40 0x7f32041a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 -- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32040772a0 msgr2=0x7f320419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32040772a0 0x7f320419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 -- 172.21.15.119:0/1813138775 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32041a5b50 con 0x7f3204076060 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f3203400640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f31f4000c00 tx=0x7f31f402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4043020 con 0x7f3204076060 2024-03-28T18:44:04.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31f40054e0 con 0x7f3204076060 2024-03-28T18:44:04.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32041a5de0 con 0x7f3204076060 2024-03-28T18:44:04.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.503+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4003c40 con 0x7f3204076060 2024-03-28T18:44:04.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.504+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32041a6240 con 0x7f3204076060 2024-03-28T18:44:04.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.505+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3204110270 con 0x7f3204076060 2024-03-28T18:44:04.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.507+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f31f4046020 con 0x7f3204076060 2024-03-28T18:44:04.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.509+0000 7f31fbe00640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 0x7f31e80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:04.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.509+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f31f40f0910 con 0x7f3204076060 2024-03-28T18:44:04.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.509+0000 7f3202a00640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 0x7f31e80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:04.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.510+0000 7f3202a00640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 0x7f31e80a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f320419f440 tx=0x7f31f0005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:04.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.512+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31f404d020 con 0x7f3204076060 2024-03-28T18:44:04.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.741+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f3204061c40 con 0x7f31e80a26b0 2024-03-28T18:44:04.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.845+0000 7f31fbe00640 1 -- 172.21.15.119:0/1813138775 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f3204061c40 con 0x7f31e80a26b0 2024-03-28T18:44:04.846 INFO:teuthology.orchestra.run.smithi119.stdout:Retention added to path / 2024-03-28T18:44:04.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.847+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 msgr2=0x7f31e80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.847+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 0x7f31e80a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f320419f440 tx=0x7f31f0005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.847+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 msgr2=0x7f320419df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.847+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f31f4000c00 tx=0x7f31f402c650 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 shutdown_connections 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f31e80a26b0 0x7f31e80a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3204077c40 0x7f32041a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32040772a0 0x7f320419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 --2- 172.21.15.119:0/1813138775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3204076060 0x7f320419df30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 >> 172.21.15.119:0/1813138775 conn(0x7f32040fee90 msgr2=0x7f3204100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 shutdown_connections 2024-03-28T18:44:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:04.848+0000 7f320a800640 1 -- 172.21.15.119:0/1813138775 wait complete. 2024-03-28T18:44:04.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 -- 172.21.15.119:0/3456516215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 msgr2=0x7fd848101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 --2- 172.21.15.119:0/3456516215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848101930 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd838009900 tx=0x7fd83802bc40 comp rx=0 tx=0).stop 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 -- 172.21.15.119:0/3456516215 shutdown_connections 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 --2- 172.21.15.119:0/3456516215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 0x7fd848109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 --2- 172.21.15.119:0/3456516215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 --2- 172.21.15.119:0/3456516215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848101930 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 -- 172.21.15.119:0/3456516215 >> 172.21.15.119:0/3456516215 conn(0x7fd8480fca80 msgr2=0x7fd8480feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:05.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 -- 172.21.15.119:0/3456516215 shutdown_connections 2024-03-28T18:44:05.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.041+0000 7fd847200640 1 -- 172.21.15.119:0/3456516215 wait complete. 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 Processor -- start 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 -- start start 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 0x7fd84819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84810d170 con 0x7fd8481030c0 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84810cff0 con 0x7fd848101510 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd847200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84810d2f0 con 0x7fd848102700 2024-03-28T18:44:05.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.042+0000 7fd845e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58198/0 (socket says 172.21.15.119:58198) 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845e00640 1 -- 172.21.15.119:0/883993930 learned_addr learned my addr 172.21.15.119:0/883993930 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd846800640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 0x7fd84819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 -- 172.21.15.119:0/883993930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 msgr2=0x7fd848197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 -- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 msgr2=0x7fd84819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 0x7fd84819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 -- 172.21.15.119:0/883993930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84819f430 con 0x7fd848102700 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845e00640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd845400640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848197d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e9f0 tx=0x7fd83400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400ccc0 con 0x7fd848102700 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8340035f0 con 0x7fd848102700 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.043+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84819f720 con 0x7fd848102700 2024-03-28T18:44:05.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.044+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400ce60 con 0x7fd848102700 2024-03-28T18:44:05.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.044+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84819fc70 con 0x7fd848102700 2024-03-28T18:44:05.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.045+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8481088b0 con 0x7fd848102700 2024-03-28T18:44:05.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.047+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fd8340037a0 con 0x7fd848102700 2024-03-28T18:44:05.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.048+0000 7fd83ea00640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 0x7fd82c0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.049+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd8340cea70 con 0x7fd848102700 2024-03-28T18:44:05.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.049+0000 7fd845e00640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 0x7fd82c0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.049+0000 7fd845e00640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 0x7fd82c0a4aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c8f0 tx=0x7fd838005b50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:05.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.053+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd83409bc40 con 0x7fd848102700 2024-03-28T18:44:05.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.240+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 24) v1 ==== 137411+0+0 (secure 0 0 0) 0x7fd834097bb0 con 0x7fd848102700 2024-03-28T18:44:05.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.290+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fd848061ae0 con 0x7fd82c0a25e0 2024-03-28T18:44:05.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.301+0000 7fd83ea00640 1 -- 172.21.15.119:0/883993930 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7fd848061ae0 con 0x7fd82c0a25e0 2024-03-28T18:44:05.302 INFO:teuthology.orchestra.run.smithi119.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T18:44:04", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 msgr2=0x7fd82c0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 0x7fd82c0a4aa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd83802c8f0 tx=0x7fd838005b50 comp rx=0 tx=0).stop 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 msgr2=0x7fd848197d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848197d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd83400e9f0 tx=0x7fd83400edf0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 shutdown_connections 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd82c0a25e0 0x7fd82c0a4aa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481030c0 0x7fd84819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd848102700 0x7fd848197d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 --2- 172.21.15.119:0/883993930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd848101510 0x7fd848197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 >> 172.21.15.119:0/883993930 conn(0x7fd8480fca80 msgr2=0x7fd848105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:05.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 shutdown_connections 2024-03-28T18:44:05.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.304+0000 7fd847200640 1 -- 172.21.15.119:0/883993930 wait complete. 2024-03-28T18:44:05.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T18:44:05.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:05 smithi196 ceph-mon[36300]: mgrmap e23: x(active, since 1.03331s), standbys: y, z 2024-03-28T18:44:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:05 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:05 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:05 smithi196 ceph-mon[36300]: [28/Mar/2024:18:44:04] ENGINE Bus STARTING 2024-03-28T18:44:05.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:05 smithi196 ceph-mon[36300]: [28/Mar/2024:18:44:04] ENGINE Serving on http://172.21.15.119:8765 2024-03-28T18:44:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:05 smithi131 ceph-mon[36065]: mgrmap e23: x(active, since 1.03331s), standbys: y, z 2024-03-28T18:44:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:05 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:05 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:05 smithi131 ceph-mon[36065]: [28/Mar/2024:18:44:04] ENGINE Bus STARTING 2024-03-28T18:44:05.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:05 smithi131 ceph-mon[36065]: [28/Mar/2024:18:44:04] ENGINE Serving on http://172.21.15.119:8765 2024-03-28T18:44:05.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 -- 172.21.15.119:0/1693939715 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 msgr2=0x7f7940109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 --2- 172.21.15.119:0/1693939715 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f7940109960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7938009ca0 tx=0x7f793802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 -- 172.21.15.119:0/1693939715 shutdown_connections 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 --2- 172.21.15.119:0/1693939715 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f7940109960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 --2- 172.21.15.119:0/1693939715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 0x7f7940102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 --2- 172.21.15.119:0/1693939715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 -- 172.21.15.119:0/1693939715 >> 172.21.15.119:0/1693939715 conn(0x7f79400fcab0 msgr2=0x7f79400feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 -- 172.21.15.119:0/1693939715 shutdown_connections 2024-03-28T18:44:05.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.501+0000 7f7944a00640 1 -- 172.21.15.119:0/1693939715 wait complete. 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.502+0000 7f7944a00640 1 Processor -- start 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.502+0000 7f7944a00640 1 -- start start 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 0x7f7940197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794010d1b0 con 0x7f7940101520 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794010d030 con 0x7f79401030d0 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f7944a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794010d330 con 0x7f7940102710 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 0x7f7940197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58220/0 (socket says 172.21.15.119:58220) 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 -- 172.21.15.119:0/1712527862 learned_addr learned my addr 172.21.15.119:0/1712527862 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 -- 172.21.15.119:0/1712527862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 msgr2=0x7f7940197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 0x7f7940197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 -- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 msgr2=0x7f7940197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793fe00640 1 -- 172.21.15.119:0/1712527862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794019f390 con 0x7f79401030d0 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.503+0000 7f793f400640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f793fe00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7938037040 tx=0x7f7938032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938041020 con 0x7f79401030d0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794019f620 con 0x7f79401030d0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7938034cd0 con 0x7f79401030d0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938039040 con 0x7f79401030d0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.504+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794019fb60 con 0x7f79401030d0 2024-03-28T18:44:05.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.505+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79401087e0 con 0x7f79401030d0 2024-03-28T18:44:05.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.507+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 24) v1 ==== 137411+0+0 (secure 0 0 0) 0x7f7938044020 con 0x7f79401030d0 2024-03-28T18:44:05.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.508+0000 7f7937e00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 0x7f79240a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:05.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.508+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f79380f0cc0 con 0x7f79401030d0 2024-03-28T18:44:05.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.509+0000 7f793f400640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 0x7f79240a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:05.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.509+0000 7f793f400640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 0x7f79240a4e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7940102570 tx=0x7f7930005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.517+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79380f7080 con 0x7f79401030d0 2024-03-28T18:44:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:05 smithi119 ceph-mon[31813]: mgrmap e23: x(active, since 1.03331s), standbys: y, z 2024-03-28T18:44:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:05 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:05 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:05 smithi119 ceph-mon[31813]: [28/Mar/2024:18:44:04] ENGINE Bus STARTING 2024-03-28T18:44:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:05 smithi119 ceph-mon[31813]: [28/Mar/2024:18:44:04] ENGINE Serving on http://172.21.15.119:8765 2024-03-28T18:44:05.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.783+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7940061ae0 con 0x7f79240a2960 2024-03-28T18:44:05.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.792+0000 7f7937e00640 1 -- 172.21.15.119:0/1712527862 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f7940061ae0 con 0x7f79240a2960 2024-03-28T18:44:05.794 INFO:teuthology.orchestra.run.smithi119.stdout:/ 1m 6m3h 2024-03-28T18:44:05.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 msgr2=0x7f79240a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 0x7f79240a4e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7940102570 tx=0x7f7930005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:05.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 msgr2=0x7f794019ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:05.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7938037040 tx=0x7f7938032f40 comp rx=0 tx=0).stop 2024-03-28T18:44:05.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 shutdown_connections 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f79240a2960 0x7f79240a4e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f79401030d0 0x7f794019ed20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7940102710 0x7f7940197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 --2- 172.21.15.119:0/1712527862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7940101520 0x7f7940197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 >> 172.21.15.119:0/1712527862 conn(0x7f79400fcab0 msgr2=0x7f79401059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 shutdown_connections 2024-03-28T18:44:05.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:05.795+0000 7f7944a00640 1 -- 172.21.15.119:0/1712527862 wait complete. 2024-03-28T18:44:05.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T18:44:05.888 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T18:44:05.899 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T18:44:05.926 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:44:05.926 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T18:44:05.926 INFO:tasks.workunit:timeout=3h 2024-03-28T18:44:05.926 INFO:tasks.workunit:cleanup=True 2024-03-28T18:44:05.926 DEBUG:teuthology.orchestra.run.smithi119:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-28 18:41:28.666770612 +0000 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-28 18:41:28.666770612 +0000 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-28 18:43:43.991483768 +0000 2024-03-28T18:44:05.966 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-28 18:41:28.666770612 +0000 2024-03-28T18:44:05.966 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T18:44:05.967 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T18:44:06.088 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T18:44:06.132 INFO:tasks.workunit.client.0.smithi119.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: [28/Mar/2024:18:44:04] ENGINE Serving on https://172.21.15.119:7150 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: [28/Mar/2024:18:44:04] ENGINE Bus STARTED 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:44:06.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:06 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: [28/Mar/2024:18:44:04] ENGINE Serving on https://172.21.15.119:7150 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: [28/Mar/2024:18:44:04] ENGINE Bus STARTED 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:06 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: [28/Mar/2024:18:44:04] ENGINE Serving on https://172.21.15.119:7150 2024-03-28T18:44:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: [28/Mar/2024:18:44:04] ENGINE Bus STARTED 2024-03-28T18:44:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:06.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:06 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:07 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:07 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:44:07.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:07.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:07 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:44:08.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:44:08.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:08 smithi119 ceph-mon[31813]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.586 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:44:08.587 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:08 smithi196 ceph-mon[36300]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:44:08.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:44:08.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:44:08.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:08 smithi131 ceph-mon[36065]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:44:09.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:09 smithi119 ceph-mon[31813]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:44:09.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:44:09.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:09.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:09.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:44:09.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:44:09.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:09 smithi196 ceph-mon[36300]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:44:09.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:09.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-28T18:44:09.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:09 smithi131 ceph-mon[36065]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:10.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:10 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:10 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: Updating smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:10.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:10 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:11 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:11.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:11.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:11.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:11 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: Updating smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:11 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:44:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T18:44:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:12 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:12 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T18:44:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:12.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:44:12.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:44:12.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:12 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:44:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:14 smithi196 ceph-mon[36300]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T18:44:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:14 smithi131 ceph-mon[36065]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T18:44:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:14 smithi119 ceph-mon[31813]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T18:44:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:16 smithi196 ceph-mon[36300]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T18:44:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:16 smithi131 ceph-mon[36065]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T18:44:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:16 smithi119 ceph-mon[31813]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T18:44:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:18 smithi196 ceph-mon[36300]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:18.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:18 smithi131 ceph-mon[36065]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:18 smithi119 ceph-mon[31813]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:20.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:20 smithi119 ceph-mon[31813]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:20.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:20 smithi196 ceph-mon[36300]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:20 smithi131 ceph-mon[36065]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T18:44:22.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:22 smithi196 ceph-mon[36300]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-28T18:44:22.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:22 smithi131 ceph-mon[36065]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-28T18:44:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:22 smithi119 ceph-mon[31813]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-03-28T18:44:24.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:44:24.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.248+0000 7fc097a00640 1 -- 172.21.15.119:0/2965950084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 msgr2=0x7fc09010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.248+0000 7fc097a00640 1 --2- 172.21.15.119:0/2965950084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc09010e590 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc088009930 tx=0x7fc08802bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 -- 172.21.15.119:0/2965950084 shutdown_connections 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 --2- 172.21.15.119:0/2965950084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc09010e590 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 --2- 172.21.15.119:0/2965950084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 --2- 172.21.15.119:0/2965950084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc090107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 -- 172.21.15.119:0/2965950084 >> 172.21.15.119:0/2965950084 conn(0x7fc090076bd0 msgr2=0x7fc0900fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 -- 172.21.15.119:0/2965950084 shutdown_connections 2024-03-28T18:44:24.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.249+0000 7fc097a00640 1 -- 172.21.15.119:0/2965950084 wait complete. 2024-03-28T18:44:24.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 Processor -- start 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 -- start start 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc090198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc09019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090111b60 con 0x7fc0901076d0 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901119e0 con 0x7fc0900ffc10 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.250+0000 7fc097a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090111ce0 con 0x7fc0901005b0 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc08fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc090198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37634/0 (socket says 172.21.15.119:37634) 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 -- 172.21.15.119:0/695774447 learned_addr learned my addr 172.21.15.119:0/695774447 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc09019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 -- 172.21.15.119:0/695774447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 msgr2=0x7fc090198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc090198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 -- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 msgr2=0x7fc09019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc09019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 -- 172.21.15.119:0/695774447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09019cf30 con 0x7fc0900ffc10 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc09019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc095000640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e9f0 tx=0x7fc08000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.251+0000 7fc08d600640 1 -- 172.21.15.119:0/695774447 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000cd80 con 0x7fc0900ffc10 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.252+0000 7fc08d600640 1 -- 172.21.15.119:0/695774447 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc080010910 con 0x7fc0900ffc10 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.252+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09019d220 con 0x7fc0900ffc10 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.252+0000 7fc08d600640 1 -- 172.21.15.119:0/695774447 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800188f0 con 0x7fc0900ffc10 2024-03-28T18:44:24.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.252+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0901a4ab0 con 0x7fc0900ffc10 2024-03-28T18:44:24.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 Processor -- start 2024-03-28T18:44:24.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc08d600640 1 -- 172.21.15.119:0/695774447 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7fc080010ac0 con 0x7fc0900ffc10 2024-03-28T18:44:24.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 -- start start 2024-03-28T18:44:24.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd5213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd530a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd485250 con 0x5616dd520f20 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd42da30 con 0x5616dd52e6a0 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc09a783740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd4812f0 con 0x5616dd521c00 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.255+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36398/0 (socket says 172.21.15.119:36398) 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd5213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 -- 172.21.15.119:0/1367174396 learned_addr learned my addr 172.21.15.119:0/1367174396 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd530a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 -- 172.21.15.119:0/1367174396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 msgr2=0x5616dd530a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd530a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 -- 172.21.15.119:0/1367174396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 msgr2=0x5616dd5213a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd5213a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 -- 172.21.15.119:0/1367174396 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616dd418690 con 0x5616dd521c00 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd530a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:24.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc095a00640 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc088006010 tx=0x7fc088005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34f0240a3016371 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.256+0000 7fc08d600640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 0x7fc0740a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc086000640 1 -- 172.21.15.119:0/1367174396 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088044020 con 0x5616dd521c00 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc08d600640 1 -- 172.21.15.119:0/695774447 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc0800cf480 con 0x7fc0900ffc10 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc086000640 1 -- 172.21.15.119:0/1367174396 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc088004240 con 0x5616dd521c00 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc086000640 1 -- 172.21.15.119:0/1367174396 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088005640 con 0x5616dd521c00 2024-03-28T18:44:24.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc095000640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 0x7fc0740a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 -- 172.21.15.119:0/1367174396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 msgr2=0x5616dd520950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc088006010 tx=0x7fc088005dd0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc095000640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 0x7fc0740a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0800ccab0 tx=0x7fc0800d18d0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 -- 172.21.15.119:0/1367174396 shutdown_connections 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd530a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd5213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 --2- 172.21.15.119:0/1367174396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd520950 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.257+0000 7fc09a783740 1 -- 172.21.15.119:0/1367174396 >> 172.21.15.119:0/1367174396 conn(0x5616dd4e8200 msgr2=0x5616dd51b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.258+0000 7fc09a783740 1 -- 172.21.15.119:0/1367174396 shutdown_connections 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.258+0000 7fc09a783740 1 -- 172.21.15.119:0/1367174396 wait complete. 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.258+0000 7fc09a783740 1 -- start start 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.258+0000 7fc09a783740 1 Processor -- start 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd52b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd5314b0 con 0x5616dd521c00 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd440e80 con 0x5616dd52e6a0 2024-03-28T18:44:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dd531630 con 0x5616dd520f20 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc09a783740 10 client.0 _ll_register_callbacks cb 0x5616dd453130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36410/0 (socket says 172.21.15.119:36410) 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 -- 172.21.15.119:0/1481546145 learned_addr learned my addr 172.21.15.119:0/1481546145 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd52b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 -- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 msgr2=0x5616dd52b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 -- 172.21.15.119:0/1481546145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 msgr2=0x5616dd52b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd52b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.259+0000 7fc095a00640 1 -- 172.21.15.119:0/1481546145 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616dd2637d0 con 0x5616dd520f20 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc095a00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc088009850 tx=0x7fc08802c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf7348f8f5ade842 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08804e070 con 0x5616dd520f20 2024-03-28T18:44:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc088004240 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc08802cc90 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc08805c050 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc09a783740 10 client.25528 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616dd1993f0 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc088004240 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.260+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc088003850 con 0x5616dd520f20 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 10 client.25528 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 20 client.25528 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd53e3d0 0x5616dd5407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 4 client.25528 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616dd492790 con 0x5616dd53e3d0 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd53e3d0 0x5616dd5407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd53e3d0 0x5616dd5407c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c7ab03203a2f081 server_cookie=139fc64b6ea84a1b in_seq=0 out_seq=0 2024-03-28T18:44:24.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.261+0000 7fc07aa00640 10 client.25528 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc07aa00640 1 -- 172.21.15.119:0/1481546145 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5616dd492790 con 0x5616dd53e3d0 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc07aa00640 10 client.25528 handle_command_reply: tid=0 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 client.25528 shutdown 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 msgr2=0x5616dd52dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc088009850 tx=0x7fc08802c330 comp rx=0 tx=0).stop 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 shutdown_connections 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd53e3d0 0x5616dd5407c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd52b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc07aa00640 0 client.25528 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 --2- 172.21.15.119:0/1481546145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd52dd60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 >> 172.21.15.119:0/1481546145 conn(0x5616dd4e8200 msgr2=0x5616dd51d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 shutdown_connections 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.262+0000 7fc09a783740 1 -- 172.21.15.119:0/1481546145 wait complete. 2024-03-28T18:44:24.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.263+0000 7fc09a783740 20 client.25528 trim_cache size 0 max 16384 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 Processor -- start 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 -- start start 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd51a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901ab1f0 con 0x5616dd521c00 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901acb70 con 0x5616dd52e6a0 2024-03-28T18:44:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc09a783740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901ab850 con 0x5616dd520f20 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc095a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36422/0 (socket says 172.21.15.119:36422) 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc095a00640 1 -- 172.21.15.119:0/2111204585 learned_addr learned my addr 172.21.15.119:0/2111204585 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.280+0000 7fc095a00640 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc08fe00640 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd51a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 -- 172.21.15.119:0/2111204585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 msgr2=0x5616dd52d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 -- 172.21.15.119:0/2111204585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 msgr2=0x5616dd51a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd51a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 -- 172.21.15.119:0/2111204585 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616dd1590f0 con 0x5616dd520f20 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc095a00640 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc088009820 tx=0x7fc08802df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f9852f8147e8dfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc085600640 1 -- 172.21.15.119:0/2111204585 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08804e030 con 0x5616dd520f20 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc085600640 1 -- 172.21.15.119:0/2111204585 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc08804f040 con 0x5616dd520f20 2024-03-28T18:44:24.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.281+0000 7fc085600640 1 -- 172.21.15.119:0/2111204585 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc088044030 con 0x5616dd520f20 2024-03-28T18:44:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- 172.21.15.119:0/2111204585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 msgr2=0x5616dd53a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc088009820 tx=0x7fc08802df90 comp rx=0 tx=0).stop 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- 172.21.15.119:0/2111204585 shutdown_connections 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd52d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd521c00 0x5616dd51a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 --2- 172.21.15.119:0/2111204585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd520f20 0x5616dd53a8d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- 172.21.15.119:0/2111204585 >> 172.21.15.119:0/2111204585 conn(0x5616dd4e8200 msgr2=0x5616dd51d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- 172.21.15.119:0/2111204585 shutdown_connections 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- 172.21.15.119:0/2111204585 wait complete. 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.282+0000 7fc09a783740 1 -- start start 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 Processor -- start 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd527f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd5988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901ac510 con 0x5616dd520f20 2024-03-28T18:44:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901abeb0 con 0x5616dd52e6a0 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0901a93b0 con 0x5616dd521c00 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc09a783740 10 client.0 _ll_register_callbacks cb 0x7fc0800cc8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc08fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc095a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd527f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc08fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36440/0 (socket says 172.21.15.119:36440) 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.283+0000 7fc08fe00640 1 -- 172.21.15.119:0/1940496022 learned_addr learned my addr 172.21.15.119:0/1940496022 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc095a00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd5988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 -- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 msgr2=0x5616dd5988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd5988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 -- 172.21.15.119:0/1940496022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 msgr2=0x5616dd527f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd527f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 -- 172.21.15.119:0/1940496022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616dd598e80 con 0x5616dd521c00 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc095a00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd5988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:24.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc08fe00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x5616dd52a950 tx=0x7fc07c01fd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d76b7086bfa7097b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc07aa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c01f310 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc07c010b60 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.284+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc08804f040 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc07c02e050 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 10 client.25534 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616dd59a1f0 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc08804f040 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc07c01d4d0 con 0x5616dd521c00 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 10 client.25534 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 10 client.25534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 20 client.25534 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd59a900 0x5616dd59cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 4 client.25534 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:44:24.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5616dd48cae0 con 0x5616dd59a900 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.285+0000 7fc095a00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd59a900 0x5616dd59cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc07aa00640 10 client.25534 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc095a00640 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd59a900 0x5616dd59cd20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5277956108032c3a server_cookie=cad2240c2933f000 in_seq=0 out_seq=0 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc07aa00640 1 -- 172.21.15.119:0/1940496022 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5616dd48cae0 con 0x5616dd59a900 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc07aa00640 10 client.25534 handle_command_reply: tid=0 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 client.25534 shutdown 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 msgr2=0x5616dd528450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x5616dd52a950 tx=0x7fc07c01fd60 comp rx=0 tx=0).stop 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 shutdown_connections 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616dd59a900 0x5616dd59cd20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5616dd52e6a0 0x5616dd5988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5616dd521c00 0x5616dd528450 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 --2- 172.21.15.119:0/1940496022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616dd520f20 0x5616dd527f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc07aa00640 0 client.25534 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:24.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 >> 172.21.15.119:0/1940496022 conn(0x5616dd4e8200 msgr2=0x5616dd529890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 shutdown_connections 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 1 -- 172.21.15.119:0/1940496022 wait complete. 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.286+0000 7fc09a783740 20 client.25534 trim_cache size 0 max 16384 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:44:24.291 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 msgr2=0x7fc0740a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 0x7fc0740a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0800ccab0 tx=0x7fc0800d18d0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 msgr2=0x7fc090197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e9f0 tx=0x7fc08000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 shutdown_connections 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc0740a2a00 0x7fc0740a4ec0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0901076d0 0x7fc09019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0901005b0 0x7fc090198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 --2- 172.21.15.119:0/695774447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc0900ffc10 0x7fc090197f40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 >> 172.21.15.119:0/695774447 conn(0x7fc090076bd0 msgr2=0x7fc0900fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 shutdown_connections 2024-03-28T18:44:24.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.287+0000 7fc097a00640 1 -- 172.21.15.119:0/695774447 wait complete. 2024-03-28T18:44:24.306 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:44:24.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 -- 172.21.15.119:0/550386293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40103850 msgr2=0x7f9c40103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 --2- 172.21.15.119:0/550386293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40103850 0x7f9c40103cd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c009900 tx=0x7f9c2c02bc40 comp rx=0 tx=0).stop 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 -- 172.21.15.119:0/550386293 shutdown_connections 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 --2- 172.21.15.119:0/550386293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40104210 0x7f9c40110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 --2- 172.21.15.119:0/550386293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40103850 0x7f9c40103cd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 --2- 172.21.15.119:0/550386293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40109860 0x7f9c40109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 -- 172.21.15.119:0/550386293 >> 172.21.15.119:0/550386293 conn(0x7f9c400ff2e0 msgr2=0x7f9c40101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 -- 172.21.15.119:0/550386293 shutdown_connections 2024-03-28T18:44:24.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.514+0000 7f9c47400640 1 -- 172.21.15.119:0/550386293 wait complete. 2024-03-28T18:44:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 Processor -- start 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 -- start start 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 0x7f9c4019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 0x7f9c4019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40113cc0 con 0x7f9c40103850 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40113b40 con 0x7f9c40109860 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.515+0000 7f9c47400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c40113e40 con 0x7f9c40104210 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c3fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 0x7f9c4019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c44a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 0x7f9c4019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c3fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36464/0 (socket says 172.21.15.119:36464) 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c3fe00640 1 -- 172.21.15.119:0/3217767249 learned_addr learned my addr 172.21.15.119:0/3217767249 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 -- 172.21.15.119:0/3217767249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 msgr2=0x7f9c4019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 -- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 msgr2=0x7f9c4019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 0x7f9c4019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 -- 172.21.15.119:0/3217767249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4019f170 con 0x7f9c40109860 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c3fe00640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c44a00640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 0x7f9c4019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.516+0000 7f9c45400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 0x7f9c4019ea40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800ec50 tx=0x7f9c3800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.517+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38003b70 con 0x7f9c40109860 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.517+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c38003d10 con 0x7f9c40109860 2024-03-28T18:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.517+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4019f460 con 0x7f9c40109860 2024-03-28T18:44:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.517+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38010890 con 0x7f9c40109860 2024-03-28T18:44:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.517+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c401a6cf0 con 0x7f9c40109860 2024-03-28T18:44:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.518+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c40068490 con 0x7f9c40109860 2024-03-28T18:44:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.520+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f9c38003fc0 con 0x7f9c40109860 2024-03-28T18:44:24.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.521+0000 7f9c3d600640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 0x7f9c280a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:24.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.521+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9c380cf5d0 con 0x7f9c40109860 2024-03-28T18:44:24.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.521+0000 7f9c44a00640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 0x7f9c280a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:24.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.522+0000 7f9c44a00640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 0x7f9c280a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009f40 tx=0x7f9c34005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:24.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.525+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c3809c100 con 0x7f9c40109860 2024-03-28T18:44:24.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:24 smithi131 ceph-mon[36065]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:24.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.763+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c4010b830 con 0x7f9c40109860 2024-03-28T18:44:24.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.765+0000 7f9c3d600640 1 -- 172.21.15.119:0/3217767249 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f9c3809ba00 con 0x7f9c40109860 2024-03-28T18:44:24.765 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:44:24.765 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:44:24.765 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.767+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 msgr2=0x7f9c280a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.767+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 0x7f9c280a4f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009f40 tx=0x7f9c34005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 msgr2=0x7f9c4019ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 0x7f9c4019ea40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800ec50 tx=0x7f9c3800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 shutdown_connections 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9c280a2a80 0x7f9c280a4f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9c40109860 0x7f9c4019ea40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c40104210 0x7f9c4019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 --2- 172.21.15.119:0/3217767249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c40103850 0x7f9c4019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 >> 172.21.15.119:0/3217767249 conn(0x7f9c400ff2e0 msgr2=0x7f9c400ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 shutdown_connections 2024-03-28T18:44:24.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.768+0000 7f9c47400640 1 -- 172.21.15.119:0/3217767249 wait complete. 2024-03-28T18:44:24.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:24 smithi119 ceph-mon[31813]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:24.785 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:24.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:44:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:24 smithi196 ceph-mon[36300]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:24.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.997+0000 7effba600640 1 -- 172.21.15.119:0/708795824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4109850 msgr2=0x7effb4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.997+0000 7effba600640 1 --2- 172.21.15.119:0/708795824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4109850 0x7effb4109c30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7effa4009cd0 tx=0x7effa402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 -- 172.21.15.119:0/708795824 shutdown_connections 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 --2- 172.21.15.119:0/708795824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4104200 0x7effb4110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 --2- 172.21.15.119:0/708795824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4103840 0x7effb4103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 --2- 172.21.15.119:0/708795824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4109850 0x7effb4109c30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 -- 172.21.15.119:0/708795824 >> 172.21.15.119:0/708795824 conn(0x7effb40ff2f0 msgr2=0x7effb4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 -- 172.21.15.119:0/708795824 shutdown_connections 2024-03-28T18:44:24.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.998+0000 7effba600640 1 -- 172.21.15.119:0/708795824 wait complete. 2024-03-28T18:44:24.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 Processor -- start 2024-03-28T18:44:24.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 -- start start 2024-03-28T18:44:24.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4104200 0x7effb419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4109850 0x7effb419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4113c60 con 0x7effb4104200 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4113ae0 con 0x7effb4109850 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effba600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb4113de0 con 0x7effb4103840 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effb3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effb3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36476/0 (socket says 172.21.15.119:36476) 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effb3400640 1 -- 172.21.15.119:0/3071722248 learned_addr learned my addr 172.21.15.119:0/3071722248 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effb2a00640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4104200 0x7effb419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:24.999+0000 7effb3e00640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4109850 0x7effb419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 -- 172.21.15.119:0/3071722248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4109850 msgr2=0x7effb419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4109850 0x7effb419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 -- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4104200 msgr2=0x7effb419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4104200 0x7effb419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 -- 172.21.15.119:0/3071722248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb419f130 con 0x7effb4103840 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.000+0000 7effb3400640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7effa4004af0 tx=0x7effa4032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.001+0000 7effabe00640 1 -- 172.21.15.119:0/3071722248 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4041020 con 0x7effb4103840 2024-03-28T18:44:25.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.001+0000 7effabe00640 1 -- 172.21.15.119:0/3071722248 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effa4034cd0 con 0x7effb4103840 2024-03-28T18:44:25.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.001+0000 7effabe00640 1 -- 172.21.15.119:0/3071722248 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa4039040 con 0x7effb4103840 2024-03-28T18:44:25.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.001+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb419f3c0 con 0x7effb4103840 2024-03-28T18:44:25.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.001+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb41a6cb0 con 0x7effb4103840 2024-03-28T18:44:25.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.004+0000 7effabe00640 1 -- 172.21.15.119:0/3071722248 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7effa4044020 con 0x7effb4103840 2024-03-28T18:44:25.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.004+0000 7effbd415740 1 Processor -- start 2024-03-28T18:44:25.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.004+0000 7effbd415740 1 -- start start 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53778080 0x559e53780630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e53783050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e536db2c0 con 0x559e53780c00 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e53683c90 con 0x559e537776d0 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effbd415740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e536d7360 con 0x559e53778080 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37750/0 (socket says 172.21.15.119:37750) 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53778080 0x559e53780630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 -- 172.21.15.119:0/4275970926 learned_addr learned my addr 172.21.15.119:0/4275970926 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb2a00640 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e53783050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 -- 172.21.15.119:0/4275970926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53778080 msgr2=0x559e53780630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53778080 0x559e53780630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 -- 172.21.15.119:0/4275970926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 msgr2=0x559e53783050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e53783050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 -- 172.21.15.119:0/4275970926 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e536a8080 con 0x559e537776d0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.005+0000 7effb3e00640 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7effac009c20 tx=0x7effac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccbf0f8df4338b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effabe00640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 0x7eff980a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effa8c00640 1 -- 172.21.15.119:0/4275970926 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac01e070 con 0x559e537776d0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effa8c00640 1 -- 172.21.15.119:0/4275970926 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effac019b10 con 0x559e537776d0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effa8c00640 1 -- 172.21.15.119:0/4275970926 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac00eae0 con 0x559e537776d0 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effabe00640 1 -- 172.21.15.119:0/3071722248 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7effa40f0aa0 con 0x7effb4103840 2024-03-28T18:44:25.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.006+0000 7effb3400640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 0x7eff980a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- 172.21.15.119:0/4275970926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 msgr2=0x559e53777ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7effac009c20 tx=0x7effac011810 comp rx=0 tx=0).stop 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effb3400640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 0x7eff980a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7effa40ee1e0 tx=0x7effa40f44a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- 172.21.15.119:0/4275970926 shutdown_connections 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e53783050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53778080 0x559e53780630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 --2- 172.21.15.119:0/4275970926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e53777ab0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- 172.21.15.119:0/4275970926 >> 172.21.15.119:0/4275970926 conn(0x559e53771da0 msgr2=0x559e53772180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- 172.21.15.119:0/4275970926 shutdown_connections 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- 172.21.15.119:0/4275970926 wait complete. 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 -- start start 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.007+0000 7effbd415740 1 Processor -- start 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e537804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53780c00 0x559e5377ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e53783b70 con 0x559e53778080 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e536970d0 con 0x559e53780c00 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e53783cf0 con 0x559e537776d0 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effbd415740 10 client.0 _ll_register_callbacks cb 0x559e5369be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e537804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40864/0 (socket says 172.21.15.119:40864) 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb2a00640 1 -- 172.21.15.119:0/364128864 learned_addr learned my addr 172.21.15.119:0/364128864 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb3e00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53780c00 0x559e5377ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.008+0000 7effb2a00640 1 -- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 msgr2=0x559e537804c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effb2a00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e537804c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effb2a00640 1 -- 172.21.15.119:0/364128864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53780c00 msgr2=0x559e5377ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effb2a00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53780c00 0x559e5377ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effb2a00640 1 -- 172.21.15.119:0/364128864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e533ef3f0 con 0x559e53778080 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effb2a00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effa00126d0 tx=0x7effa0012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48259aa59902b4f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7eff9d600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0004050 con 0x559e53778080 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7effa00041f0 con 0x559e53778080 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effac019b10 con 0x559e53778080 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7effa00043f0 con 0x559e53778080 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effbd415740 10 client.15411 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.009+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559e534b97d0 con 0x559e53778080 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effac019b10 con 0x559e53778080 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7effa0003bc0 con 0x559e53778080 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 10 client.15411 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 10 client.15411 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 20 client.15411 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e53790aa0 0x559e53792e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 4 client.15411 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e536e8800 con 0x559e53790aa0 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.010+0000 7effb3e00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e53790aa0 0x559e53792e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.011+0000 7effb3e00640 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e53790aa0 0x559e53792e90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6ce1b29ef7aa0c6 server_cookie=4a4501ea47ee9e6b in_seq=0 out_seq=0 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.011+0000 7eff9d600640 10 client.15411 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.011+0000 7eff9d600640 1 -- 172.21.15.119:0/364128864 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559e536e8800 con 0x559e53790aa0 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.011+0000 7eff9d600640 10 client.15411 handle_command_reply: tid=0 2024-03-28T18:44:25.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 client.15411 shutdown 2024-03-28T18:44:25.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 msgr2=0x559e5377d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effa00126d0 tx=0x7effa0012b90 comp rx=0 tx=0).stop 2024-03-28T18:44:25.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 shutdown_connections 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e53790aa0 0x559e53792e90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53780c00 0x559e5377ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e5377d850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 --2- 172.21.15.119:0/364128864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e537804c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7eff9d600640 0 client.15411 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 >> 172.21.15.119:0/364128864 conn(0x559e53771da0 msgr2=0x559e53773af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 shutdown_connections 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 1 -- 172.21.15.119:0/364128864 wait complete. 2024-03-28T18:44:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.012+0000 7effbd415740 20 client.15411 trim_cache size 0 max 16384 2024-03-28T18:44:25.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.025+0000 7effbd415740 1 Processor -- start 2024-03-28T18:44:25.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.025+0000 7effbd415740 1 -- start start 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e5377ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e5377aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41af580 con 0x559e53780c00 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41b0f00 con 0x559e53778080 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effbd415740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41afbe0 con 0x559e537776d0 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e5377ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37772/0 (socket says 172.21.15.119:37772) 2024-03-28T18:44:25.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 -- 172.21.15.119:0/2345096794 learned_addr learned my addr 172.21.15.119:0/2345096794 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb3e00640 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e5377aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 -- 172.21.15.119:0/2345096794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 msgr2=0x559e5377ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e5377ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 -- 172.21.15.119:0/2345096794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 msgr2=0x559e5377aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e5377aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb2a00640 1 -- 172.21.15.119:0/2345096794 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e5376d700 con 0x559e53778080 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.026+0000 7effb3e00640 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e5377aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.027+0000 7effb2a00640 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7effa0003fb0 tx=0x7effa00092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f58f327ac23d5693 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.027+0000 7eff9fe00640 1 -- 172.21.15.119:0/2345096794 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0031070 con 0x559e53778080 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.027+0000 7eff9fe00640 1 -- 172.21.15.119:0/2345096794 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effa000fe00 con 0x559e53778080 2024-03-28T18:44:25.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.027+0000 7eff9fe00640 1 -- 172.21.15.119:0/2345096794 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0003ab0 con 0x559e53778080 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- 172.21.15.119:0/2345096794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 msgr2=0x559e537804c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7effa0003fb0 tx=0x7effa00092e0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- 172.21.15.119:0/2345096794 shutdown_connections 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53780c00 0x559e5377aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e53778080 0x559e537804c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 --2- 172.21.15.119:0/2345096794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e537776d0 0x559e5377ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- 172.21.15.119:0/2345096794 >> 172.21.15.119:0/2345096794 conn(0x559e53771da0 msgr2=0x559e53773a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- 172.21.15.119:0/2345096794 shutdown_connections 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- 172.21.15.119:0/2345096794 wait complete. 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 -- start start 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.028+0000 7effbd415740 1 Processor -- start 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e537889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53780c00 0x559e53785260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41b08a0 con 0x559e53778080 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41b0240 con 0x559e537776d0 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb41ad740 con 0x559e53780c00 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effbd415740 10 client.0 _ll_register_callbacks cb 0x7effa00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e537889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40872/0 (socket says 172.21.15.119:40872) 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 -- 172.21.15.119:0/734640204 learned_addr learned my addr 172.21.15.119:0/734640204 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb3e00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53780c00 0x559e53785260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 -- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53780c00 msgr2=0x559e53785260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53780c00 0x559e53785260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 -- 172.21.15.119:0/734640204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 msgr2=0x559e537889a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e537889a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.029+0000 7effb2a00640 1 -- 172.21.15.119:0/734640204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e537857a0 con 0x559e53778080 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7effb2a00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7effa0003630 tx=0x7effa000fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb80bb42f30d2e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7eff9d600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0031070 con 0x559e53778080 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7effa0003a40 con 0x559e53778080 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effa000fe00 con 0x559e53778080 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7effa0003c60 con 0x559e53778080 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7effbd415740 10 client.15417 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559e537ecfb0 con 0x559e53778080 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.030+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effa000fe00 con 0x559e53778080 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7effa00107f0 con 0x559e53778080 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 10 client.15417 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 10 client.15417 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 20 client.15417 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e537ed610 0x559e537efa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 4 client.15417 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559e536e2b50 con 0x559e537ed610 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effb3e00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e537ed610 0x559e537efa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7effb3e00640 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e537ed610 0x559e537efa30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f3b7f97217d73d15 server_cookie=e99b85699c3d4ec1 in_seq=0 out_seq=0 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.031+0000 7eff9d600640 10 client.15417 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7eff9d600640 1 -- 172.21.15.119:0/734640204 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559e536e2b50 con 0x559e537ed610 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7eff9d600640 10 client.15417 handle_command_reply: tid=0 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 client.15417 shutdown 2024-03-28T18:44:25.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 msgr2=0x559e53784d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7effa0003630 tx=0x7effa000fb50 comp rx=0 tx=0).stop 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 shutdown_connections 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x559e537ed610 0x559e537efa30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559e53780c00 0x559e53785260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559e53778080 0x559e53784d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 --2- 172.21.15.119:0/734640204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559e537776d0 0x559e537889a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 >> 172.21.15.119:0/734640204 conn(0x559e53771da0 msgr2=0x559e53770620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7eff9d600640 0 client.15417 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 shutdown_connections 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 1 -- 172.21.15.119:0/734640204 wait complete. 2024-03-28T18:44:25.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.032+0000 7effbd415740 20 client.15417 trim_cache size 0 max 16384 2024-03-28T18:44:25.037 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:25.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 msgr2=0x7eff980a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 0x7eff980a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7effa40ee1e0 tx=0x7effa40f44a0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 msgr2=0x7effb419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7effa4004af0 tx=0x7effa4032ea0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 shutdown_connections 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff980a2a80 0x7eff980a4f40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7effb4109850 0x7effb419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effb4104200 0x7effb419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 --2- 172.21.15.119:0/3071722248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7effb4103840 0x7effb419a140 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 >> 172.21.15.119:0/3071722248 conn(0x7effb40ff2f0 msgr2=0x7effb40ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 shutdown_connections 2024-03-28T18:44:25.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.033+0000 7effba600640 1 -- 172.21.15.119:0/3071722248 wait complete. 2024-03-28T18:44:25.050 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:25.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 -- 172.21.15.119:0/1628426968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 msgr2=0x7f6a68107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 --2- 172.21.15.119:0/1628426968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a68107a20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009930 tx=0x7f6a5802bc20 comp rx=0 tx=0).stop 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 -- 172.21.15.119:0/1628426968 shutdown_connections 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 --2- 172.21.15.119:0/1628426968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 0x7f6a6810e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 --2- 172.21.15.119:0/1628426968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a68101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 --2- 172.21.15.119:0/1628426968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a68107a20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 -- 172.21.15.119:0/1628426968 >> 172.21.15.119:0/1628426968 conn(0x7f6a68078aa0 msgr2=0x7f6a68078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 -- 172.21.15.119:0/1628426968 shutdown_connections 2024-03-28T18:44:25.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.249+0000 7f6a6d000640 1 -- 172.21.15.119:0/1628426968 wait complete. 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.250+0000 7f6a6d000640 1 Processor -- start 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.250+0000 7f6a6d000640 1 -- start start 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 0x7f6a6819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a6819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68111a10 con 0x7f6a68101ff0 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68111890 con 0x7f6a68101630 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a6d000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68111b90 con 0x7f6a68107640 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37810/0 (socket says 172.21.15.119:37810) 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 -- 172.21.15.119:0/2776887774 learned_addr learned my addr 172.21.15.119:0/2776887774 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a66800640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a6819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a5d400640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 0x7f6a6819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 -- 172.21.15.119:0/2776887774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 msgr2=0x7f6a6819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a6819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 -- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 msgr2=0x7f6a6819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 0x7f6a6819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.251+0000 7f6a65e00640 1 -- 172.21.15.119:0/2776887774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6819f140 con 0x7f6a68101630 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009430 tx=0x7f6a5802cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a5f400640 1 -- 172.21.15.119:0/2776887774 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58041070 con 0x7f6a68101630 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6819f3d0 con 0x7f6a68101630 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a5f400640 1 -- 172.21.15.119:0/2776887774 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a58004450 con 0x7f6a68101630 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a5f400640 1 -- 172.21.15.119:0/2776887774 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58003dc0 con 0x7f6a68101630 2024-03-28T18:44:25.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.252+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681a6cc0 con 0x7f6a68101630 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.255+0000 7f6a6fc28740 1 Processor -- start 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.255+0000 7f6a5f400640 1 -- 172.21.15.119:0/2776887774 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f6a58044070 con 0x7f6a68101630 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.255+0000 7f6a6fc28740 1 -- start start 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d61630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d64050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123cbc2c0 con 0x557123d59080 2024-03-28T18:44:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123c64c90 con 0x557123d586d0 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123cb8360 con 0x557123d61c00 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37820/0 (socket says 172.21.15.119:37820) 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a5d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d64050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 -- 172.21.15.119:0/2630390529 learned_addr learned my addr 172.21.15.119:0/2630390529 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d61630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 -- 172.21.15.119:0/2630390529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 msgr2=0x557123d64050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d64050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 -- 172.21.15.119:0/2630390529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 msgr2=0x557123d61630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d61630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a66800640 1 -- 172.21.15.119:0/2630390529 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557123c89080 con 0x557123d586d0 2024-03-28T18:44:25.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.256+0000 7f6a5d400640 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d64050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a66800640 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009f50 tx=0x7f6a600118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18e844a70e45a879 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a57400640 1 -- 172.21.15.119:0/2630390529 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004d20 con 0x557123d586d0 2024-03-28T18:44:25.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a5f400640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 0x7f6a4c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a57400640 1 -- 172.21.15.119:0/2630390529 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a60004050 con 0x557123d586d0 2024-03-28T18:44:25.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a57400640 1 -- 172.21.15.119:0/2630390529 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60005240 con 0x557123d586d0 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 0x7f6a4c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.257+0000 7f6a5f400640 1 -- 172.21.15.119:0/2776887774 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6a580f09e0 con 0x7f6a68101630 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- 172.21.15.119:0/2630390529 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 msgr2=0x557123d58ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009f50 tx=0x7f6a600118d0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a65e00640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 0x7f6a4c0a4ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a580ee120 tx=0x7f6a58004ec0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- 172.21.15.119:0/2630390529 shutdown_connections 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d64050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d61630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/2630390529 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d58ab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- 172.21.15.119:0/2630390529 >> 172.21.15.119:0/2630390529 conn(0x557123d52da0 msgr2=0x557123d53180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- 172.21.15.119:0/2630390529 shutdown_connections 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- 172.21.15.119:0/2630390529 wait complete. 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.258+0000 7f6a6fc28740 1 -- start start 2024-03-28T18:44:25.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 Processor -- start 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d5e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d5ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123d64b70 con 0x557123d59080 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123c780d0 con 0x557123d586d0 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557123d64cf0 con 0x557123d61c00 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a6fc28740 10 client.0 _ll_register_callbacks cb 0x557123c7ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a66800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a66800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37830/0 (socket says 172.21.15.119:37830) 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a66800640 1 -- 172.21.15.119:0/1878128297 learned_addr learned my addr 172.21.15.119:0/1878128297 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d5e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.259+0000 7f6a66800640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d5ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 -- 172.21.15.119:0/1878128297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 msgr2=0x557123d5ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d5ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 -- 172.21.15.119:0/1878128297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 msgr2=0x557123d5e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d5e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 -- 172.21.15.119:0/1878128297 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571239d03f0 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d5e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a66800640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009250 tx=0x7f6a60005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4e989672cfdae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a4be00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004a40 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6a6001f040 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.260+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a60004050 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a60034050 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 10 client.25547 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557123a9a7d0 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a60004050 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6a60019430 con 0x557123d586d0 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 10 client.25547 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 10 client.25547 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:44:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 20 client.25547 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d71960 0x557123d73d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 4 client.25547 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557123cc9800 con 0x557123d71960 2024-03-28T18:44:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.261+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d71960 0x557123d73d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.262+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d71960 0x557123d73d50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0ea3e2ba1899172 server_cookie=d2de161919849703 in_seq=0 out_seq=0 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.262+0000 7f6a4be00640 10 client.25547 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a4be00640 1 -- 172.21.15.119:0/1878128297 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557123cc9800 con 0x557123d71960 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a4be00640 10 client.25547 handle_command_reply: tid=0 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 client.25547 shutdown 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 msgr2=0x557123d614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009250 tx=0x7f6a60005bd0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 shutdown_connections 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d61c00 0x557123d5ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d71960 0x557123d73d50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d59080 0x557123d5e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1878128297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d586d0 0x557123d614a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 >> 172.21.15.119:0/1878128297 conn(0x557123d52da0 msgr2=0x557123d54af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a4be00640 0 client.25547 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 shutdown_connections 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1878128297 wait complete. 2024-03-28T18:44:25.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.263+0000 7f6a6fc28740 20 client.25547 trim_cache size 0 max 16384 2024-03-28T18:44:25.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.276+0000 7f6a6fc28740 1 Processor -- start 2024-03-28T18:44:25.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 -- start start 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d5d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d60f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681af650 con 0x557123d586d0 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681b0fd0 con 0x557123d59080 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681afcb0 con 0x557123d52da0 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a66800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d5d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a5d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d60f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a66800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a5d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d60f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40928/0 (socket says 172.21.15.119:40928) 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.277+0000 7f6a5d400640 1 -- 172.21.15.119:0/906755294 learned_addr learned my addr 172.21.15.119:0/906755294 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 -- 172.21.15.119:0/906755294 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 msgr2=0x557123d614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 -- 172.21.15.119:0/906755294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 msgr2=0x557123d60f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d60f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 -- 172.21.15.119:0/906755294 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557123d5d7e0 con 0x557123d52da0 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a66800640 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d5d100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009560 tx=0x7f6a600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ee2c8a77a68757 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a56a00640 1 -- 172.21.15.119:0/906755294 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004000 con 0x557123d52da0 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a56a00640 1 -- 172.21.15.119:0/906755294 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a60003b10 con 0x557123d52da0 2024-03-28T18:44:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.278+0000 7f6a56a00640 1 -- 172.21.15.119:0/906755294 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600052f0 con 0x557123d52da0 2024-03-28T18:44:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- 172.21.15.119:0/906755294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 msgr2=0x557123d5d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d5d100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009560 tx=0x7f6a600046c0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- 172.21.15.119:0/906755294 shutdown_connections 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d60f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/906755294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d5d100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- 172.21.15.119:0/906755294 >> 172.21.15.119:0/906755294 conn(0x557123d5eed0 msgr2=0x557123d51eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- 172.21.15.119:0/906755294 shutdown_connections 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- 172.21.15.119:0/906755294 wait complete. 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 -- start start 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.279+0000 7f6a6fc28740 1 Processor -- start 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d65500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d65f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681b0970 con 0x557123d586d0 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681b0310 con 0x557123d59080 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681ad810 con 0x557123d52da0 2024-03-28T18:44:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a6fc28740 10 client.0 _ll_register_callbacks cb 0x7f6a500097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a66800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d65500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40942/0 (socket says 172.21.15.119:40942) 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 -- 172.21.15.119:0/1162411751 learned_addr learned my addr 172.21.15.119:0/1162411751 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a66800640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d65f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 -- 172.21.15.119:0/1162411751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 msgr2=0x557123d65500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d65500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 -- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 msgr2=0x557123d65f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d65f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a5d400640 1 -- 172.21.15.119:0/1162411751 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557123dcc050 con 0x557123d586d0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.280+0000 7f6a66800640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a5d400640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x557123d5e090 tx=0x7f6a500122f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ce93e539668e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a4be00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50022240 con 0x557123d586d0 2024-03-28T18:44:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6a50004050 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a60003b10 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a50004250 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a6fc28740 10 client.15435 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.281+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557123dcd3a0 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a60003b10 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6a50003710 con 0x557123d586d0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 10 client.15435 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 10 client.15435 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 20 client.15435 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d61c00 0x557123dcfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 4 client.15435 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557123cc3b50 con 0x557123d61c00 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.282+0000 7f6a66800640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d61c00 0x557123dcfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a66800640 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d61c00 0x557123dcfb70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=97db481e9b4cdba0 server_cookie=63e40e46133ba0f1 in_seq=0 out_seq=0 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a4be00640 10 client.15435 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a4be00640 1 -- 172.21.15.119:0/1162411751 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557123cc3b50 con 0x557123d61c00 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a4be00640 10 client.15435 handle_command_reply: tid=0 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 client.15435 shutdown 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 msgr2=0x557123d65a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x557123d5e090 tx=0x7f6a500122f0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 shutdown_connections 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557123d59080 0x557123d65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x557123d61c00 0x557123dcfb70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557123d586d0 0x557123d65a40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 --2- 172.21.15.119:0/1162411751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557123d52da0 0x557123d65500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a4be00640 0 client.15435 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 >> 172.21.15.119:0/1162411751 conn(0x557123d5eed0 msgr2=0x557123d51eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 shutdown_connections 2024-03-28T18:44:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 1 -- 172.21.15.119:0/1162411751 wait complete. 2024-03-28T18:44:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.283+0000 7f6a6fc28740 20 client.15435 trim_cache size 0 max 16384 2024-03-28T18:44:25.288 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 msgr2=0x7f6a4c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 0x7f6a4c0a4ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a580ee120 tx=0x7f6a58004ec0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 msgr2=0x7f6a6819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009430 tx=0x7f6a5802cb10 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 shutdown_connections 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6a4c0a2a00 0x7f6a4c0a4ec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a68107640 0x7f6a6819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a68101ff0 0x7f6a6819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 --2- 172.21.15.119:0/2776887774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6a68101630 0x7f6a6819a150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 >> 172.21.15.119:0/2776887774 conn(0x7f6a68078aa0 msgr2=0x7f6a6810cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 shutdown_connections 2024-03-28T18:44:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.284+0000 7f6a6d000640 1 -- 172.21.15.119:0/2776887774 wait complete. 2024-03-28T18:44:25.302 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:25.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:44:25.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 -- 172.21.15.119:0/1425225724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f00107eb0 msgr2=0x7f0f00108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1425225724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f00107eb0 0x7f0f00108290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009900 tx=0x7f0ef002bc50 comp rx=0 tx=0).stop 2024-03-28T18:44:25.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 -- 172.21.15.119:0/1425225724 shutdown_connections 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1425225724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0010cf80 0x7f0f00114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1425225724 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f00108860 0x7f0f0010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1425225724 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f00107eb0 0x7f0f00108290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 -- 172.21.15.119:0/1425225724 >> 172.21.15.119:0/1425225724 conn(0x7f0f00075be0 msgr2=0x7f0f00078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 -- 172.21.15.119:0/1425225724 shutdown_connections 2024-03-28T18:44:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.517+0000 7f0f07a00640 1 -- 172.21.15.119:0/1425225724 wait complete. 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.518+0000 7f0f07a00640 1 Processor -- start 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 -- start start 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f00107eb0 0x7f0f0019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0010cf80 0x7f0f0019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00118080 con 0x7f0f00107eb0 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00117f00 con 0x7f0f0010cf80 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f07a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00118200 con 0x7f0f001a2e00 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f05a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0ef7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0010cf80 0x7f0f0019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f05a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36596/0 (socket says 172.21.15.119:36596) 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f05000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f00107eb0 0x7f0f0019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.519+0000 7f0f05a00640 1 -- 172.21.15.119:0/1873591136 learned_addr learned my addr 172.21.15.119:0/1873591136 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 -- 172.21.15.119:0/1873591136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0010cf80 msgr2=0x7f0f0019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0010cf80 0x7f0f0019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 -- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f00107eb0 msgr2=0x7f0f0019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f00107eb0 0x7f0f0019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 -- 172.21.15.119:0/1873591136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f001a38f0 con 0x7f0f001a2e00 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f05a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800eb10 tx=0x7f0ef800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0efe000640 1 -- 172.21.15.119:0/1873591136 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8004b20 con 0x7f0f001a2e00 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0efe000640 1 -- 172.21.15.119:0/1873591136 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef8004cc0 con 0x7f0f001a2e00 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0efe000640 1 -- 172.21.15.119:0/1873591136 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010760 con 0x7f0f001a2e00 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f001a3be0 con 0x7f0f001a2e00 2024-03-28T18:44:25.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.520+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f001ab380 con 0x7f0f001a2e00 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 Processor -- start 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0efe000640 1 -- 172.21.15.119:0/1873591136 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f0ef8005020 con 0x7f0f001a2e00 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 -- start start 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5570758026d0 0x557075802ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x55707580e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570757662c0 con 0x55707580bc00 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55707570ec90 con 0x557075803080 2024-03-28T18:44:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.525+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557075762360 con 0x5570758026d0 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37858/0 (socket says 172.21.15.119:37858) 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 -- 172.21.15.119:0/3797204866 learned_addr learned my addr 172.21.15.119:0/3797204866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5570758026d0 0x557075802ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x55707580e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 -- 172.21.15.119:0/3797204866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5570758026d0 msgr2=0x557075802ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5570758026d0 0x557075802ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 -- 172.21.15.119:0/3797204866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 msgr2=0x55707580e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x55707580e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 -- 172.21.15.119:0/3797204866 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557075733080 con 0x557075803080 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x55707580e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0f05000640 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009900 tx=0x7f0ef00095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba126a7735d0b919 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0efe000640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 0x7f0ee40a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0efe000640 1 -- 172.21.15.119:0/1873591136 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0ef80cf510 con 0x7f0f001a2e00 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.526+0000 7f0ef6000640 1 -- 172.21.15.119:0/3797204866 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0042070 con 0x557075803080 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0ef6000640 1 -- 172.21.15.119:0/3797204866 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ef002c6b0 con 0x557075803080 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0ef6000640 1 -- 172.21.15.119:0/3797204866 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0004f60 con 0x557075803080 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 0x7f0ee40a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/3797204866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 msgr2=0x55707580b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009900 tx=0x7f0ef00095e0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/3797204866 shutdown_connections 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x55707580e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557075803080 0x55707580b630 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/3797204866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5570758026d0 0x557075802ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/3797204866 >> 172.21.15.119:0/3797204866 conn(0x5570757fcda0 msgr2=0x5570757fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 0x7f0ee40a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f00107060 tx=0x7f0ee8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/3797204866 shutdown_connections 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.527+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/3797204866 wait complete. 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 -- start start 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 Processor -- start 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557075803080 0x557075808940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55707580bc00 0x557075809080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55707580eb10 con 0x557075803080 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570757220d0 con 0x5570758026d0 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.528+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55707580ec90 con 0x55707580bc00 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f0a7a5740 10 client.0 _ll_register_callbacks cb 0x557075726e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37864/0 (socket says 172.21.15.119:37864) 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 -- 172.21.15.119:0/763124747 learned_addr learned my addr 172.21.15.119:0/763124747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05a00640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55707580bc00 0x557075809080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557075803080 0x557075808940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 -- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55707580bc00 msgr2=0x557075809080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55707580bc00 0x557075809080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 -- 172.21.15.119:0/763124747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557075803080 msgr2=0x557075808940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557075803080 0x557075808940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 -- 172.21.15.119:0/763124747 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55707547a3f0 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0f05000640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ef002c190 tx=0x7f0ef002bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ebd81fa3c4cde39 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.529+0000 7f0eeea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0042040 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0ef0005160 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557075773800 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef0005360 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 10 client.25565 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5570755447d0 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557075773800 con 0x5570758026d0 2024-03-28T18:44:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0ef0050450 con 0x5570758026d0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 10 client.25565 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 10 client.25565 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.530+0000 7f0f0a7a5740 20 client.25565 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55707581b8a0 0x55707581dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0f0a7a5740 4 client.25565 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55707576db50 con 0x55707581b8a0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55707581b8a0 0x55707581dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55707581b8a0 0x55707581dc90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59c6c7ceabb61186 server_cookie=a2643b1cbb3b5f80 in_seq=0 out_seq=0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.531+0000 7f0eeea00640 10 client.25565 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0eeea00640 1 -- 172.21.15.119:0/763124747 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55707576db50 con 0x55707581b8a0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0eeea00640 10 client.25565 handle_command_reply: tid=0 2024-03-28T18:44:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 client.25565 shutdown 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 msgr2=0x55707580b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0ef002c190 tx=0x7f0ef002bc50 comp rx=0 tx=0).stop 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 shutdown_connections 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55707581b8a0 0x55707581dc90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55707580bc00 0x557075809080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557075803080 0x557075808940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/763124747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707580b5b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0eeea00640 0 client.25565 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 >> 172.21.15.119:0/763124747 conn(0x5570757fcda0 msgr2=0x5570757feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 shutdown_connections 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/763124747 wait complete. 2024-03-28T18:44:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.533+0000 7f0f0a7a5740 20 client.25565 trim_cache size 0 max 16384 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.546+0000 7f0f0a7a5740 1 Processor -- start 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 -- start start 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x557075807aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x5570757fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b3960 con 0x55707580bc00 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b52e0 con 0x5570758026d0 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b3fc0 con 0x557075803080 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x557075807aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f05a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x5570757fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36634/0 (socket says 172.21.15.119:36634) 2024-03-28T18:44:25.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 -- 172.21.15.119:0/139848967 learned_addr learned my addr 172.21.15.119:0/139848967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 -- 172.21.15.119:0/139848967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 msgr2=0x557075807aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x557075807aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 -- 172.21.15.119:0/139848967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 msgr2=0x5570757fecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x5570757fecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0ef7e00640 1 -- 172.21.15.119:0/139848967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55707580e3c0 con 0x557075803080 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f05000640 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x557075807aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.547+0000 7f0f05a00640 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x5570757fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.548+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80097e0 tx=0x7f0ee8013050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88025fbd87affc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.548+0000 7f0ef5600640 1 -- 172.21.15.119:0/139848967 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee802f030 con 0x557075803080 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.548+0000 7f0ef5600640 1 -- 172.21.15.119:0/139848967 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee801dd90 con 0x557075803080 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.548+0000 7f0ef5600640 1 -- 172.21.15.119:0/139848967 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8004ae0 con 0x557075803080 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/139848967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 msgr2=0x5570757fe7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80097e0 tx=0x7f0ee8013050 comp rx=0 tx=0).stop 2024-03-28T18:44:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/139848967 shutdown_connections 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x5570757fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x5570757fe7b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/139848967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x557075807aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/139848967 >> 172.21.15.119:0/139848967 conn(0x5570757fcda0 msgr2=0x5570757fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/139848967 shutdown_connections 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/139848967 wait complete. 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 -- start start 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.549+0000 7f0f0a7a5740 1 Processor -- start 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x55707581edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x557075875dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b4c80 con 0x55707580bc00 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b4620 con 0x5570758026d0 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f001b1b20 con 0x557075803080 2024-03-28T18:44:25.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f0a7a5740 10 client.0 _ll_register_callbacks cb 0x7f0ef0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37892/0 (socket says 172.21.15.119:37892) 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 -- 172.21.15.119:0/28072447 learned_addr learned my addr 172.21.15.119:0/28072447 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0ef7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x55707581edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05a00640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x557075875dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 -- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 msgr2=0x55707581edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x55707581edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 -- 172.21.15.119:0/28072447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 msgr2=0x557075875dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x557075875dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0f05000640 1 -- 172.21.15.119:0/28072447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570758763c0 con 0x5570758026d0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.550+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x55707581edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0f05000640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005c40 tx=0x7f0ef0050c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dba71031fb9e612 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0eeea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:25.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0042040 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0ef0005160 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ee801dd90 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ef0067050 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0f0a7a5740 10 client.25571 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5570758776a0 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.551+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee801dd90 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0ef005f020 con 0x5570758026d0 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 10 client.25571 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 10 client.25571 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 20 client.25571 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557075877e20 0x55707587a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 4 client.25571 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5570756eb500 con 0x557075877e20 2024-03-28T18:44:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557075877e20 0x55707587a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0ef7e00640 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557075877e20 0x55707587a240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bfb2f649fdc5e137 server_cookie=ad9fa776e2db0caf in_seq=0 out_seq=0 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.552+0000 7f0eeea00640 10 client.25571 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0eeea00640 1 -- 172.21.15.119:0/28072447 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5570756eb500 con 0x557075877e20 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0eeea00640 10 client.25571 handle_command_reply: tid=0 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 client.25571 shutdown 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 msgr2=0x55707581e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0005c40 tx=0x7f0ef0050c10 comp rx=0 tx=0).stop 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 shutdown_connections 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557075877e20 0x55707587a240 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55707580bc00 0x557075875dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557075803080 0x55707581edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 --2- 172.21.15.119:0/28072447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5570758026d0 0x55707581e890 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0eeea00640 0 client.25571 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 >> 172.21.15.119:0/28072447 conn(0x5570757fcda0 msgr2=0x5570757fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 shutdown_connections 2024-03-28T18:44:25.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 1 -- 172.21.15.119:0/28072447 wait complete. 2024-03-28T18:44:25.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.553+0000 7f0f0a7a5740 20 client.25571 trim_cache size 0 max 16384 2024-03-28T18:44:25.558 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 msgr2=0x7f0ee40a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 0x7f0ee40a4f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f00107060 tx=0x7f0ee8005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 msgr2=0x7f0f001a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800eb10 tx=0x7f0ef800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 shutdown_connections 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ee40a2a80 0x7f0ee40a4f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f001a2e00 0x7f0f001a32b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0010cf80 0x7f0f0019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 --2- 172.21.15.119:0/1873591136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f00107eb0 0x7f0f0019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 >> 172.21.15.119:0/1873591136 conn(0x7f0f00075be0 msgr2=0x7f0f00077ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 shutdown_connections 2024-03-28T18:44:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.554+0000 7f0f07a00640 1 -- 172.21.15.119:0/1873591136 wait complete. 2024-03-28T18:44:25.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:25 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3217767249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:25.570 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:25.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:44:25.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 -- 172.21.15.119:0/993784049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 msgr2=0x7fd9b4103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 --2- 172.21.15.119:0/993784049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b4103cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009930 tx=0x7fd99c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:25.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 -- 172.21.15.119:0/993784049 shutdown_connections 2024-03-28T18:44:25.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 --2- 172.21.15.119:0/993784049 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4104200 0x7fd9b4110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 --2- 172.21.15.119:0/993784049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b4103cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 --2- 172.21.15.119:0/993784049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4109850 0x7fd9b4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 -- 172.21.15.119:0/993784049 >> 172.21.15.119:0/993784049 conn(0x7fd9b40ff2f0 msgr2=0x7fd9b4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 -- 172.21.15.119:0/993784049 shutdown_connections 2024-03-28T18:44:25.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.780+0000 7fd9bae00640 1 -- 172.21.15.119:0/993784049 wait complete. 2024-03-28T18:44:25.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.781+0000 7fd9bae00640 1 Processor -- start 2024-03-28T18:44:25.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.781+0000 7fd9bae00640 1 -- start start 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4104200 0x7fd9b419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 0x7fd9b419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b4113cb0 con 0x7fd9b4104200 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b4113b30 con 0x7fd9b4109850 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9bae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b4113e30 con 0x7fd9b4103840 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36672/0 (socket says 172.21.15.119:36672) 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9b3e00640 1 -- 172.21.15.119:0/483766104 learned_addr learned my addr 172.21.15.119:0/483766104 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 0x7fd9b419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.782+0000 7fd9abe00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4104200 0x7fd9b419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 -- 172.21.15.119:0/483766104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 msgr2=0x7fd9b419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 0x7fd9b419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 -- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4104200 msgr2=0x7fd9b419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4104200 0x7fd9b419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 -- 172.21.15.119:0/483766104 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b419f1a0 con 0x7fd9b4103840 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 0x7fd9b419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b3e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d950 tx=0x7fd9a400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b1600640 1 -- 172.21.15.119:0/483766104 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4017070 con 0x7fd9b4103840 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b419f490 con 0x7fd9b4103840 2024-03-28T18:44:25.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b1600640 1 -- 172.21.15.119:0/483766104 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9a4009c30 con 0x7fd9b4103840 2024-03-28T18:44:25.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9b1600640 1 -- 172.21.15.119:0/483766104 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a400f6b0 con 0x7fd9b4103840 2024-03-28T18:44:25.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.783+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b41a6d20 con 0x7fd9b4103840 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.786+0000 7fd9bdbc1740 1 Processor -- start 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.786+0000 7fd9bdbc1740 1 -- start start 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b1600640 1 -- 172.21.15.119:0/483766104 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7fd9a40109d0 con 0x7fd9b4103840 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b53050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744aab2c0 con 0x55e744b50c00 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744a53c90 con 0x55e744b476d0 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744aa7360 con 0x55e744b48080 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b53050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37914/0 (socket says 172.21.15.119:37914) 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 -- 172.21.15.119:0/325074373 learned_addr learned my addr 172.21.15.119:0/325074373 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 -- 172.21.15.119:0/325074373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 msgr2=0x55e744b50630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 -- 172.21.15.119:0/325074373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 msgr2=0x55e744b53050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b53050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.787+0000 7fd9b8e00640 1 -- 172.21.15.119:0/325074373 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e744a78080 con 0x55e744b476d0 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0117e0 tx=0x7fd9ac011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4baf9ffabfd6d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9a9600640 1 -- 172.21.15.119:0/325074373 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac011310 con 0x55e744b476d0 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9a9600640 1 -- 172.21.15.119:0/325074373 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9ac003e50 con 0x55e744b476d0 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9a9600640 1 -- 172.21.15.119:0/325074373 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac0045b0 con 0x55e744b476d0 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9b1600640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 0x7fd9980a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9b1600640 1 -- 172.21.15.119:0/483766104 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd9a40cebd0 con 0x7fd9b4103840 2024-03-28T18:44:25.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.788+0000 7fd9b3e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 0x7fd9980a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/325074373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 msgr2=0x55e744b47ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0117e0 tx=0x7fd9ac011ca0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/325074373 shutdown_connections 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b53050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/325074373 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b47ab0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/325074373 >> 172.21.15.119:0/325074373 conn(0x55e744b41da0 msgr2=0x55e744b42180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/325074373 shutdown_connections 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/325074373 wait complete. 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9b3e00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 0x7fd9980a4ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40cc200 tx=0x7fd9a40d25d0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.789+0000 7fd9bdbc1740 1 -- start start 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 Processor -- start 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b50500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744b53b70 con 0x55e744b50c00 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744a670d0 con 0x55e744b476d0 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e744b53cf0 con 0x55e744b48080 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9bdbc1740 10 client.0 _ll_register_callbacks cb 0x55e744a6be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36690/0 (socket says 172.21.15.119:36690) 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9abe00640 1 -- 172.21.15.119:0/2132165362 learned_addr learned my addr 172.21.15.119:0/2132165362 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.790+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b50500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 -- 172.21.15.119:0/2132165362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 msgr2=0x55e744b50500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b50500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 -- 172.21.15.119:0/2132165362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 msgr2=0x55e744b4de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 -- 172.21.15.119:0/2132165362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7447bf3f0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9abe00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006010 tx=0x7fd99c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81c46e5607403783 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9a2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c02cbd0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9ac003e50 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.791+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd99c0046c0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd99c0048c0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 10 client.25606 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e7448897d0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9ac003e50 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd99c0035f0 con 0x55e744b48080 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 10 client.25606 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 10 client.25606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 20 client.25606 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744b60a60 0x55e744b62e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 4 client.25606 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:25.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e744ab8800 con 0x55e744b60a60 2024-03-28T18:44:25.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.792+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744b60a60 0x55e744b62e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.793+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744b60a60 0x55e744b62e50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c6df1e8ba92e9a3 server_cookie=6932f46dfbd0a5f3 in_seq=0 out_seq=0 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.793+0000 7fd9a2000640 10 client.25606 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9a2000640 1 -- 172.21.15.119:0/2132165362 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e744ab8800 con 0x55e744b60a60 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9a2000640 10 client.25606 handle_command_reply: tid=0 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 client.25606 shutdown 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 msgr2=0x55e744b4d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006010 tx=0x7fd99c009560 comp rx=0 tx=0).stop 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 shutdown_connections 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744b60a60 0x55e744b62e50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b4d890 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9a2000640 0 client.25606 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/2132165362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b50500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 >> 172.21.15.119:0/2132165362 conn(0x55e744b41da0 msgr2=0x55e744b43af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 shutdown_connections 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/2132165362 wait complete. 2024-03-28T18:44:25.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.794+0000 7fd9bdbc1740 20 client.25606 trim_cache size 0 max 16384 2024-03-28T18:44:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 Processor -- start 2024-03-28T18:44:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 -- start start 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b4ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41af640 con 0x55e744b50c00 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41b0fc0 con 0x55e744b476d0 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41afca0 con 0x55e744b48080 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b4ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b4ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37924/0 (socket says 172.21.15.119:37924) 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.808+0000 7fd9b8e00640 1 -- 172.21.15.119:0/3851442609 learned_addr learned my addr 172.21.15.119:0/3851442609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 -- 172.21.15.119:0/3851442609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 msgr2=0x55e744b4ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b4ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 -- 172.21.15.119:0/3851442609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 msgr2=0x55e744b4aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 -- 172.21.15.119:0/3851442609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e744b4c840 con 0x55e744b48080 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9abe00640 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02c170 tx=0x7fd99c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deef7d867e79918b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9a8c00640 1 -- 172.21.15.119:0/3851442609 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c057070 con 0x55e744b48080 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9a8c00640 1 -- 172.21.15.119:0/3851442609 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd99c003bb0 con 0x55e744b48080 2024-03-28T18:44:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.809+0000 7fd9a8c00640 1 -- 172.21.15.119:0/3851442609 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0053c0 con 0x55e744b48080 2024-03-28T18:44:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/3851442609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 msgr2=0x55e744b50500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02c170 tx=0x7fd99c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/3851442609 shutdown_connections 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b4aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b48080 0x55e744b50500 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/3851442609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b476d0 0x55e744b4ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/3851442609 >> 172.21.15.119:0/3851442609 conn(0x55e744b41da0 msgr2=0x55e744b40eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/3851442609 shutdown_connections 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.810+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/3851442609 wait complete. 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 -- start start 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 Processor -- start 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 0x55e744b58920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b56ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41b0960 con 0x55e744b50c00 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41b0300 con 0x55e744b48080 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b41ad800 con 0x55e744b476d0 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.811+0000 7fd9bdbc1740 10 client.0 _ll_register_callbacks cb 0x7fd99c005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 0x55e744b58920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37940/0 (socket says 172.21.15.119:37940) 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 -- 172.21.15.119:0/1540558559 learned_addr learned my addr 172.21.15.119:0/1540558559 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9b8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 0x55e744b58920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36708/0 (socket says 172.21.15.119:36708) 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b56ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 -- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 msgr2=0x55e744b58920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 0x55e744b58920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 -- 172.21.15.119:0/1540558559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 msgr2=0x55e744b56ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b56ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 -- 172.21.15.119:0/1540558559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e744b576c0 con 0x55e744b48080 2024-03-28T18:44:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9abe00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd99c000c00 tx=0x7fd99c0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23d41fe3e7dc79f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9a2000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.812+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c057070 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd99c003bb0 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd99c004ee0 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd99c069050 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 10 client.25589 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e744bc5700 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd99c003bb0 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd99c04b840 con 0x55e744b48080 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 10 client.25589 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 10 client.25589 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 20 client.25589 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744bc5d60 0x55e744bc8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 4 client.25589 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:25.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.813+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e744ab2b50 con 0x55e744bc5d60 2024-03-28T18:44:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744bc5d60 0x55e744bc8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9b8e00640 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744bc5d60 0x55e744bc8180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca994b8701fb579 server_cookie=24557598bbe1b026 in_seq=0 out_seq=0 2024-03-28T18:44:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9a2000640 10 client.25589 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9a2000640 1 -- 172.21.15.119:0/1540558559 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e744ab2b50 con 0x55e744bc5d60 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9a2000640 10 client.25589 handle_command_reply: tid=0 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 client.25589 shutdown 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 msgr2=0x55e744b58e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd99c000c00 tx=0x7fd99c0038a0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 shutdown_connections 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55e744bc5d60 0x55e744bc8180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e744b50c00 0x55e744b56ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e744b48080 0x55e744b58e60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 --2- 172.21.15.119:0/1540558559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e744b476d0 0x55e744b58920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9a2000640 0 client.25589 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:25.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.814+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 >> 172.21.15.119:0/1540558559 conn(0x55e744b41da0 msgr2=0x55e744b48460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 shutdown_connections 2024-03-28T18:44:25.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bdbc1740 1 -- 172.21.15.119:0/1540558559 wait complete. 2024-03-28T18:44:25.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bdbc1740 20 client.25589 trim_cache size 0 max 16384 2024-03-28T18:44:25.819 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 msgr2=0x7fd9980a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 0x7fd9980a4ec0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40cc200 tx=0x7fd9a40d25d0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 msgr2=0x7fd9b419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d950 tx=0x7fd9a400de10 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 shutdown_connections 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd9980a2a00 0x7fd9980a4ec0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd9b4109850 0x7fd9b419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b4104200 0x7fd9b419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 --2- 172.21.15.119:0/483766104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9b4103840 0x7fd9b419a1b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.815+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 >> 172.21.15.119:0/483766104 conn(0x7fd9b40ff2f0 msgr2=0x7fd9b40ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:25.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.816+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 shutdown_connections 2024-03-28T18:44:25.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:25.816+0000 7fd9bae00640 1 -- 172.21.15.119:0/483766104 wait complete. 2024-03-28T18:44:25.834 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:25.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:44:25.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:25 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3217767249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:25.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:25 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3217767249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:26.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 -- 172.21.15.119:0/3840728967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 msgr2=0x7f42f01148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 --2- 172.21.15.119:0/3840728967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01148d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009c10 tx=0x7f42e802bee0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 -- 172.21.15.119:0/3840728967 shutdown_connections 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 --2- 172.21.15.119:0/3840728967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01148d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 --2- 172.21.15.119:0/3840728967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f00764f0 0x7f42f0111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 --2- 172.21.15.119:0/3840728967 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f0075bd0 0x7f42f0075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 -- 172.21.15.119:0/3840728967 >> 172.21.15.119:0/3840728967 conn(0x7f42f00ff2f0 msgr2=0x7f42f0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.050+0000 7f42f6c00640 1 -- 172.21.15.119:0/3840728967 shutdown_connections 2024-03-28T18:44:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.051+0000 7f42f6c00640 1 -- 172.21.15.119:0/3840728967 wait complete. 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.051+0000 7f42f6c00640 1 Processor -- start 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 -- start start 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f0075bd0 0x7f42f019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f00764f0 0x7f42f019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0118090 con 0x7f42f0075bd0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0117f10 con 0x7f42f00764f0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f6c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0118210 con 0x7f42f01124e0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42efe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f0075bd0 0x7f42f019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36726/0 (socket says 172.21.15.119:36726) 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42f4c00640 1 -- 172.21.15.119:0/563013518 learned_addr learned my addr 172.21.15.119:0/563013518 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.052+0000 7f42e7e00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f00764f0 0x7f42f019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 -- 172.21.15.119:0/563013518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f00764f0 msgr2=0x7f42f019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f00764f0 0x7f42f019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 -- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f0075bd0 msgr2=0x7f42f019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f0075bd0 0x7f42f019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 -- 172.21.15.119:0/563013518 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f01a35a0 con 0x7f42f01124e0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f4c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033420 tx=0x7f42e8033a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42ed600640 1 -- 172.21.15.119:0/563013518 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8043020 con 0x7f42f01124e0 2024-03-28T18:44:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42ed600640 1 -- 172.21.15.119:0/563013518 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42e8003b80 con 0x7f42f01124e0 2024-03-28T18:44:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42ed600640 1 -- 172.21.15.119:0/563013518 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e803e6b0 con 0x7f42f01124e0 2024-03-28T18:44:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f01a3830 con 0x7f42f01124e0 2024-03-28T18:44:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.053+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f01ab120 con 0x7f42f01124e0 2024-03-28T18:44:26.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.056+0000 7f42ed600640 1 -- 172.21.15.119:0/563013518 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f42e8046020 con 0x7f42f01124e0 2024-03-28T18:44:26.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.056+0000 7f42f9a13740 1 Processor -- start 2024-03-28T18:44:26.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 -- start start 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfa9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8df0d2c0 con 0x557e8dfb2c00 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8deb5c90 con 0x557e8dfa96d0 2024-03-28T18:44:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42f9a13740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8df09360 con 0x557e8dfaa080 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36732/0 (socket says 172.21.15.119:36732) 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfa9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 -- 172.21.15.119:0/2401412666 learned_addr learned my addr 172.21.15.119:0/2401412666 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 -- 172.21.15.119:0/2401412666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 msgr2=0x557e8dfa9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfa9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 -- 172.21.15.119:0/2401412666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 msgr2=0x557e8dfb5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.057+0000 7f42efe00640 1 -- 172.21.15.119:0/2401412666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e8deda080 con 0x557e8dfaa080 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42efe00640 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009cd0 tx=0x7f42e0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98be5a608f47e99d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42ed600640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 0x7f42d40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42ed600640 1 -- 172.21.15.119:0/563013518 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f42e80f17d0 con 0x7f42f01124e0 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42e5200640 1 -- 172.21.15.119:0/2401412666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e001e070 con 0x557e8dfaa080 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42e5200640 1 -- 172.21.15.119:0/2401412666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42e0019b10 con 0x557e8dfaa080 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42e5200640 1 -- 172.21.15.119:0/2401412666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e000eb50 con 0x557e8dfaa080 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.058+0000 7f42e7e00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 0x7f42d40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- 172.21.15.119:0/2401412666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 msgr2=0x557e8dfb2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009cd0 tx=0x7f42e0011810 comp rx=0 tx=0).stop 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42e7e00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 0x7f42d40a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f42f00786d0 tx=0x7f42d8004730 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- 172.21.15.119:0/2401412666 shutdown_connections 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb2630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2401412666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfa9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- 172.21.15.119:0/2401412666 >> 172.21.15.119:0/2401412666 conn(0x557e8dfa3da0 msgr2=0x557e8dfa4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- 172.21.15.119:0/2401412666 shutdown_connections 2024-03-28T18:44:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- 172.21.15.119:0/2401412666 wait complete. 2024-03-28T18:44:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.059+0000 7f42f9a13740 1 -- start start 2024-03-28T18:44:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 Processor -- start 2024-03-28T18:44:26.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 0x557e8dfb2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfaa080 0x557e8dfaf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 0x557e8dfafe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8dfb5b70 con 0x557e8dfaa080 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8dec90d0 con 0x557e8dfb2c00 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e8dfb5cf0 con 0x557e8dfa96d0 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f9a13740 10 client.0 _ll_register_callbacks cb 0x557e8decde80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 0x557e8dfb2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42f4c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 0x557e8dfafe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 0x557e8dfb2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36740/0 (socket says 172.21.15.119:36740) 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfaa080 0x557e8dfaf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.060+0000 7f42efe00640 1 -- 172.21.15.119:0/3205486365 learned_addr learned my addr 172.21.15.119:0/3205486365 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 -- 172.21.15.119:0/3205486365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 msgr2=0x557e8dfb2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 0x557e8dfb2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 -- 172.21.15.119:0/3205486365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfaa080 msgr2=0x557e8dfaf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfaa080 0x557e8dfaf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 -- 172.21.15.119:0/3205486365 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e8dc213f0 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f4c00640 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 0x557e8dfafe10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42e80eeeb0 tx=0x7f42e80bd850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce00f4d709942ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42dd600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e80ef250 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f42e8004b40 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42e0019b10 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42e80bdd10 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f9a13740 10 client.25601 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.061+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557e8dceb7d0 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42e0019b10 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f42e80bd160 con 0x557e8dfb2c00 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 10 client.25601 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 20 client.25601 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:26.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8dfc2a00 0x557e8dfc4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 4 client.25601 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e8df1a800 con 0x557e8dfc2a00 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.062+0000 7f42efe00640 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8dfc2a00 0x557e8dfc4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.063+0000 7f42efe00640 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8dfc2a00 0x557e8dfc4e40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d7997e23de03574 server_cookie=9dc3a3c3b75d1007 in_seq=0 out_seq=0 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.063+0000 7f42dd600640 10 client.25601 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.063+0000 7f42dd600640 1 -- 172.21.15.119:0/3205486365 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557e8df1a800 con 0x557e8dfc2a00 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42dd600640 10 client.25601 handle_command_reply: tid=0 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 client.25601 shutdown 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 msgr2=0x557e8dfafe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 0x557e8dfafe10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42e80eeeb0 tx=0x7f42e80bd850 comp rx=0 tx=0).stop 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 shutdown_connections 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8dfc2a00 0x557e8dfc4e40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfb2c00 0x557e8dfafe10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfaa080 0x557e8dfaf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42dd600640 0 client.25601 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:26.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 --2- 172.21.15.119:0/3205486365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfa96d0 0x557e8dfb2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 >> 172.21.15.119:0/3205486365 conn(0x557e8dfa3da0 msgr2=0x557e8dfa5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 shutdown_connections 2024-03-28T18:44:26.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 1 -- 172.21.15.119:0/3205486365 wait complete. 2024-03-28T18:44:26.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.064+0000 7f42f9a13740 20 client.25601 trim_cache size 0 max 16384 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 Processor -- start 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 -- start start 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b3950 con 0x557e8dfb2c00 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b52d0 con 0x557e8dfa96d0 2024-03-28T18:44:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f9a13740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b3fb0 con 0x557e8dfaa080 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42f4c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.079+0000 7f42efe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36748/0 (socket says 172.21.15.119:36748) 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 -- 172.21.15.119:0/2327827078 learned_addr learned my addr 172.21.15.119:0/2327827078 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42efe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37968/0 (socket says 172.21.15.119:37968) 2024-03-28T18:44:26.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42f4c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41112/0 (socket says 172.21.15.119:41112) 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 -- 172.21.15.119:0/2327827078 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 msgr2=0x557e8dfae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 -- 172.21.15.119:0/2327827078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 msgr2=0x557e8dfb2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 -- 172.21.15.119:0/2327827078 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e8dfadde0 con 0x557e8dfaa080 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42efe00640 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42f4c00640 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.080+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42d8020930 tx=0x7f42d801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f2773379193634 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42dfe00640 1 -- 172.21.15.119:0/2327827078 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d802f070 con 0x557e8dfaa080 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42dfe00640 1 -- 172.21.15.119:0/2327827078 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42d801fb50 con 0x557e8dfaa080 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42dfe00640 1 -- 172.21.15.119:0/2327827078 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d801d610 con 0x557e8dfaa080 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 -- 172.21.15.119:0/2327827078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 msgr2=0x557e8dfb1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42d8020930 tx=0x7f42d801c000 comp rx=0 tx=0).stop 2024-03-28T18:44:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 -- 172.21.15.119:0/2327827078 shutdown_connections 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfb2c00 0x557e8dfb2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfaa080 0x557e8dfb1fc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2327827078 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfa96d0 0x557e8dfae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 -- 172.21.15.119:0/2327827078 >> 172.21.15.119:0/2327827078 conn(0x557e8dfa3da0 msgr2=0x557e8dfa2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 -- 172.21.15.119:0/2327827078 shutdown_connections 2024-03-28T18:44:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.081+0000 7f42f9a13740 1 -- 172.21.15.119:0/2327827078 wait complete. 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 -- start start 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 Processor -- start 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 0x557e8dfb8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfb2c00 0x557e8e01d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b4c70 con 0x557e8dfa96d0 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b4610 con 0x557e8dfaa080 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f01b1b10 con 0x557e8dfb2c00 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.082+0000 7f42f9a13740 10 client.0 _ll_register_callbacks cb 0x7f42e80eed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37978/0 (socket says 172.21.15.119:37978) 2024-03-28T18:44:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 -- 172.21.15.119:0/2045334115 learned_addr learned my addr 172.21.15.119:0/2045334115 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42f4c00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfb2c00 0x557e8e01d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42efe00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 0x557e8dfb8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 -- 172.21.15.119:0/2045334115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfb2c00 msgr2=0x557e8e01d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfb2c00 0x557e8e01d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 -- 172.21.15.119:0/2045334115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 msgr2=0x557e8dfb8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 0x557e8dfb8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 -- 172.21.15.119:0/2045334115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e8e01d850 con 0x557e8dfaa080 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42efe00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 0x557e8dfb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42e7e00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42d8010c90 tx=0x7f42d802aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de067bb2ff2fcef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42dd600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d802f040 con 0x557e8dfaa080 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42d801fb50 con 0x557e8dfaa080 2024-03-28T18:44:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.083+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f42d802a620 con 0x557e8dfaa080 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42d802a840 con 0x557e8dfaa080 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 10 client.25607 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557e8e01eb50 con 0x557e8dfaa080 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42d801fb50 con 0x557e8dfaa080 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f42d8032070 con 0x557e8dfaa080 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 10 client.25607 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 10 client.25607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 20 client.25607 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8e01f0c0 0x557e8e0214e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 4 client.25607 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.084+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557e8df14b50 con 0x557e8e01f0c0 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42efe00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8e01f0c0 0x557e8e0214e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42efe00640 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8e01f0c0 0x557e8e0214e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b429da30fec2eb28 server_cookie=47b17bae0b991a6f in_seq=0 out_seq=0 2024-03-28T18:44:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42dd600640 10 client.25607 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42dd600640 1 -- 172.21.15.119:0/2045334115 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557e8df14b50 con 0x557e8e01f0c0 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42dd600640 10 client.25607 handle_command_reply: tid=0 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 client.25607 shutdown 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 msgr2=0x557e8dfb8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42d8010c90 tx=0x7f42d802aa80 comp rx=0 tx=0).stop 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 shutdown_connections 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557e8e01f0c0 0x557e8e0214e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557e8dfb2c00 0x557e8e01d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42dd600640 0 client.25607 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557e8dfaa080 0x557e8dfb8ff0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 --2- 172.21.15.119:0/2045334115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557e8dfa96d0 0x557e8dfb8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 >> 172.21.15.119:0/2045334115 conn(0x557e8dfa3da0 msgr2=0x557e8dfaa460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.085+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 shutdown_connections 2024-03-28T18:44:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f9a13740 1 -- 172.21.15.119:0/2045334115 wait complete. 2024-03-28T18:44:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f9a13740 20 client.25607 trim_cache size 0 max 16384 2024-03-28T18:44:26.091 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 msgr2=0x7f42d40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 0x7f42d40a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f42f00786d0 tx=0x7f42d8004730 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 msgr2=0x7f42f01a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033420 tx=0x7f42e8033a20 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 shutdown_connections 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42d40a2a00 0x7f42d40a4ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f42f01124e0 0x7f42f01a2e70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42f00764f0 0x7f42f019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 --2- 172.21.15.119:0/563013518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42f0075bd0 0x7f42f019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.086+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 >> 172.21.15.119:0/563013518 conn(0x7f42f00ff2f0 msgr2=0x7f42f01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.087+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 shutdown_connections 2024-03-28T18:44:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:26.087+0000 7f42f6c00640 1 -- 172.21.15.119:0/563013518 wait complete. 2024-03-28T18:44:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:26 smithi119 ceph-mon[31813]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:44:26.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:26 smithi196 ceph-mon[36300]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:44:26.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:26 smithi131 ceph-mon[36065]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:44:27.103 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:44:27.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:44:27.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 -- 172.21.15.119:0/1510443479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44074360 msgr2=0x7f0a441120f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 --2- 172.21.15.119:0/1510443479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44074360 0x7f0a441120f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0a34009900 tx=0x7f0a3402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 -- 172.21.15.119:0/1510443479 shutdown_connections 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 --2- 172.21.15.119:0/1510443479 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44112630 0x7f0a44114a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 --2- 172.21.15.119:0/1510443479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44074360 0x7f0a441120f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 --2- 172.21.15.119:0/1510443479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44073a40 0x7f0a44073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.331+0000 7f0a4c800640 1 -- 172.21.15.119:0/1510443479 >> 172.21.15.119:0/1510443479 conn(0x7f0a440fd130 msgr2=0x7f0a440ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.332+0000 7f0a4c800640 1 -- 172.21.15.119:0/1510443479 shutdown_connections 2024-03-28T18:44:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.332+0000 7f0a4c800640 1 -- 172.21.15.119:0/1510443479 wait complete. 2024-03-28T18:44:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 Processor -- start 2024-03-28T18:44:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 -- start start 2024-03-28T18:44:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44074360 0x7f0a4419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 0x7f0a441a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44118080 con 0x7f0a44074360 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44117f00 con 0x7f0a44073a40 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.333+0000 7f0a4c800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44118200 con 0x7f0a44112630 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 0x7f0a441a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37982/0 (socket says 172.21.15.119:37982) 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a49e00640 1 -- 172.21.15.119:0/3679962828 learned_addr learned my addr 172.21.15.119:0/3679962828 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a49400640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44074360 0x7f0a4419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 -- 172.21.15.119:0/3679962828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 msgr2=0x7f0a4419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 -- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44074360 msgr2=0x7f0a4419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44074360 0x7f0a4419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 -- 172.21.15.119:0/3679962828 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a441a3570 con 0x7f0a44112630 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a49e00640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.334+0000 7f0a4a800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 0x7f0a441a2e40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000e940 tx=0x7f0a4000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.335+0000 7f0a3aa00640 1 -- 172.21.15.119:0/3679962828 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4000ccc0 con 0x7f0a44112630 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.335+0000 7f0a3aa00640 1 -- 172.21.15.119:0/3679962828 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a400035f0 con 0x7f0a44112630 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.335+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a441a3860 con 0x7f0a44112630 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.335+0000 7f0a3aa00640 1 -- 172.21.15.119:0/3679962828 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4000ce60 con 0x7f0a44112630 2024-03-28T18:44:27.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.335+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a441ab0f0 con 0x7f0a44112630 2024-03-28T18:44:27.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.338+0000 7f0a4f401740 1 Processor -- start 2024-03-28T18:44:27.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.338+0000 7f0a3aa00640 1 -- 172.21.15.119:0/3679962828 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f0a400037a0 con 0x7f0a44112630 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.338+0000 7f0a4f401740 1 -- start start 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d5b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d66970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664cbfeb0 con 0x564664d64580 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664c68a60 con 0x564664d5b080 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a4f401740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664cbbf50 con 0x564664d5b9a0 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36776/0 (socket says 172.21.15.119:36776) 2024-03-28T18:44:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a49e00640 1 -- 172.21.15.119:0/1379956970 learned_addr learned my addr 172.21.15.119:0/1379956970 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a49400640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d5b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.339+0000 7f0a49400640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d66970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 -- 172.21.15.119:0/1379956970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 msgr2=0x564664d5b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d5b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 -- 172.21.15.119:0/1379956970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 msgr2=0x564664d66970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d66970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 -- 172.21.15.119:0/1379956970 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564664a9c7e0 con 0x564664d5b9a0 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49400640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d66970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49e00640 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00e8e0 tx=0x7f0a3c00eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e37a33bb49d43fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a3aa00640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 0x7f0a300a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a2f000640 1 -- 172.21.15.119:0/1379956970 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c004d20 con 0x564664d5b9a0 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a2f000640 1 -- 172.21.15.119:0/1379956970 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3c010e90 con 0x564664d5b9a0 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a2f000640 1 -- 172.21.15.119:0/1379956970 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c01a6e0 con 0x564664d5b9a0 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a3aa00640 1 -- 172.21.15.119:0/3679962828 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0a400cf1c0 con 0x7f0a44112630 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.340+0000 7f0a49400640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 0x7f0a300a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 -- 172.21.15.119:0/1379956970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 msgr2=0x564664d64040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00e8e0 tx=0x7f0a3c00eda0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 -- 172.21.15.119:0/1379956970 shutdown_connections 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d66970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d64040 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 --2- 172.21.15.119:0/1379956970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d5b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 -- 172.21.15.119:0/1379956970 >> 172.21.15.119:0/1379956970 conn(0x564664d557b0 msgr2=0x564664d55b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 -- 172.21.15.119:0/1379956970 shutdown_connections 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a49400640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 0x7f0a300a4ec0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a34005ab0 tx=0x7f0a34005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.341+0000 7f0a4f401740 1 -- 172.21.15.119:0/1379956970 wait complete. 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 -- start start 2024-03-28T18:44:27.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 Processor -- start 2024-03-28T18:44:27.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664d67370 con 0x564664d64580 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664c7beb0 con 0x564664d5b080 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564664d674f0 con 0x564664d5b9a0 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a4f401740 10 client.0 _ll_register_callbacks cb 0x564664c70150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a49400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a49400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d612e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36782/0 (socket says 172.21.15.119:36782) 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38008/0 (socket says 172.21.15.119:38008) 2024-03-28T18:44:27.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.342+0000 7f0a49400640 1 -- 172.21.15.119:0/2430272682 learned_addr learned my addr 172.21.15.119:0/2430272682 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a4a800640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 -- 172.21.15.119:0/2430272682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 msgr2=0x564664d612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 -- 172.21.15.119:0/2430272682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 msgr2=0x564664d618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d618f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 -- 172.21.15.119:0/2430272682 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564664afcf70 con 0x564664d5b080 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a49e00640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000f80 tx=0x7f0a3c005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cbb6561930e8e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a23400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c027020 con 0x564664d5b080 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0a3c003970 con 0x564664d5b080 2024-03-28T18:44:27.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.343+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a3c010e90 con 0x564664d5b080 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a3c02f050 con 0x564664d5b080 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 10 client.25619 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564664994170 con 0x564664d5b080 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a3c010e90 con 0x564664d5b080 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0a3c022070 con 0x564664d5b080 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 10 client.25619 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:27.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 20 client.25619 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664d745d0 0x564664d769c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 4 client.25619 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.344+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564664ccd3f0 con 0x564664d745d0 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.345+0000 7f0a49400640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664d745d0 0x564664d769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.345+0000 7f0a49400640 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664d745d0 0x564664d769c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c1c72c8eef6ca0f server_cookie=a588157dfc586768 in_seq=0 out_seq=0 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.345+0000 7f0a23400640 10 client.25619 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a23400640 1 -- 172.21.15.119:0/2430272682 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564664ccd3f0 con 0x564664d745d0 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a23400640 10 client.25619 handle_command_reply: tid=0 2024-03-28T18:44:27.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 client.25619 shutdown 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 msgr2=0x564664d63f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000f80 tx=0x7f0a3c005870 comp rx=0 tx=0).stop 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 shutdown_connections 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664d745d0 0x564664d769c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d64580 0x564664d618f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b9a0 0x564664d612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 --2- 172.21.15.119:0/2430272682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63f50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 >> 172.21.15.119:0/2430272682 conn(0x564664d557b0 msgr2=0x564664d57500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a23400640 0 client.25619 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 shutdown_connections 2024-03-28T18:44:27.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.346+0000 7f0a4f401740 1 -- 172.21.15.119:0/2430272682 wait complete. 2024-03-28T18:44:27.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.347+0000 7f0a4f401740 20 client.25619 trim_cache size 0 max 16384 2024-03-28T18:44:27.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.364+0000 7f0a4f401740 1 Processor -- start 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.364+0000 7f0a4f401740 1 -- start start 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d63f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b39c0 con 0x564664d5b9a0 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b5340 con 0x564664d5b080 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4f401740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b4020 con 0x564664d64580 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36792/0 (socket says 172.21.15.119:36792) 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 -- 172.21.15.119:0/748063714 learned_addr learned my addr 172.21.15.119:0/748063714 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a49400640 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d63f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a49e00640 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 -- 172.21.15.119:0/748063714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 msgr2=0x564664d63a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 -- 172.21.15.119:0/748063714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 msgr2=0x564664d63f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d63f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.365+0000 7f0a4a800640 1 -- 172.21.15.119:0/748063714 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564664d61ef0 con 0x564664d64580 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4a800640 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a400cc910 tx=0x7f0a4009b520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fec6f20c73623496 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a2e600640 1 -- 172.21.15.119:0/748063714 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4009aff0 con 0x564664d64580 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a2e600640 1 -- 172.21.15.119:0/748063714 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a40003b10 con 0x564664d64580 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a2e600640 1 -- 172.21.15.119:0/748063714 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40099a20 con 0x564664d64580 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 -- 172.21.15.119:0/748063714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 msgr2=0x564664d546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a400cc910 tx=0x7f0a4009b520 comp rx=0 tx=0).stop 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 -- 172.21.15.119:0/748063714 shutdown_connections 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d64580 0x564664d546b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d63f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 --2- 172.21.15.119:0/748063714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d5b080 0x564664d63a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.366+0000 7f0a4f401740 1 -- 172.21.15.119:0/748063714 >> 172.21.15.119:0/748063714 conn(0x564664d557b0 msgr2=0x564664d53fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 -- 172.21.15.119:0/748063714 shutdown_connections 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 -- 172.21.15.119:0/748063714 wait complete. 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 -- start start 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 Processor -- start 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.367+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d69810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4f401740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 0x564664dcfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4f401740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b4ce0 con 0x564664d5b9a0 2024-03-28T18:44:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4f401740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b4680 con 0x564664d64580 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4f401740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441b1b80 con 0x564664d5b080 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4f401740 10 client.0 _ll_register_callbacks cb 0x7f0a34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36802/0 (socket says 172.21.15.119:36802) 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 -- 172.21.15.119:0/3900934125 learned_addr learned my addr 172.21.15.119:0/3900934125 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4a800640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 0x564664dcfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49400640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d69810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 -- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 msgr2=0x564664dcfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 0x564664dcfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 -- 172.21.15.119:0/3900934125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 msgr2=0x564664d69810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d69810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 -- 172.21.15.119:0/3900934125 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564664d5a3b0 con 0x564664d5b080 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a4a800640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 0x564664dcfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a49e00640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c01d040 tx=0x7f0a3c0094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51ec0bd57881a60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.368+0000 7f0a23400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c038060 con 0x564664d5b080 2024-03-28T18:44:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564664cc7740 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0a3c010bc0 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a3c027070 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 10 client.25648 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564664d50c20 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564664cc7740 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0a3c022040 con 0x564664d5b080 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 10 client.25648 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 20 client.25648 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664dd1690 0x564664dd3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 4 client.25648 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:44:27.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.369+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564664c45a40 con 0x564664dd1690 2024-03-28T18:44:27.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.370+0000 7f0a49400640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664dd1690 0x564664dd3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.370+0000 7f0a49400640 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664dd1690 0x564664dd3ab0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f18722e9f808724 server_cookie=9bcf2b660d1a28d9 in_seq=0 out_seq=0 2024-03-28T18:44:27.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.370+0000 7f0a23400640 10 client.25648 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.371+0000 7f0a23400640 1 -- 172.21.15.119:0/3900934125 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564664c45a40 con 0x564664dd1690 2024-03-28T18:44:27.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.371+0000 7f0a23400640 10 client.25648 handle_command_reply: tid=0 2024-03-28T18:44:27.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.371+0000 7f0a4f401740 1 client.25648 shutdown 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.371+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 msgr2=0x564664d692d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.371+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c01d040 tx=0x7f0a3c0094b0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 shutdown_connections 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564664dd1690 0x564664dd3ab0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564664d64580 0x564664dcfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564664d5b9a0 0x564664d69810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 --2- 172.21.15.119:0/3900934125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564664d5b080 0x564664d692d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a23400640 0 client.25648 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 >> 172.21.15.119:0/3900934125 conn(0x564664d557b0 msgr2=0x564664d54830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 shutdown_connections 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 1 -- 172.21.15.119:0/3900934125 wait complete. 2024-03-28T18:44:27.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4f401740 20 client.25648 trim_cache size 0 max 16384 2024-03-28T18:44:27.381 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:44:27.381 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:44:27.381 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "bb4c79a5-af1a-4966-817f-f018ca217874", 2024-03-28T18:44:27.381 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:44:27.381 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:44:27.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 msgr2=0x7f0a300a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.372+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 0x7f0a300a4ec0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a34005ab0 tx=0x7f0a34005a20 comp rx=0 tx=0).stop 2024-03-28T18:44:27.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 msgr2=0x7f0a441a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 0x7f0a441a2e40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000e940 tx=0x7f0a4000ee00 comp rx=0 tx=0).stop 2024-03-28T18:44:27.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 shutdown_connections 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a300a2a00 0x7f0a300a4ec0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a44112630 0x7f0a441a2e40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a44074360 0x7f0a4419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 --2- 172.21.15.119:0/3679962828 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a44073a40 0x7f0a4419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 >> 172.21.15.119:0/3679962828 conn(0x7f0a440fd130 msgr2=0x7f0a440fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 shutdown_connections 2024-03-28T18:44:27.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.373+0000 7f0a4c800640 1 -- 172.21.15.119:0/3679962828 wait complete. 2024-03-28T18:44:27.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:44:27.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:44:27.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 -- 172.21.15.119:0/3336720584 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294109850 msgr2=0x7fb294109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 --2- 172.21.15.119:0/3336720584 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294109850 0x7fb294109c30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb284009ca0 tx=0x7fb28402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 -- 172.21.15.119:0/3336720584 shutdown_connections 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 --2- 172.21.15.119:0/3336720584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294104200 0x7fb294110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 --2- 172.21.15.119:0/3336720584 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294103840 0x7fb294103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 --2- 172.21.15.119:0/3336720584 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294109850 0x7fb294109c30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 -- 172.21.15.119:0/3336720584 >> 172.21.15.119:0/3336720584 conn(0x7fb2940ff2f0 msgr2=0x7fb294101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 -- 172.21.15.119:0/3336720584 shutdown_connections 2024-03-28T18:44:27.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.606+0000 7fb29a000640 1 -- 172.21.15.119:0/3336720584 wait complete. 2024-03-28T18:44:27.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.607+0000 7fb29a000640 1 Processor -- start 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 -- start start 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294103840 0x7fb29419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294104200 0x7fb29419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294113c20 con 0x7fb294103840 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294113aa0 con 0x7fb294104200 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb29a000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb294113da0 con 0x7fb294109850 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb293e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb293400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294103840 0x7fb29419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb292a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294104200 0x7fb29419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb293e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36814/0 (socket says 172.21.15.119:36814) 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.608+0000 7fb293e00640 1 -- 172.21.15.119:0/1843473581 learned_addr learned my addr 172.21.15.119:0/1843473581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 -- 172.21.15.119:0/1843473581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294104200 msgr2=0x7fb29419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294104200 0x7fb29419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 -- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294103840 msgr2=0x7fb29419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294103840 0x7fb29419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 -- 172.21.15.119:0/1843473581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29419f100 con 0x7fb294109850 2024-03-28T18:44:27.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb293e00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00e970 tx=0x7fb28c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb28be00640 1 -- 172.21.15.119:0/1843473581 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c00ccc0 con 0x7fb294109850 2024-03-28T18:44:27.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29419f3f0 con 0x7fb294109850 2024-03-28T18:44:27.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb28be00640 1 -- 172.21.15.119:0/1843473581 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb28c00ce60 con 0x7fb294109850 2024-03-28T18:44:27.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb28be00640 1 -- 172.21.15.119:0/1843473581 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c010490 con 0x7fb294109850 2024-03-28T18:44:27.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.609+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2941a6c80 con 0x7fb294109850 2024-03-28T18:44:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.612+0000 7fb29cc56740 1 Processor -- start 2024-03-28T18:44:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.612+0000 7fb28be00640 1 -- 172.21.15.119:0/1843473581 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7fb28c005020 con 0x7fb294109850 2024-03-28T18:44:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 -- start start 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee13950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ed78250 con 0x55a74ee1d290 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ed20a30 con 0x55a74ee14c00 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb29cc56740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ed742f0 con 0x55a74ee13f20 2024-03-28T18:44:27.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb293400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb293400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36822/0 (socket says 172.21.15.119:36822) 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb293400640 1 -- 172.21.15.119:0/412514544 learned_addr learned my addr 172.21.15.119:0/412514544 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb292a00640 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee13950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.613+0000 7fb292a00640 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 -- 172.21.15.119:0/412514544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 msgr2=0x55a74ee13950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee13950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 -- 172.21.15.119:0/412514544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 msgr2=0x55a74ee1f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 -- 172.21.15.119:0/412514544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a74ed0b690 con 0x55a74ee13f20 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb293400640 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb284004af0 tx=0x7fb284005fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba3d28f321a57ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb28be00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 0x7fb2780a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb27fe00640 1 -- 172.21.15.119:0/412514544 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284043020 con 0x55a74ee13f20 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb27fe00640 1 -- 172.21.15.119:0/412514544 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb284003970 con 0x55a74ee13f20 2024-03-28T18:44:27.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb27fe00640 1 -- 172.21.15.119:0/412514544 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284039040 con 0x55a74ee13f20 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb292a00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 0x7fb2780a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.614+0000 7fb28be00640 1 -- 172.21.15.119:0/1843473581 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb28c0cf4a0 con 0x7fb294109850 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 -- 172.21.15.119:0/412514544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 msgr2=0x55a74ee143a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb284004af0 tx=0x7fb284005fc0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb292a00640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 0x7fb2780a4ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb280009730 tx=0x7fb280005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 -- 172.21.15.119:0/412514544 shutdown_connections 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee143a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 --2- 172.21.15.119:0/412514544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee13950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 -- 172.21.15.119:0/412514544 >> 172.21.15.119:0/412514544 conn(0x55a74eddb200 msgr2=0x55a74ee0e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 -- 172.21.15.119:0/412514544 shutdown_connections 2024-03-28T18:44:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.615+0000 7fb29cc56740 1 -- 172.21.15.119:0/412514544 wait complete. 2024-03-28T18:44:27.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 -- start start 2024-03-28T18:44:27.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 Processor -- start 2024-03-28T18:44:27.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ee200a0 con 0x55a74ee1d290 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ed33e80 con 0x55a74ee14c00 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.616+0000 7fb29cc56740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a74ee20220 con 0x55a74ee13f20 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb29cc56740 10 client.0 _ll_register_callbacks cb 0x55a74ed46130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38062/0 (socket says 172.21.15.119:38062) 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb293400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 -- 172.21.15.119:0/2472248894 learned_addr learned my addr 172.21.15.119:0/2472248894 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb293e00640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 -- 172.21.15.119:0/2472248894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 msgr2=0x55a74ee1c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 -- 172.21.15.119:0/2472248894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 msgr2=0x55a74ee1a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 -- 172.21.15.119:0/2472248894 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a74eb567d0 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb293400640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb293e00640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb292a00640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb280021d60 tx=0x7fb2800096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f13aacd6c2d6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.617+0000 7fb27cc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280004050 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb284003970 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb2800041f0 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2800043f0 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb29cc56740 10 client.25637 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a74ea8c3f0 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.618+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb284003970 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb280028020 con 0x55a74ee14c00 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 10 client.25637 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 10 client.25637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 20 client.25637 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee2d130 0x55a74ee2f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 4 client.25637 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a74ed85790 con 0x55a74ee2d130 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb293400640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee2d130 0x55a74ee2f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb293400640 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee2d130 0x55a74ee2f540 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6365b51dc74d855 server_cookie=51ec239184771730 in_seq=0 out_seq=0 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.619+0000 7fb27cc00640 10 client.25637 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.620+0000 7fb27cc00640 1 -- 172.21.15.119:0/2472248894 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a74ed85790 con 0x55a74ee2d130 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.620+0000 7fb27cc00640 10 client.25637 handle_command_reply: tid=0 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.620+0000 7fb29cc56740 1 client.25637 shutdown 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 msgr2=0x55a74ee19cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb280021d60 tx=0x7fb2800096c0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 shutdown_connections 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee2d130 0x55a74ee2f540 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee1d290 0x55a74ee1a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee14c00 0x55a74ee19cf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb27cc00640 0 client.25637 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 --2- 172.21.15.119:0/2472248894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 >> 172.21.15.119:0/2472248894 conn(0x55a74eddb200 msgr2=0x55a74ee106c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 shutdown_connections 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 1 -- 172.21.15.119:0/2472248894 wait complete. 2024-03-28T18:44:27.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.621+0000 7fb29cc56740 20 client.25637 trim_cache size 0 max 16384 2024-03-28T18:44:27.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.638+0000 7fb29cc56740 1 Processor -- start 2024-03-28T18:44:27.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.638+0000 7fb29cc56740 1 -- start start 2024-03-28T18:44:27.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee1c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941af590 con 0x55a74ee14c00 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941b0f10 con 0x55a74ee1d290 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb29cc56740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941afbf0 con 0x55a74ee13f20 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38068/0 (socket says 172.21.15.119:38068) 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb292a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee1c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 -- 172.21.15.119:0/162361543 learned_addr learned my addr 172.21.15.119:0/162361543 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293400640 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 -- 172.21.15.119:0/162361543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 msgr2=0x55a74ee1c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 -- 172.21.15.119:0/162361543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 msgr2=0x55a74ee1c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee1c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293e00640 1 -- 172.21.15.119:0/162361543 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a74ea4c0f0 con 0x55a74ee1d290 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.639+0000 7fb293400640 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb293e00640 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0982a0 tx=0x7fb28c09b2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c2e8d86980dca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb27f400640 1 -- 172.21.15.119:0/162361543 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0b3020 con 0x55a74ee1d290 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb27f400640 1 -- 172.21.15.119:0/162361543 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb28c09fa20 con 0x55a74ee1d290 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb27f400640 1 -- 172.21.15.119:0/162361543 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0ccd20 con 0x55a74ee1d290 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 -- 172.21.15.119:0/162361543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 msgr2=0x55a74ee1e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0982a0 tx=0x7fb28c09b2f0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 -- 172.21.15.119:0/162361543 shutdown_connections 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee1d290 0x55a74ee1e680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee1c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 --2- 172.21.15.119:0/162361543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee13f20 0x55a74ee1c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 -- 172.21.15.119:0/162361543 >> 172.21.15.119:0/162361543 conn(0x55a74eddb200 msgr2=0x55a74ee0e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 -- 172.21.15.119:0/162361543 shutdown_connections 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.640+0000 7fb29cc56740 1 -- 172.21.15.119:0/162361543 wait complete. 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 -- start start 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 Processor -- start 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee168d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee1d290 0x55a74ee16e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.641+0000 7fb29cc56740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941b08b0 con 0x55a74ee14c00 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb29cc56740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941b0250 con 0x55a74ee13f20 2024-03-28T18:44:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb29cc56740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2941ad750 con 0x55a74ee1d290 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb29cc56740 10 client.0 _ll_register_callbacks cb 0x7fb2800219e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38074/0 (socket says 172.21.15.119:38074) 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 -- 172.21.15.119:0/3724979108 learned_addr learned my addr 172.21.15.119:0/3724979108 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293e00640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee1d290 0x55a74ee16e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb292a00640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 -- 172.21.15.119:0/3724979108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee1d290 msgr2=0x55a74ee16e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee1d290 0x55a74ee16e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 -- 172.21.15.119:0/3724979108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 msgr2=0x55a74ee168d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee168d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 -- 172.21.15.119:0/3724979108 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a74ee17d70 con 0x55a74ee13f20 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb292a00640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee168d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb293400640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb284009ca0 tx=0x7fb28402c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f5c65233899da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb27cc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.642+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284043070 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb28c09fa20 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb284032d50 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb284039040 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a74ee88b30 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb28c09fa20 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb284004030 con 0x55a74ee13f20 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 20 client.25643 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee88f10 0x55a74ee8b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 4 client.25643 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.643+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a74ed7fae0 con 0x55a74ee88f10 2024-03-28T18:44:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb292a00640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee88f10 0x55a74ee8b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb292a00640 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee88f10 0x55a74ee8b300 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe41f92ec198eb87 server_cookie=95686aed1ee8944e in_seq=0 out_seq=0 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb27cc00640 10 client.25643 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb27cc00640 1 -- 172.21.15.119:0/3724979108 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a74ed7fae0 con 0x55a74ee88f10 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb27cc00640 10 client.25643 handle_command_reply: tid=0 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb29cc56740 1 client.25643 shutdown 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 msgr2=0x55a74ee16390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.644+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb284009ca0 tx=0x7fb28402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 shutdown_connections 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a74ee88f10 0x55a74ee8b300 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a74ee1d290 0x55a74ee16e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a74ee14c00 0x55a74ee168d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb27cc00640 0 client.25643 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 --2- 172.21.15.119:0/3724979108 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a74ee13f20 0x55a74ee16390 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 >> 172.21.15.119:0/3724979108 conn(0x55a74eddb200 msgr2=0x55a74ee18550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 shutdown_connections 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 1 -- 172.21.15.119:0/3724979108 wait complete. 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29cc56740 20 client.25643 trim_cache size 0 max 16384 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "bb4c79a5-af1a-4966-817f-f018ca217874": { 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "bb4c79a5-af1a-4966-817f-f018ca217874", 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:44:27.650 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:44:27.651 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:44:27.651 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:44:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 msgr2=0x7fb2780a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 0x7fb2780a4ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb280009730 tx=0x7fb280005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 msgr2=0x7fb29419e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.645+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00e970 tx=0x7fb28c00ed70 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 shutdown_connections 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb2780a2a00 0x7fb2780a4ec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb294109850 0x7fb29419e9d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb294104200 0x7fb29419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 --2- 172.21.15.119:0/1843473581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb294103840 0x7fb29419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 >> 172.21.15.119:0/1843473581 conn(0x7fb2940ff2f0 msgr2=0x7fb2940ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 shutdown_connections 2024-03-28T18:44:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:27.646+0000 7fb29a000640 1 -- 172.21.15.119:0/1843473581 wait complete. 2024-03-28T18:44:27.662 INFO:tasks.cephfs.filesystem:scrub status for tag:bb4c79a5-af1a-4966-817f-f018ca217874 - {'path': '/', 'tag': 'bb4c79a5-af1a-4966-817f-f018ca217874', 'options': 'recursive,force'} 2024-03-28T18:44:27.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:27 smithi119 ceph-mon[31813]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:27.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:27 smithi196 ceph-mon[36300]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:27.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:27 smithi131 ceph-mon[36065]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:28 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:44:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:28 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:28 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:44:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:28 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:28 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:44:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:28 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:44:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:28 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:28 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:44:28.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:28 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:28 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:44:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:28 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:44:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:28 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:28 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:44:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:28 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:44:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:28 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:44:29.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:29 smithi119 ceph-mon[31813]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:44:29.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:29 smithi196 ceph-mon[36300]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:44:29.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:29 smithi131 ceph-mon[36065]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T18:44:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:32 smithi119 ceph-mon[31813]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:32 smithi196 ceph-mon[36300]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:32 smithi131 ceph-mon[36065]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:34 smithi131 ceph-mon[36065]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:34.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:34 smithi119 ceph-mon[31813]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:34 smithi196 ceph-mon[36300]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:36 smithi119 ceph-mon[31813]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:36 smithi196 ceph-mon[36300]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:36 smithi131 ceph-mon[36065]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:44:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:38 smithi119 ceph-mon[31813]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:38.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:38 smithi196 ceph-mon[36300]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:38 smithi131 ceph-mon[36065]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:44:40.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:40 smithi119 ceph-mon[31813]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-28T18:44:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:40 smithi196 ceph-mon[36300]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-28T18:44:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:40 smithi131 ceph-mon[36065]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-28T18:44:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:42 smithi119 ceph-mon[31813]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-28T18:44:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:42 smithi196 ceph-mon[36300]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-28T18:44:42.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:42 smithi131 ceph-mon[36065]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-28T18:44:44.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:44 smithi119 ceph-mon[31813]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:44 smithi196 ceph-mon[36300]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:44 smithi131 ceph-mon[36065]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:46 smithi119 ceph-mon[31813]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:46 smithi196 ceph-mon[36300]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:46 smithi131 ceph-mon[36065]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:48.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:48 smithi119 ceph-mon[31813]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:48 smithi196 ceph-mon[36300]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:48 smithi131 ceph-mon[36065]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:50 smithi119 ceph-mon[31813]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T18:44:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:50 smithi196 ceph-mon[36300]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T18:44:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:50 smithi131 ceph-mon[36065]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-28T18:44:52.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:52 smithi119 ceph-mon[31813]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-28T18:44:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:52 smithi196 ceph-mon[36300]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-28T18:44:52.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:52 smithi131 ceph-mon[36065]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-28T18:44:54.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:54 smithi119 ceph-mon[31813]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:54.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:54 smithi196 ceph-mon[36300]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:54 smithi131 ceph-mon[36065]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-28T18:44:56.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:56 smithi119 ceph-mon[31813]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:56.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:56 smithi196 ceph-mon[36300]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:56.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:56 smithi131 ceph-mon[36065]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-28T18:44:57.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:44:57.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:44:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 -- 172.21.15.119:0/917088934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c073a70 msgr2=0x7f1f7c073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 --2- 172.21.15.119:0/917088934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c073a70 0x7f1f7c073e50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009900 tx=0x7f1f6c02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 -- 172.21.15.119:0/917088934 shutdown_connections 2024-03-28T18:44:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 --2- 172.21.15.119:0/917088934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c10ddb0 0x7f1f7c1101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 --2- 172.21.15.119:0/917088934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c074390 0x7f1f7c10d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 --2- 172.21.15.119:0/917088934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c073a70 0x7f1f7c073e50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 -- 172.21.15.119:0/917088934 >> 172.21.15.119:0/917088934 conn(0x7f1f7c0fd040 msgr2=0x7f1f7c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 -- 172.21.15.119:0/917088934 shutdown_connections 2024-03-28T18:44:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.868+0000 7f1f81800640 1 -- 172.21.15.119:0/917088934 wait complete. 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.869+0000 7f1f81800640 1 Processor -- start 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.869+0000 7f1f81800640 1 -- start start 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c074390 0x7f1f7c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c10ddb0 0x7f1f7c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c113b80 con 0x7f1f7c10ddb0 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c113a00 con 0x7f1f7c073a70 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f81800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c113d00 con 0x7f1f7c074390 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44544/0 (socket says 172.21.15.119:44544) 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 -- 172.21.15.119:0/407492749 learned_addr learned my addr 172.21.15.119:0/407492749 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7b400640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c074390 0x7f1f7c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f80e00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c10ddb0 0x7f1f7c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 -- 172.21.15.119:0/407492749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c074390 msgr2=0x7f1f7c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c074390 0x7f1f7c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 -- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c10ddb0 msgr2=0x7f1f7c19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c10ddb0 0x7f1f7c19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.870+0000 7f1f7be00640 1 -- 172.21.15.119:0/407492749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c19f010 con 0x7f1f7c073a70 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f7be00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f78c00640 1 -- 172.21.15.119:0/407492749 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c042020 con 0x7f1f7c073a70 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f78c00640 1 -- 172.21.15.119:0/407492749 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c0054e0 con 0x7f1f7c073a70 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c19f2a0 con 0x7f1f7c073a70 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f78c00640 1 -- 172.21.15.119:0/407492749 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c003b20 con 0x7f1f7c073a70 2024-03-28T18:44:57.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.871+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c1a6b90 con 0x7f1f7c073a70 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.874+0000 7f1f85ba0740 1 Processor -- start 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.874+0000 7f1f85ba0740 1 -- start start 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.874+0000 7f1f78c00640 1 -- 172.21.15.119:0/407492749 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f1f6c045020 con 0x7f1f7c073a70 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e9460 0x55acea4eb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea444250 con 0x55acea4e0dd0 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea3eca30 con 0x55acea4e00f0 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea4402f0 con 0x55acea4e9460 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f80e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44552/0 (socket says 172.21.15.119:44552) 2024-03-28T18:44:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 -- 172.21.15.119:0/2672842747 learned_addr learned my addr 172.21.15.119:0/2672842747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:57.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7b400640 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e9460 0x55acea4eb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 -- 172.21.15.119:0/2672842747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e9460 msgr2=0x55acea4eb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e9460 0x55acea4eb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 -- 172.21.15.119:0/2672842747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 msgr2=0x55acea4dfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4dfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.875+0000 7f1f7be00640 1 -- 172.21.15.119:0/2672842747 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acea3d7690 con 0x55acea4e00f0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f7be00640 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0ee200 tx=0x7f1f6c0f4ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5f3834014916e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f71600640 1 -- 172.21.15.119:0/2672842747 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c100020 con 0x55acea4e00f0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f71600640 1 -- 172.21.15.119:0/2672842747 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c0f9d10 con 0x55acea4e00f0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f71600640 1 -- 172.21.15.119:0/2672842747 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0f5040 con 0x55acea4e00f0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f78c00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 0x7f1f600a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f78c00640 1 -- 172.21.15.119:0/407492749 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f6c0f0b20 con 0x7f1f7c073a70 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.876+0000 7f1f80e00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 0x7f1f600a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 -- 172.21.15.119:0/2672842747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 msgr2=0x55acea4e0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0ee200 tx=0x7f1f6c0f4ba0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 -- 172.21.15.119:0/2672842747 shutdown_connections 2024-03-28T18:44:57.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e9460 0x55acea4eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e00f0 0x55acea4e0570 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/2672842747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4dfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 -- 172.21.15.119:0/2672842747 >> 172.21.15.119:0/2672842747 conn(0x55acea4a7200 msgr2=0x55acea4dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 -- 172.21.15.119:0/2672842747 shutdown_connections 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f80e00640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 0x7f1f600a4ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0693d0 tx=0x7f1f74009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.877+0000 7f1f85ba0740 1 -- 172.21.15.119:0/2672842747 wait complete. 2024-03-28T18:44:57.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.878+0000 7f1f85ba0740 1 -- start start 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.878+0000 7f1f85ba0740 1 Processor -- start 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4e64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea4ec090 con 0x55acea4e0dd0 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea3ffe80 con 0x55acea4e9460 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acea4ec210 con 0x55acea4e00f0 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f85ba0740 10 client.0 _ll_register_callbacks cb 0x55acea412130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47660/0 (socket says 172.21.15.119:47660) 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 -- 172.21.15.119:0/1625165633 learned_addr learned my addr 172.21.15.119:0/1625165633 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f80e00640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4e64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 -- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 msgr2=0x55acea4e64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4e64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 -- 172.21.15.119:0/1625165633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 msgr2=0x55acea4e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f7be00640 1 -- 172.21.15.119:0/1625165633 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acea2227d0 con 0x55acea4e00f0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.879+0000 7f1f80e00640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4e64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f7be00640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0b9520 tx=0x7f1f6c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e402527098df2209 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c100070 con 0x55acea4e00f0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f6c0bb3c0 con 0x55acea4e00f0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f6c0f9d10 con 0x55acea4e00f0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f6c0b9360 con 0x55acea4e00f0 2024-03-28T18:44:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f85ba0740 10 client.25678 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55acea1583f0 con 0x55acea4e00f0 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f6c0f9d10 con 0x55acea4e00f0 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.880+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f6c0fb040 con 0x55acea4e00f0 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 10 client.25678 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 10 client.25678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 20 client.25678 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea4f8f70 0x55acea4fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 4 client.25678 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acea451790 con 0x55acea4f8f70 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.881+0000 7f1f7b400640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea4f8f70 0x55acea4fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.882+0000 7f1f66000640 10 client.25678 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.882+0000 7f1f7b400640 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea4f8f70 0x55acea4fb360 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365f2fd86d005927 server_cookie=602c71190aba271f in_seq=0 out_seq=0 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.882+0000 7f1f66000640 1 -- 172.21.15.119:0/1625165633 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55acea451790 con 0x55acea4f8f70 2024-03-28T18:44:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.882+0000 7f1f66000640 10 client.25678 handle_command_reply: tid=0 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.882+0000 7f1f85ba0740 1 client.25678 shutdown 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 msgr2=0x55acea4e8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c0b9520 tx=0x7f1f6c005fd0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 shutdown_connections 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea4f8f70 0x55acea4fb360 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4e64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e0dd0 0x55acea4e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1625165633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e00f0 0x55acea4e8be0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 >> 172.21.15.119:0/1625165633 conn(0x55acea4a7200 msgr2=0x55acea4dc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f66000640 0 client.25678 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 shutdown_connections 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1625165633 wait complete. 2024-03-28T18:44:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.883+0000 7f1f85ba0740 20 client.25678 trim_cache size 0 max 16384 2024-03-28T18:44:57.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.902+0000 7f1f85ba0740 1 Processor -- start 2024-03-28T18:44:57.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 -- start start 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1af3c0 con 0x55acea4e00f0 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1b0d40 con 0x55acea4e9460 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1afa20 con 0x55acea4e0dd0 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f80e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47674/0 (socket says 172.21.15.119:47674) 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.903+0000 7f1f7b400640 1 -- 172.21.15.119:0/1372458335 learned_addr learned my addr 172.21.15.119:0/1372458335 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 -- 172.21.15.119:0/1372458335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 msgr2=0x55acea4ea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4ea850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 -- 172.21.15.119:0/1372458335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 msgr2=0x55acea4e86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4e86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 -- 172.21.15.119:0/1372458335 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acea1180f0 con 0x55acea4e0dd0 2024-03-28T18:44:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f80e00640 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7be00640 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f7b400640 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55acea4fb990 tx=0x7f1f68005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1d54c176dd5b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f70c00640 1 -- 172.21.15.119:0/1372458335 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68004030 con 0x55acea4e0dd0 2024-03-28T18:44:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f70c00640 1 -- 172.21.15.119:0/1372458335 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f68020c50 con 0x55acea4e0dd0 2024-03-28T18:44:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.904+0000 7f1f70c00640 1 -- 172.21.15.119:0/1372458335 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68003990 con 0x55acea4e0dd0 2024-03-28T18:44:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1372458335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 msgr2=0x55acea4e8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55acea4fb990 tx=0x7f1f68005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1372458335 shutdown_connections 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4e8be0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/1372458335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4e86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1372458335 >> 172.21.15.119:0/1372458335 conn(0x55acea4a7200 msgr2=0x55acea4da670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1372458335 shutdown_connections 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.905+0000 7f1f85ba0740 1 -- 172.21.15.119:0/1372458335 wait complete. 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 -- start start 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 Processor -- start 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4f0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1b06e0 con 0x55acea4e00f0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1b0080 con 0x55acea4e9460 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c1ad580 con 0x55acea4e0dd0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.906+0000 7f1f85ba0740 10 client.0 _ll_register_callbacks cb 0x7f1f74009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47684/0 (socket says 172.21.15.119:47684) 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 -- 172.21.15.119:0/274226346 learned_addr learned my addr 172.21.15.119:0/274226346 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f80e00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7be00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 -- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 msgr2=0x55acea4f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 -- 172.21.15.119:0/274226346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 msgr2=0x55acea4f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 -- 172.21.15.119:0/274226346 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55acea4f08c0 con 0x55acea4e0dd0 2024-03-28T18:44:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f80e00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7be00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f7b400640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000c00 tx=0x7f1f68021e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b969b7b53193c658 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.907+0000 7f1f66000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6802e070 con 0x55acea4e0dd0 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f68020770 con 0x55acea4e0dd0 2024-03-28T18:44:57.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f68020c50 con 0x55acea4e0dd0 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f6803a050 con 0x55acea4e0dd0 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 10 client.25684 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55acea554a50 con 0x55acea4e0dd0 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f68020c50 con 0x55acea4e0dd0 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f6801e480 con 0x55acea4e0dd0 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 10 client.25684 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 10 client.25684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.908+0000 7f1f85ba0740 20 client.25684 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea5551b0 0x55acea5575d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f85ba0740 4 client.25684 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:44:57.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55acea44bae0 con 0x55acea5551b0 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f7be00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea5551b0 0x55acea5575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f7be00640 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea5551b0 0x55acea5575d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ea3fee046d5f612 server_cookie=d4fe3086a4524558 in_seq=0 out_seq=0 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f66000640 10 client.25684 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f66000640 1 -- 172.21.15.119:0/274226346 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55acea44bae0 con 0x55acea5551b0 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f66000640 10 client.25684 handle_command_reply: tid=0 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f85ba0740 1 client.25684 shutdown 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.909+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 msgr2=0x55acea4f40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f68000c00 tx=0x7f1f68021e10 comp rx=0 tx=0).stop 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 shutdown_connections 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55acea5551b0 0x55acea5575d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55acea4e9460 0x55acea4f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55acea4e0dd0 0x55acea4f40c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 --2- 172.21.15.119:0/274226346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55acea4e00f0 0x55acea4f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f66000640 0 client.25684 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 >> 172.21.15.119:0/274226346 conn(0x55acea4a7200 msgr2=0x55acea4e4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 shutdown_connections 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 1 -- 172.21.15.119:0/274226346 wait complete. 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f85ba0740 20 client.25684 trim_cache size 0 max 16384 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:44:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 msgr2=0x7f1f600a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 0x7f1f600a4ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0693d0 tx=0x7f1f74009500 comp rx=0 tx=0).stop 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 msgr2=0x7f1f7c19a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.910+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c000c00 tx=0x7f1f6c004360 comp rx=0 tx=0).stop 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 shutdown_connections 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f600a2a00 0x7f1f600a4ec0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f7c10ddb0 0x7f1f7c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f7c074390 0x7f1f7c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 --2- 172.21.15.119:0/407492749 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f7c073a70 0x7f1f7c19a020 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:57.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 >> 172.21.15.119:0/407492749 conn(0x7f1f7c0fd040 msgr2=0x7f1f7c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:57.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 shutdown_connections 2024-03-28T18:44:57.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:57.911+0000 7f1f81800640 1 -- 172.21.15.119:0/407492749 wait complete. 2024-03-28T18:44:57.928 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:44:57.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:44:58.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 -- 172.21.15.119:0/3000360685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f4107590 msgr2=0x7f55f4107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 --2- 172.21.15.119:0/3000360685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f4107590 0x7f55f4107970 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009900 tx=0x7f55e402bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 -- 172.21.15.119:0/3000360685 shutdown_connections 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 --2- 172.21.15.119:0/3000360685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f40691c0 0x7f55f410e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 --2- 172.21.15.119:0/3000360685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4068800 0x7f55f4068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 --2- 172.21.15.119:0/3000360685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f4107590 0x7f55f4107970 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 -- 172.21.15.119:0/3000360685 >> 172.21.15.119:0/3000360685 conn(0x7f55f4075fb0 msgr2=0x7f55f40763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 -- 172.21.15.119:0/3000360685 shutdown_connections 2024-03-28T18:44:58.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.117+0000 7f55f9c00640 1 -- 172.21.15.119:0/3000360685 wait complete. 2024-03-28T18:44:58.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.118+0000 7f55f9c00640 1 Processor -- start 2024-03-28T18:44:58.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.118+0000 7f55f9c00640 1 -- start start 2024-03-28T18:44:58.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 0x7f55f4100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4107590 0x7f55f41011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f41118d0 con 0x7f55f4068800 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4111750 con 0x7f55f4107590 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f4111a50 con 0x7f55f40691c0 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 0x7f55f4100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f9200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4107590 0x7f55f41011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47702/0 (socket says 172.21.15.119:47702) 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 -- 172.21.15.119:0/2197888821 learned_addr learned my addr 172.21.15.119:0/2197888821 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 0x7f55f4100720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57248/0 (socket says 172.21.15.119:57248) 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 -- 172.21.15.119:0/2197888821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4107590 msgr2=0x7f55f41011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4107590 0x7f55f41011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 -- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 msgr2=0x7f55f4100720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.119+0000 7f55f3400640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 0x7f55f4100720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f3400640 1 -- 172.21.15.119:0/2197888821 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f41a6270 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f3400640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55e000e9f0 tx=0x7f55e000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e000cd80 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55e0005500 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f41a6560 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0010690 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.120+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f41a6a60 con 0x7f55f40691c0 2024-03-28T18:44:58.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.121+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f40683b0 con 0x7f55f40691c0 2024-03-28T18:44:58.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.124+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f55e00046c0 con 0x7f55f40691c0 2024-03-28T18:44:58.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.126+0000 7f55f0c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 0x7f55d80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.126+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f55e00cf4c0 con 0x7f55f40691c0 2024-03-28T18:44:58.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.126+0000 7f55f3e00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 0x7f55d80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.127+0000 7f55f3e00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 0x7f55d80a4ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55e402efd0 tx=0x7f55e4000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.129+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55e009bff0 con 0x7f55f40691c0 2024-03-28T18:44:58.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.394+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f4101c50 con 0x7f55f40691c0 2024-03-28T18:44:58.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.395+0000 7f55f0c00640 1 -- 172.21.15.119:0/2197888821 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f55e009b8f0 con 0x7f55f40691c0 2024-03-28T18:44:58.395 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:44:58.395 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:44:58.396 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 msgr2=0x7f55d80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 0x7f55d80a4ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55e402efd0 tx=0x7f55e4000bc0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 msgr2=0x7f55f4100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55e000e9f0 tx=0x7f55e000edf0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 shutdown_connections 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f55d80a2a00 0x7f55d80a4ec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f55f4107590 0x7f55f41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f55f40691c0 0x7f55f4100c60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 --2- 172.21.15.119:0/2197888821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4068800 0x7f55f4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 >> 172.21.15.119:0/2197888821 conn(0x7f55f4075fb0 msgr2=0x7f55f40ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 shutdown_connections 2024-03-28T18:44:58.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.398+0000 7f55f9c00640 1 -- 172.21.15.119:0/2197888821 wait complete. 2024-03-28T18:44:58.415 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:58.415 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:44:58.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:58 smithi119 ceph-mon[31813]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 -- 172.21.15.119:0/88251075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3258108860 msgr2=0x7f325810c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 --2- 172.21.15.119:0/88251075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3258108860 0x7f325810c8a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3244009900 tx=0x7f324402bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 -- 172.21.15.119:0/88251075 shutdown_connections 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 --2- 172.21.15.119:0/88251075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f325810cf80 0x7f3258114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 --2- 172.21.15.119:0/88251075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3258108860 0x7f325810c8a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 --2- 172.21.15.119:0/88251075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f3258108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 -- 172.21.15.119:0/88251075 >> 172.21.15.119:0/88251075 conn(0x7f3258075be0 msgr2=0x7f3258078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 -- 172.21.15.119:0/88251075 shutdown_connections 2024-03-28T18:44:58.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.611+0000 7f325d000640 1 -- 172.21.15.119:0/88251075 wait complete. 2024-03-28T18:44:58.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 Processor -- start 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 -- start start 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f325819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 0x7f32581a3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258118080 con 0x7f3258107eb0 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258117f00 con 0x7f32581a2ee0 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.612+0000 7f325d000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258118200 con 0x7f325810cf80 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47716/0 (socket says 172.21.15.119:47716) 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255400640 1 -- 172.21.15.119:0/962550272 learned_addr learned my addr 172.21.15.119:0/962550272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255e00640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f325819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 0x7f32581a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 -- 172.21.15.119:0/962550272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 msgr2=0x7f325819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 -- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 msgr2=0x7f325819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f325819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 -- 172.21.15.119:0/962550272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32581a39d0 con 0x7f32581a2ee0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255400640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3255e00640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f325819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.613+0000 7f3256800640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 0x7f32581a3390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f325000ec50 tx=0x7f325000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.614+0000 7f324ea00640 1 -- 172.21.15.119:0/962550272 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250003b70 con 0x7f32581a2ee0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.614+0000 7f324ea00640 1 -- 172.21.15.119:0/962550272 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3250003d10 con 0x7f32581a2ee0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.614+0000 7f324ea00640 1 -- 172.21.15.119:0/962550272 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250010890 con 0x7f32581a2ee0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.614+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32581a3cc0 con 0x7f32581a2ee0 2024-03-28T18:44:58.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.614+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32581ab460 con 0x7f32581a2ee0 2024-03-28T18:44:58.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.617+0000 7f325fc5f740 1 Processor -- start 2024-03-28T18:44:58.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f324ea00640 1 -- 172.21.15.119:0/962550272 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f3250003fc0 con 0x7f32581a2ee0 2024-03-28T18:44:58.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 -- start start 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f60dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f619050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f5712c0 con 0x55800f60d6d0 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f519c90 con 0x55800f616c00 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.618+0000 7f325fc5f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f56d360 con 0x55800f60e080 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47732/0 (socket says 172.21.15.119:47732) 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 -- 172.21.15.119:0/386009386 learned_addr learned my addr 172.21.15.119:0/386009386 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255400640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f60dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255400640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f619050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 -- 172.21.15.119:0/386009386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 msgr2=0x55800f619050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f619050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 -- 172.21.15.119:0/386009386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 msgr2=0x55800f60dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f60dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255e00640 1 -- 172.21.15.119:0/386009386 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55800f53e080 con 0x55800f60e080 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f3255400640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f619050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.619+0000 7f324ea00640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 0x7f323c0a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f3255e00640 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f32480115f0 tx=0x7f3248011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3156e069b82dc72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f324ea00640 1 -- 172.21.15.119:0/962550272 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f32500cf570 con 0x7f32581a2ee0 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f3255400640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 0x7f323c0a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f3243400640 1 -- 172.21.15.119:0/386009386 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324801e070 con 0x55800f60e080 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f3243400640 1 -- 172.21.15.119:0/386009386 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f324800fb20 con 0x55800f60e080 2024-03-28T18:44:58.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.620+0000 7f3243400640 1 -- 172.21.15.119:0/386009386 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248019c40 con 0x55800f60e080 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 -- 172.21.15.119:0/386009386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 msgr2=0x55800f616630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f32480115f0 tx=0x7f3248011ab0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f3255400640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 0x7f323c0a4f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32581070a0 tx=0x7f3244005b50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 -- 172.21.15.119:0/386009386 shutdown_connections 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f619050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f616630 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 --2- 172.21.15.119:0/386009386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f60dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 -- 172.21.15.119:0/386009386 >> 172.21.15.119:0/386009386 conn(0x55800f607da0 msgr2=0x55800f608180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 -- 172.21.15.119:0/386009386 shutdown_connections 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.621+0000 7f325fc5f740 1 -- 172.21.15.119:0/386009386 wait complete. 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 -- start start 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 Processor -- start 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f6138c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 0x55800f613ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f619b70 con 0x55800f616c00 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f52d0d0 con 0x55800f60d6d0 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.622+0000 7f325fc5f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800f619cf0 con 0x55800f60e080 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f325fc5f740 10 client.0 _ll_register_callbacks cb 0x55800f531e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44620/0 (socket says 172.21.15.119:44620) 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 -- 172.21.15.119:0/2187959004 learned_addr learned my addr 172.21.15.119:0/2187959004 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f6138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3256800640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 0x55800f613ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 -- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 msgr2=0x55800f6138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f6138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 -- 172.21.15.119:0/2187959004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 msgr2=0x55800f613ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 0x55800f613ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 -- 172.21.15.119:0/2187959004 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55800f2853f0 con 0x55800f60d6d0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3256800640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 0x55800f613ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3255e00640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f32480115c0 tx=0x7f324800f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87abce8222588e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.623+0000 7f3237e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248027060 con 0x55800f60d6d0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f324800fb20 con 0x55800f60d6d0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f32480199c0 con 0x55800f60d6d0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f324801e040 con 0x55800f60d6d0 2024-03-28T18:44:58.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f325fc5f740 10 client.25676 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:58.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55800f34f7d0 con 0x55800f60d6d0 2024-03-28T18:44:58.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.624+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f324800fb20 con 0x55800f60d6d0 2024-03-28T18:44:58.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3248021030 con 0x55800f60d6d0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 10 client.25676 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 10 client.25676 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 20 client.25676 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6268b0 0x55800f628ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 4 client.25676 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55800f57e800 con 0x55800f6268b0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.625+0000 7f3255400640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6268b0 0x55800f628ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.626+0000 7f3255400640 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6268b0 0x55800f628ca0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c2b2a873b3f06cf server_cookie=35578af7b76bcdb4 in_seq=0 out_seq=0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.626+0000 7f3237e00640 10 client.25676 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f3237e00640 1 -- 172.21.15.119:0/2187959004 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55800f57e800 con 0x55800f6268b0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f3237e00640 10 client.25676 handle_command_reply: tid=0 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 client.25676 shutdown 2024-03-28T18:44:58.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 msgr2=0x55800f616530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f32480115c0 tx=0x7f324800f870 comp rx=0 tx=0).stop 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 shutdown_connections 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6268b0 0x55800f628ca0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f616c00 0x55800f613ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f6138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 --2- 172.21.15.119:0/2187959004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f616530 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 >> 172.21.15.119:0/2187959004 conn(0x55800f607da0 msgr2=0x55800f609af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f3237e00640 0 client.25676 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 shutdown_connections 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 1 -- 172.21.15.119:0/2187959004 wait complete. 2024-03-28T18:44:58.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.627+0000 7f325fc5f740 20 client.25676 trim_cache size 0 max 16384 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.641+0000 7f325fc5f740 1 Processor -- start 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.641+0000 7f325fc5f740 1 -- start start 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f6288c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60e080 0x55800f615ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b3a30 con 0x55800f60e080 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b53b0 con 0x55800f60d6d0 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f325fc5f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b4090 con 0x55800f616c00 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3255400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60e080 0x55800f615ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3255e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f6288c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47752/0 (socket says 172.21.15.119:47752) 2024-03-28T18:44:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 -- 172.21.15.119:0/1446158405 learned_addr learned my addr 172.21.15.119:0/1446158405 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 -- 172.21.15.119:0/1446158405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 msgr2=0x55800f6288c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f6288c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 -- 172.21.15.119:0/1446158405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60e080 msgr2=0x55800f615ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60e080 0x55800f615ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3256800640 1 -- 172.21.15.119:0/1446158405 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55800f60f490 con 0x55800f616c00 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.642+0000 7f3255e00640 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f6288c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.643+0000 7f3256800640 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f325000cba0 tx=0x7f325009b830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f5c2daf3c0b861 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.643+0000 7f3242a00640 1 -- 172.21.15.119:0/1446158405 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325009c0a0 con 0x55800f616c00 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.643+0000 7f3242a00640 1 -- 172.21.15.119:0/1446158405 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f325009b9a0 con 0x55800f616c00 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.643+0000 7f3242a00640 1 -- 172.21.15.119:0/1446158405 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500ccdf0 con 0x55800f616c00 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- 172.21.15.119:0/1446158405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 msgr2=0x55800f616530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f325000cba0 tx=0x7f325009b830 comp rx=0 tx=0).stop 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- 172.21.15.119:0/1446158405 shutdown_connections 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f616c00 0x55800f616530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60e080 0x55800f615ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1446158405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f60d6d0 0x55800f6288c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- 172.21.15.119:0/1446158405 >> 172.21.15.119:0/1446158405 conn(0x55800f607da0 msgr2=0x55800f606eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- 172.21.15.119:0/1446158405 shutdown_connections 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- 172.21.15.119:0/1446158405 wait complete. 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 -- start start 2024-03-28T18:44:58.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.644+0000 7f325fc5f740 1 Processor -- start 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f61eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f61d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b4d50 con 0x55800f60d6d0 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b46f0 con 0x55800f616c00 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32581b1bf0 con 0x55800f60e080 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f325fc5f740 10 client.0 _ll_register_callbacks cb 0x7f324801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f61eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47768/0 (socket says 172.21.15.119:47768) 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255400640 1 -- 172.21.15.119:0/1759360519 learned_addr learned my addr 172.21.15.119:0/1759360519 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3256800640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f61d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255400640 1 -- 172.21.15.119:0/1759360519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 msgr2=0x55800f61d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.645+0000 7f3255400640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f61d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3255400640 1 -- 172.21.15.119:0/1759360519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 msgr2=0x55800f61eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3255400640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f61eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3255400640 1 -- 172.21.15.119:0/1759360519 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55800f61da90 con 0x55800f60e080 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3255e00640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f61eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3255400640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f324402c240 tx=0x7f324402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b78cb1338d5b5cda server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3237e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324402c900 con 0x55800f60e080 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f324402caa0 con 0x55800f60e080 2024-03-28T18:44:58.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f325009b9a0 con 0x55800f60e080 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f324402cca0 con 0x55800f60e080 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f325fc5f740 10 client.25708 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.646+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55800f682e50 con 0x55800f60e080 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f325009b9a0 con 0x55800f60e080 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f32440039d0 con 0x55800f60e080 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 10 client.25708 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 10 client.25708 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 20 client.25708 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6835b0 0x55800f6859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 4 client.25708 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55800f578b50 con 0x55800f6835b0 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.647+0000 7f3255e00640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6835b0 0x55800f6859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f3237e00640 10 client.25708 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f3255e00640 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6835b0 0x55800f6859d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b6c185e1db8793 server_cookie=68e14c799b43fb72 in_seq=0 out_seq=0 2024-03-28T18:44:58.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f3237e00640 1 -- 172.21.15.119:0/1759360519 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55800f578b50 con 0x55800f6835b0 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f3237e00640 10 client.25708 handle_command_reply: tid=0 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 client.25708 shutdown 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 msgr2=0x55800f61cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f324402c240 tx=0x7f324402bc30 comp rx=0 tx=0).stop 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 shutdown_connections 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55800f6835b0 0x55800f6859d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55800f616c00 0x55800f61d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55800f60e080 0x55800f61cee0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 --2- 172.21.15.119:0/1759360519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55800f60d6d0 0x55800f61eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f3237e00640 0 client.25708 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 >> 172.21.15.119:0/1759360519 conn(0x55800f607da0 msgr2=0x55800f60e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 shutdown_connections 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 1 -- 172.21.15.119:0/1759360519 wait complete. 2024-03-28T18:44:58.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.648+0000 7f325fc5f740 20 client.25708 trim_cache size 0 max 16384 2024-03-28T18:44:58.653 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:58.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 msgr2=0x7f323c0a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 0x7f323c0a4f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32581070a0 tx=0x7f3244005b50 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 msgr2=0x7f32581a3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 0x7f32581a3390 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f325000ec50 tx=0x7f325000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 shutdown_connections 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f323c0a2a80 0x7f323c0a4f40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32581a2ee0 0x7f32581a3390 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f325810cf80 0x7f325819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 --2- 172.21.15.119:0/962550272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3258107eb0 0x7f325819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 >> 172.21.15.119:0/962550272 conn(0x7f3258075be0 msgr2=0x7f3258077ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 shutdown_connections 2024-03-28T18:44:58.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.649+0000 7f325d000640 1 -- 172.21.15.119:0/962550272 wait complete. 2024-03-28T18:44:58.673 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:58.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:44:58.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:58 smithi196 ceph-mon[36300]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:58.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:58 smithi131 ceph-mon[36065]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 -- 172.21.15.119:0/2237626176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1680102ff0 msgr2=0x7f168010b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 --2- 172.21.15.119:0/2237626176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1680102ff0 0x7f168010b4d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f166c009900 tx=0x7f166c02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 -- 172.21.15.119:0/2237626176 shutdown_connections 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 --2- 172.21.15.119:0/2237626176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 0x7f168010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 --2- 172.21.15.119:0/2237626176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1680102ff0 0x7f168010b4d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 --2- 172.21.15.119:0/2237626176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16801026d0 0x7f1680102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 -- 172.21.15.119:0/2237626176 >> 172.21.15.119:0/2237626176 conn(0x7f16800fbef0 msgr2=0x7f16800fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 -- 172.21.15.119:0/2237626176 shutdown_connections 2024-03-28T18:44:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.906+0000 7f1686800640 1 -- 172.21.15.119:0/2237626176 wait complete. 2024-03-28T18:44:58.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 Processor -- start 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 -- start start 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16801026d0 0x7f1680197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 0x7f168019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680111900 con 0x7f168010baf0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680111780 con 0x7f16801026d0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1686800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680111a80 con 0x7f1680102ff0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1684a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1685400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16801026d0 0x7f1680197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1685e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 0x7f168019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.907+0000 7f1684a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52136/0 (socket says 172.21.15.119:52136) 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 -- 172.21.15.119:0/1181920539 learned_addr learned my addr 172.21.15.119:0/1181920539 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 -- 172.21.15.119:0/1181920539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16801026d0 msgr2=0x7f1680197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16801026d0 0x7f1680197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 -- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 msgr2=0x7f168019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 0x7f168019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 -- 172.21.15.119:0/1181920539 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168019cdd0 con 0x7f1680102ff0 2024-03-28T18:44:58.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f1684a00640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f166c000c00 tx=0x7f166c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.908+0000 7f167e000640 1 -- 172.21.15.119:0/1181920539 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c042020 con 0x7f1680102ff0 2024-03-28T18:44:58.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.909+0000 7f167e000640 1 -- 172.21.15.119:0/1181920539 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f166c0054e0 con 0x7f1680102ff0 2024-03-28T18:44:58.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.909+0000 7f167e000640 1 -- 172.21.15.119:0/1181920539 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c003a80 con 0x7f1680102ff0 2024-03-28T18:44:58.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.909+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168019d060 con 0x7f1680102ff0 2024-03-28T18:44:58.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.909+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16801a4950 con 0x7f1680102ff0 2024-03-28T18:44:58.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.912+0000 7f167e000640 1 -- 172.21.15.119:0/1181920539 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f166c045020 con 0x7f1680102ff0 2024-03-28T18:44:58.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.912+0000 7f168aa79740 1 Processor -- start 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.912+0000 7f168aa79740 1 -- start start 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c64c2c0 con 0x55ce0c6f1de0 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c5f4c90 con 0x55ce0c6e88b0 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f168aa79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c648360 con 0x55ce0c6e9260 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56864/0 (socket says 172.21.15.119:56864) 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 -- 172.21.15.119:0/2980059738 learned_addr learned my addr 172.21.15.119:0/2980059738 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685400640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1684a00640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 -- 172.21.15.119:0/2980059738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 msgr2=0x55ce0c6f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 -- 172.21.15.119:0/2980059738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 msgr2=0x55ce0c6f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685e00640 1 -- 172.21.15.119:0/2980059738 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce0c619080 con 0x55ce0c6e88b0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1685400640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f1684a00640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.913+0000 7f167e000640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 0x7f16680a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f167e000640 1 -- 172.21.15.119:0/1181920539 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f166c0f0a70 con 0x7f1680102ff0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f1685e00640 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1678009f50 tx=0x7f16780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfdbc7638c92f2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f1685400640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 0x7f16680a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f1672600640 1 -- 172.21.15.119:0/2980059738 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678004b20 con 0x55ce0c6e88b0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f1672600640 1 -- 172.21.15.119:0/2980059738 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1678004cc0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.914+0000 7f1672600640 1 -- 172.21.15.119:0/2980059738 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678005020 con 0x55ce0c6e88b0 2024-03-28T18:44:58.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 -- 172.21.15.119:0/2980059738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 msgr2=0x55ce0c6e8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f1685400640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 0x7f16680a4f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x55ce0c6e3ff0 tx=0x7f1674015040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1678009f50 tx=0x7f16780118d0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 -- 172.21.15.119:0/2980059738 shutdown_connections 2024-03-28T18:44:58.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 --2- 172.21.15.119:0/2980059738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6e8c90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 -- 172.21.15.119:0/2980059738 >> 172.21.15.119:0/2980059738 conn(0x55ce0c6e2f80 msgr2=0x55ce0c6e3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 -- 172.21.15.119:0/2980059738 shutdown_connections 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.915+0000 7f168aa79740 1 -- 172.21.15.119:0/2980059738 wait complete. 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 -- start start 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 Processor -- start 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ef250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c6f4d50 con 0x55ce0c6f1de0 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c6080d0 con 0x55ce0c6e9260 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce0c6f4ed0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f168aa79740 10 client.0 _ll_register_callbacks cb 0x55ce0c60ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:58.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f1685400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f1685400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52160/0 (socket says 172.21.15.119:52160) 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f1684a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.916+0000 7f1685400640 1 -- 172.21.15.119:0/4253575978 learned_addr learned my addr 172.21.15.119:0/4253575978 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685e00640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 -- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 msgr2=0x55ce0c6eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 -- 172.21.15.119:0/4253575978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 msgr2=0x55ce0c6ef250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ef250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 -- 172.21.15.119:0/4253575978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce0c3603f0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1684a00640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685e00640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1685400640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f167400e8f0 tx=0x7f1674004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343402bfb471c459 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1666a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:44:58.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.917+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167402e060 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1674003af0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1678004cc0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1674025040 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 10 client.25720 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ce0c42a7d0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1678004cc0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f167401f3f0 con 0x55ce0c6e88b0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 10 client.25720 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 10 client.25720 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 20 client.25720 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c701d30 0x55ce0c704120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 4 client.25720 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.918+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce0c659800 con 0x55ce0c701d30 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.919+0000 7f1684a00640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c701d30 0x55ce0c704120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.919+0000 7f1684a00640 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c701d30 0x55ce0c704120 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34ad1d685017b30a server_cookie=331d8869635822ba in_seq=0 out_seq=0 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.919+0000 7f1666a00640 10 client.25720 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f1666a00640 1 -- 172.21.15.119:0/4253575978 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce0c659800 con 0x55ce0c701d30 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f1666a00640 10 client.25720 handle_command_reply: tid=0 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 client.25720 shutdown 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 msgr2=0x55ce0c6f1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f167400e8f0 tx=0x7f1674004730 comp rx=0 tx=0).stop 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 shutdown_connections 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c701d30 0x55ce0c704120 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f1666a00640 0 client.25720 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 --2- 172.21.15.119:0/4253575978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1780 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 >> 172.21.15.119:0/4253575978 conn(0x55ce0c6e2f80 msgr2=0x55ce0c6e4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 shutdown_connections 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 1 -- 172.21.15.119:0/4253575978 wait complete. 2024-03-28T18:44:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.920+0000 7f168aa79740 20 client.25720 trim_cache size 0 max 16384 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.935+0000 7f168aa79740 1 Processor -- start 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 -- start start 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801aaf20 con 0x55ce0c6e9260 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801ac8a0 con 0x55ce0c6f1de0 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f168aa79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801ab580 con 0x55ce0c6e88b0 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f1685400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f1684a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f1685e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f1685400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52174/0 (socket says 172.21.15.119:52174) 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.936+0000 7f1685400640 1 -- 172.21.15.119:0/1329091491 learned_addr learned my addr 172.21.15.119:0/1329091491 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 -- 172.21.15.119:0/1329091491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 msgr2=0x55ce0c6f1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 -- 172.21.15.119:0/1329091491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e9260 msgr2=0x55ce0c6f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 -- 172.21.15.119:0/1329091491 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce0c6f4c30 con 0x55ce0c6f1de0 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1685e00640 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ebdb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1678009f20 tx=0x7f1678009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1378fbd709d826c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1671c00640 1 -- 172.21.15.119:0/1329091491 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678003af0 con 0x55ce0c6f1de0 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1671c00640 1 -- 172.21.15.119:0/1329091491 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1678027070 con 0x55ce0c6f1de0 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.937+0000 7f1671c00640 1 -- 172.21.15.119:0/1329091491 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16780226b0 con 0x55ce0c6f1de0 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- 172.21.15.119:0/1329091491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 msgr2=0x55ce0c6ebdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ebdb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1678009f20 tx=0x7f1678009cc0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- 172.21.15.119:0/1329091491 shutdown_connections 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6ebdb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 --2- 172.21.15.119:0/1329091491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- 172.21.15.119:0/1329091491 >> 172.21.15.119:0/1329091491 conn(0x55ce0c6e2f80 msgr2=0x55ce0c6e2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- 172.21.15.119:0/1329091491 shutdown_connections 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- 172.21.15.119:0/1329091491 wait complete. 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.938+0000 7f168aa79740 1 -- start start 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 Processor -- start 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801ac240 con 0x55ce0c6e88b0 2024-03-28T18:44:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801abbe0 con 0x55ce0c6e9260 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16801a90e0 con 0x55ce0c6f1de0 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f168aa79740 10 client.0 _ll_register_callbacks cb 0x7f1674022040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f1684a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f1684a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56894/0 (socket says 172.21.15.119:56894) 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f1684a00640 1 -- 172.21.15.119:0/2183810791 learned_addr learned my addr 172.21.15.119:0/2183810791 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.939+0000 7f1685e00640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1685400640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 -- 172.21.15.119:0/2183810791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6f1de0 msgr2=0x55ce0c6f7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 -- 172.21.15.119:0/2183810791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 msgr2=0x55ce0c6f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 -- 172.21.15.119:0/2183810791 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce0c6f8550 con 0x55ce0c6e9260 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1685400640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1684a00640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b9930 tx=0x7f166c02cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d04ac6ec79911b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1666a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c0f0e60 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f166c0bb2b0 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1678027070 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f166c0d9050 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f168aa79740 10 client.25700 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.940+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ce0c75e0e0 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1678027070 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f166c0c52c0 con 0x55ce0c6e9260 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 10 client.25700 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 10 client.25700 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 20 client.25700 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c75e860 0x55ce0c760c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 4 client.25700 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ce0c653b50 con 0x55ce0c75e860 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.941+0000 7f1685400640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c75e860 0x55ce0c760c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f1666a00640 10 client.25700 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f1685400640 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c75e860 0x55ce0c760c80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4aeead0a4b8b7659 server_cookie=6712fd5c8325c79 in_seq=0 out_seq=0 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f1666a00640 1 -- 172.21.15.119:0/2183810791 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ce0c653b50 con 0x55ce0c75e860 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f1666a00640 10 client.25700 handle_command_reply: tid=0 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 client.25700 shutdown 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 msgr2=0x55ce0c6f9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b9930 tx=0x7f166c02cb30 comp rx=0 tx=0).stop 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 shutdown_connections 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce0c6f1de0 0x55ce0c6f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce0c6e9260 0x55ce0c6f9e60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ce0c75e860 0x55ce0c760c80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 --2- 172.21.15.119:0/2183810791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce0c6e88b0 0x55ce0c6f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 >> 172.21.15.119:0/2183810791 conn(0x55ce0c6e2f80 msgr2=0x55ce0c6e4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f1666a00640 0 client.25700 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 shutdown_connections 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 1 -- 172.21.15.119:0/2183810791 wait complete. 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.942+0000 7f168aa79740 20 client.25700 trim_cache size 0 max 16384 2024-03-28T18:44:58.947 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 msgr2=0x7f16680a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 0x7f16680a4f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x55ce0c6e3ff0 tx=0x7f1674015040 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 msgr2=0x7f1680198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f166c000c00 tx=0x7f166c0042f0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 shutdown_connections 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f16680a2a80 0x7f16680a4f40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f168010baf0 0x7f168019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1680102ff0 0x7f1680198320 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 --2- 172.21.15.119:0/1181920539 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f16801026d0 0x7f1680197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 >> 172.21.15.119:0/1181920539 conn(0x7f16800fbef0 msgr2=0x7f16800fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 shutdown_connections 2024-03-28T18:44:58.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:58.943+0000 7f1686800640 1 -- 172.21.15.119:0/1181920539 wait complete. 2024-03-28T18:44:58.962 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:58.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.158+0000 7f9176800640 1 -- 172.21.15.119:0/1152859376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101520 msgr2=0x7f91701019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.158+0000 7f9176800640 1 --2- 172.21.15.119:0/1152859376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101520 0x7f91701019a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f915c009930 tx=0x7f915c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 -- 172.21.15.119:0/1152859376 shutdown_connections 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 --2- 172.21.15.119:0/1152859376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101ee0 0x7f917010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 --2- 172.21.15.119:0/1152859376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101520 0x7f91701019a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 --2- 172.21.15.119:0/1152859376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 0x7f91701079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 -- 172.21.15.119:0/1152859376 >> 172.21.15.119:0/1152859376 conn(0x7f91700fd010 msgr2=0x7f91700ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 -- 172.21.15.119:0/1152859376 shutdown_connections 2024-03-28T18:44:59.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.159+0000 7f9176800640 1 -- 172.21.15.119:0/1152859376 wait complete. 2024-03-28T18:44:59.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.160+0000 7f9176800640 1 Processor -- start 2024-03-28T18:44:59.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.160+0000 7f9176800640 1 -- start start 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101ee0 0x7f9170198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 0x7f917019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701118c0 con 0x7f91701075e0 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170111740 con 0x7f9170101ee0 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9176800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9170111a40 con 0x7f9170101520 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52220/0 (socket says 172.21.15.119:52220) 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 -- 172.21.15.119:0/2107173220 learned_addr learned my addr 172.21.15.119:0/2107173220 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9174a00640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101ee0 0x7f9170198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175e00640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 0x7f917019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 -- 172.21.15.119:0/2107173220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101ee0 msgr2=0x7f9170198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101ee0 0x7f9170198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 -- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 msgr2=0x7f917019c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 0x7f917019c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.161+0000 7f9175400640 1 -- 172.21.15.119:0/2107173220 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917019ce10 con 0x7f9170101520 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f9175400640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f916400e9f0 tx=0x7f916400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f916e000640 1 -- 172.21.15.119:0/2107173220 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916400cd80 con 0x7f9170101520 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917019d100 con 0x7f9170101520 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f916e000640 1 -- 172.21.15.119:0/2107173220 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9164010910 con 0x7f9170101520 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f916e000640 1 -- 172.21.15.119:0/2107173220 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640188f0 con 0x7f9170101520 2024-03-28T18:44:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.162+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91701a4990 con 0x7f9170101520 2024-03-28T18:44:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.165+0000 7f916e000640 1 -- 172.21.15.119:0/2107173220 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f9164010ac0 con 0x7f9170101520 2024-03-28T18:44:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 Processor -- start 2024-03-28T18:44:59.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 -- start start 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 0x564592d4b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d53c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d56640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592caa2c0 con 0x564592d541f0 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592c52c90 con 0x564592d4b670 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.166+0000 7f917abd0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592ca6360 con 0x564592d4acc0 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f916e000640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 0x7f91580a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 0x564592d4b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 0x564592d4b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52226/0 (socket says 172.21.15.119:52226) 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175e00640 1 -- 172.21.15.119:0/2537667499 learned_addr learned my addr 172.21.15.119:0/2537667499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d53c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f916e000640 1 -- 172.21.15.119:0/2107173220 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f91640cf4a0 con 0x7f9170101520 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9174a00640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d56640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175e00640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 0x7f91580a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 -- 172.21.15.119:0/2537667499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 msgr2=0x564592d4b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 0x564592d4b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 -- 172.21.15.119:0/2537667499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 msgr2=0x564592d56640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d56640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9175400640 1 -- 172.21.15.119:0/2537667499 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564592c77080 con 0x564592d4b670 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.167+0000 7f9174a00640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d56640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.168+0000 7f9175400640 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d53c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f91640cca70 tx=0x7f9164020f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe5702b40ba6a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.168+0000 7f9162600640 1 -- 172.21.15.119:0/2537667499 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916409fa20 con 0x564592d4b670 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.168+0000 7f9162600640 1 -- 172.21.15.119:0/2537667499 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91640a4070 con 0x564592d4b670 2024-03-28T18:44:59.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.168+0000 7f9162600640 1 -- 172.21.15.119:0/2537667499 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640ccf80 con 0x564592d4b670 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f9175e00640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 0x7f91580a4ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9168009fd0 tx=0x7f9168004ab0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- 172.21.15.119:0/2537667499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 msgr2=0x564592d53c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d53c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f91640cca70 tx=0x7f9164020f70 comp rx=0 tx=0).stop 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- 172.21.15.119:0/2537667499 shutdown_connections 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d56640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d53c20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 --2- 172.21.15.119:0/2537667499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4acc0 0x564592d4b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- 172.21.15.119:0/2537667499 >> 172.21.15.119:0/2537667499 conn(0x564592d45390 msgr2=0x564592d45770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- 172.21.15.119:0/2537667499 shutdown_connections 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- 172.21.15.119:0/2537667499 wait complete. 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.169+0000 7f917abd0740 1 -- start start 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 Processor -- start 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 0x564592d53b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 0x564592d514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592d57100 con 0x564592d4acc0 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592c660d0 con 0x564592d4b670 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564592d57280 con 0x564592d541f0 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f917abd0740 10 client.0 _ll_register_callbacks cb 0x564592c6ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f9174a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f9175400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 0x564592d53b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f9174a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56940/0 (socket says 172.21.15.119:56940) 2024-03-28T18:44:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f9174a00640 1 -- 172.21.15.119:0/112612090 learned_addr learned my addr 172.21.15.119:0/112612090 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.170+0000 7f9175e00640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 0x564592d514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 -- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 msgr2=0x564592d514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 0x564592d514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 -- 172.21.15.119:0/112612090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 msgr2=0x564592d53b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 0x564592d53b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 -- 172.21.15.119:0/112612090 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645929be3f0 con 0x564592d4b670 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9175400640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 0x564592d53b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9175e00640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 0x564592d514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9174a00640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f915c036040 tx=0x7f915c02cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7986e510a837aed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9156a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c004240 con 0x564592d4b670 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91640a4070 con 0x564592d4b670 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.171+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f915c0043e0 con 0x564592d4b670 2024-03-28T18:44:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f915c0046c0 con 0x564592d4b670 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 10 client.25712 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564592a887d0 con 0x564592d4b670 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91640a4070 con 0x564592d4b670 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f915c0035f0 con 0x564592d4b670 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 10 client.25712 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 10 client.25712 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 20 client.25712 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592d64080 0x564592d66470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 4 client.25712 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.172+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564592cb7800 con 0x564592d64080 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.173+0000 7f9175400640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592d64080 0x564592d66470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.173+0000 7f9156a00640 10 client.25712 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.173+0000 7f9175400640 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592d64080 0x564592d66470 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e5afe7cb68795dd server_cookie=ccf97d8feb625e5e in_seq=0 out_seq=0 2024-03-28T18:44:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f9156a00640 1 -- 172.21.15.119:0/112612090 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564592cb7800 con 0x564592d64080 2024-03-28T18:44:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f9156a00640 10 client.25712 handle_command_reply: tid=0 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 client.25712 shutdown 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 msgr2=0x564592d50ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f915c036040 tx=0x7f915c02cdd0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 shutdown_connections 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592d64080 0x564592d66470 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d541f0 0x564592d514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4b670 0x564592d50ec0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 --2- 172.21.15.119:0/112612090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d4acc0 0x564592d53b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f9156a00640 0 client.25712 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 >> 172.21.15.119:0/112612090 conn(0x564592d45390 msgr2=0x564592d40160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 shutdown_connections 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 1 -- 172.21.15.119:0/112612090 wait complete. 2024-03-28T18:44:59.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.174+0000 7f917abd0740 20 client.25712 trim_cache size 0 max 16384 2024-03-28T18:44:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.191+0000 7f917abd0740 1 Processor -- start 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.191+0000 7f917abd0740 1 -- start start 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.191+0000 7f917abd0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f917abd0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d53b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f917abd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d4c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f917abd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701aaf50 con 0x564592d541f0 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f917abd0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701ac8d0 con 0x564592d4acc0 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f917abd0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701ab5b0 con 0x564592d4b670 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56948/0 (socket says 172.21.15.119:56948) 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d53b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175400640 1 -- 172.21.15.119:0/1604295234 learned_addr learned my addr 172.21.15.119:0/1604295234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175e00640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d4c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 -- 172.21.15.119:0/1604295234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 msgr2=0x564592d535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 -- 172.21.15.119:0/1604295234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 msgr2=0x564592d4c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d4c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9174a00640 1 -- 172.21.15.119:0/1604295234 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564592d4b410 con 0x564592d4b670 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175400640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.192+0000 7f9175e00640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d4c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f9174a00640 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d53b30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f915c02c160 tx=0x7f915c02c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bd68f92577edca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f9161c00640 1 -- 172.21.15.119:0/1604295234 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c009520 con 0x564592d4b670 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f9161c00640 1 -- 172.21.15.119:0/1604295234 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f915c0035f0 con 0x564592d4b670 2024-03-28T18:44:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f9161c00640 1 -- 172.21.15.119:0/1604295234 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c004ed0 con 0x564592d4b670 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f917abd0740 1 -- 172.21.15.119:0/1604295234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 msgr2=0x564592d53b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.193+0000 7f917abd0740 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d53b30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f915c02c160 tx=0x7f915c02c320 comp rx=0 tx=0).stop 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 -- 172.21.15.119:0/1604295234 shutdown_connections 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d4c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d53b30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 --2- 172.21.15.119:0/1604295234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 -- 172.21.15.119:0/1604295234 >> 172.21.15.119:0/1604295234 conn(0x564592d45390 msgr2=0x564592d4ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 -- 172.21.15.119:0/1604295234 shutdown_connections 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 -- 172.21.15.119:0/1604295234 wait complete. 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 -- start start 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.194+0000 7f917abd0740 1 Processor -- start 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d5c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d5a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701ac270 con 0x564592d541f0 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701abc10 con 0x564592d4acc0 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701a9110 con 0x564592d4b670 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f917abd0740 10 client.0 _ll_register_callbacks cb 0x7f91680093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d5a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9174a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d5c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56962/0 (socket says 172.21.15.119:56962) 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d5a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48894/0 (socket says 172.21.15.119:48894) 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 -- 172.21.15.119:0/1039419031 learned_addr learned my addr 172.21.15.119:0/1039419031 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 -- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 msgr2=0x564592d5c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d5c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 -- 172.21.15.119:0/1039419031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 msgr2=0x564592d5a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d5a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.195+0000 7f9175400640 1 -- 172.21.15.119:0/1039419031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564592d5aad0 con 0x564592d4acc0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9174a00640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d5c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9175400640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f916409b860 tx=0x7f9164097b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef0f6ba9feb1f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9156a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640b4060 con 0x564592d4acc0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f91640ccf20 con 0x564592d4acc0 2024-03-28T18:44:59.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f915c0035f0 con 0x564592d4acc0 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91640a4070 con 0x564592d4acc0 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f917abd0740 10 client.25718 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564592dc0560 con 0x564592d4acc0 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.196+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f915c0035f0 con 0x564592d4acc0 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f91640cf040 con 0x564592d4acc0 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 10 client.25718 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 10 client.25718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 20 client.25718 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592dc0850 0x564592dc2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 4 client.25718 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564592cb1b50 con 0x564592dc0850 2024-03-28T18:44:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.197+0000 7f9174a00640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592dc0850 0x564592dc2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f9156a00640 10 client.25718 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f9174a00640 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592dc0850 0x564592dc2c40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a57b7b6f05de3205 server_cookie=a03fcef5795be362 in_seq=0 out_seq=0 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f9156a00640 1 -- 172.21.15.119:0/1039419031 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564592cb1b50 con 0x564592dc0850 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f9156a00640 10 client.25718 handle_command_reply: tid=0 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 client.25718 shutdown 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 msgr2=0x564592d5be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f916409b860 tx=0x7f9164097b70 comp rx=0 tx=0).stop 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 shutdown_connections 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x564592dc0850 0x564592dc2c40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564592d541f0 0x564592d5a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564592d4b670 0x564592d5c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 --2- 172.21.15.119:0/1039419031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564592d4acc0 0x564592d5be40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f9156a00640 0 client.25718 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.198+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 >> 172.21.15.119:0/1039419031 conn(0x564592d45390 msgr2=0x564592d4c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 shutdown_connections 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f917abd0740 1 -- 172.21.15.119:0/1039419031 wait complete. 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f917abd0740 20 client.25718 trim_cache size 0 max 16384 2024-03-28T18:44:59.203 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 msgr2=0x7f91580a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 0x7f91580a4ec0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9168009fd0 tx=0x7f9168004ab0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 msgr2=0x7f9170197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f916400e9f0 tx=0x7f916400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.199+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 shutdown_connections 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91580a2a00 0x7f91580a4ec0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91701075e0 0x7f917019c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9170101ee0 0x7f9170198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 --2- 172.21.15.119:0/2107173220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9170101520 0x7f9170197e20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 >> 172.21.15.119:0/2107173220 conn(0x7f91700fd010 msgr2=0x7f91700fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 shutdown_connections 2024-03-28T18:44:59.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.200+0000 7f9176800640 1 -- 172.21.15.119:0/2107173220 wait complete. 2024-03-28T18:44:59.216 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:59.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:44:59.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 -- 172.21.15.119:0/949090371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d81026d0 msgr2=0x7f12d8102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 --2- 172.21.15.119:0/949090371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d81026d0 0x7f12d8102ab0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f12cc009900 tx=0x7f12cc02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 -- 172.21.15.119:0/949090371 shutdown_connections 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 --2- 172.21.15.119:0/949090371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d810df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 --2- 172.21.15.119:0/949090371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d8102ff0 0x7f12d810b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 --2- 172.21.15.119:0/949090371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d81026d0 0x7f12d8102ab0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 -- 172.21.15.119:0/949090371 >> 172.21.15.119:0/949090371 conn(0x7f12d80fbef0 msgr2=0x7f12d80fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 -- 172.21.15.119:0/949090371 shutdown_connections 2024-03-28T18:44:59.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.420+0000 7f12de000640 1 -- 172.21.15.119:0/949090371 wait complete. 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.421+0000 7f12de000640 1 Processor -- start 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.421+0000 7f12de000640 1 -- start start 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d81026d0 0x7f12d819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 0x7f12d819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8111900 con 0x7f12d81026d0 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8111780 con 0x7f12d810baf0 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12de000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8111a80 con 0x7f12d8102ff0 2024-03-28T18:44:59.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 0x7f12d819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dcc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d81026d0 0x7f12d819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56990/0 (socket says 172.21.15.119:56990) 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 0x7f12d819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52278/0 (socket says 172.21.15.119:52278) 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 -- 172.21.15.119:0/2277294693 learned_addr learned my addr 172.21.15.119:0/2277294693 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 -- 172.21.15.119:0/2277294693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 msgr2=0x7f12d819a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 0x7f12d819a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 -- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d81026d0 msgr2=0x7f12d819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.422+0000 7f12dd600640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d81026d0 0x7f12d819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12dd600640 1 -- 172.21.15.119:0/2277294693 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d819f030 con 0x7f12d810baf0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12dd600640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12d000efd0 tx=0x7f12d000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12d5600640 1 -- 172.21.15.119:0/2277294693 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d0003b70 con 0x7f12d810baf0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d819f320 con 0x7f12d810baf0 2024-03-28T18:44:59.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12d5600640 1 -- 172.21.15.119:0/2277294693 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d0003d10 con 0x7f12d810baf0 2024-03-28T18:44:59.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12d5600640 1 -- 172.21.15.119:0/2277294693 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00109f0 con 0x7f12d810baf0 2024-03-28T18:44:59.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.423+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d81a6bb0 con 0x7f12d810baf0 2024-03-28T18:44:59.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.426+0000 7f12e227d740 1 Processor -- start 2024-03-28T18:44:59.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.426+0000 7f12d5600640 1 -- 172.21.15.119:0/2277294693 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f12d0004020 con 0x7f12d810baf0 2024-03-28T18:44:59.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 -- start start 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0911762c0 con 0x55d091213260 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d09111ec90 con 0x55d0912128b0 2024-03-28T18:44:59.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12e227d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d091172360 con 0x55d09121bde0 2024-03-28T18:44:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57000/0 (socket says 172.21.15.119:57000) 2024-03-28T18:44:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12d7e00640 1 -- 172.21.15.119:0/3072319794 learned_addr learned my addr 172.21.15.119:0/3072319794 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.427+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 -- 172.21.15.119:0/3072319794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 msgr2=0x55d09121e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 -- 172.21.15.119:0/3072319794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 msgr2=0x55d09121b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 -- 172.21.15.119:0/3072319794 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d091143080 con 0x55d0912128b0 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.428+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009560 tx=0x7f12c4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10de225d0bcbd8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12c9c00640 1 -- 172.21.15.119:0/3072319794 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4013dc0 con 0x55d0912128b0 2024-03-28T18:44:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12c9c00640 1 -- 172.21.15.119:0/3072319794 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c4013120 con 0x55d0912128b0 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12c9c00640 1 -- 172.21.15.119:0/3072319794 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c40035f0 con 0x55d0912128b0 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12d5600640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 0x7f12c00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12d5600640 1 -- 172.21.15.119:0/2277294693 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f12d00cf480 con 0x7f12d810baf0 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12dcc00640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 0x7f12c00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12e227d740 1 -- 172.21.15.119:0/3072319794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 msgr2=0x55d091212c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.429+0000 7f12e227d740 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009560 tx=0x7f12c4005850 comp rx=0 tx=0).stop 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 -- 172.21.15.119:0/3072319794 shutdown_connections 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 --2- 172.21.15.119:0/3072319794 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d091212c90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 -- 172.21.15.119:0/3072319794 >> 172.21.15.119:0/3072319794 conn(0x55d09120cf80 msgr2=0x55d09120d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 -- 172.21.15.119:0/3072319794 shutdown_connections 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 -- 172.21.15.119:0/3072319794 wait complete. 2024-03-28T18:44:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12dcc00640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 0x7f12c00a4ec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12cc005ab0 tx=0x7f12cc02e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 -- start start 2024-03-28T18:44:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.430+0000 7f12e227d740 1 Processor -- start 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d09121b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d09121ecf0 con 0x55d0912128b0 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0911320d0 con 0x55d091213260 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d09121ee70 con 0x55d09121bde0 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12e227d740 10 client.0 _ll_register_callbacks cb 0x55d091136e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12dcc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d09121b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57016/0 (socket says 172.21.15.119:57016) 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 -- 172.21.15.119:0/3360236702 learned_addr learned my addr 172.21.15.119:0/3360236702 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 -- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 msgr2=0x55d0912190e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912190e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 -- 172.21.15.119:0/3360236702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 msgr2=0x55d09121b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.431+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d09121b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12d7e00640 1 -- 172.21.15.119:0/3360236702 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d090e8a3f0 con 0x55d091213260 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12dd600640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b0f6cb957326cd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12be000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4004250 con 0x55d091213260 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12c4013120 con 0x55d091213260 2024-03-28T18:44:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.432+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f12c40043f0 con 0x55d091213260 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12c40035f0 con 0x55d091213260 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 10 client.25730 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d090f547d0 con 0x55d091213260 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12c4013120 con 0x55d091213260 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f12c40207c0 con 0x55d091213260 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 10 client.25730 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 10 client.25730 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 20 client.25730 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d09122bb30 0x55d09122df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 4 client.25730 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d091183800 con 0x55d09122bb30 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.433+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d09122bb30 0x55d09122df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.434+0000 7f12be000640 10 client.25730 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.434+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d09122bb30 0x55d09122df20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd2768a757f5ccba server_cookie=fd57e84fb74ded21 in_seq=0 out_seq=0 2024-03-28T18:44:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12be000640 1 -- 172.21.15.119:0/3360236702 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d091183800 con 0x55d09122bb30 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12be000640 10 client.25730 handle_command_reply: tid=0 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 client.25730 shutdown 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 msgr2=0x55d091218ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f12c4006010 tx=0x7f12c4009600 comp rx=0 tx=0).stop 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 shutdown_connections 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d09122bb30 0x55d09122df20 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912190e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d091218ad0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12be000640 0 client.25730 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 --2- 172.21.15.119:0/3360236702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d09121b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 >> 172.21.15.119:0/3360236702 conn(0x55d09120cf80 msgr2=0x55d09120ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 shutdown_connections 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 1 -- 172.21.15.119:0/3360236702 wait complete. 2024-03-28T18:44:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.435+0000 7f12e227d740 20 client.25730 trim_cache size 0 max 16384 2024-03-28T18:44:59.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 Processor -- start 2024-03-28T18:44:59.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 -- start start 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d09120c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81af3b0 con 0x55d091213260 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81b0d30 con 0x55d0912128b0 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12e227d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81afa10 con 0x55d09121bde0 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12dcc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d09120c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52290/0 (socket says 172.21.15.119:52290) 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12d7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.449+0000 7f12dd600640 1 -- 172.21.15.119:0/3386686075 learned_addr learned my addr 172.21.15.119:0/3386686075 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 -- 172.21.15.119:0/3386686075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 msgr2=0x55d09120c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d09120c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 -- 172.21.15.119:0/3386686075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 msgr2=0x55d09121b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 -- 172.21.15.119:0/3386686075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d091217020 con 0x55d09121bde0 2024-03-28T18:44:59.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d09120c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12dd600640 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f12d000cba0 tx=0x7f12d0004920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a92be85ede9c832e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12c9200640 1 -- 172.21.15.119:0/3386686075 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00b9070 con 0x55d09121bde0 2024-03-28T18:44:59.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12c9200640 1 -- 172.21.15.119:0/3386686075 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d009bfb0 con 0x55d09121bde0 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.450+0000 7f12c9200640 1 -- 172.21.15.119:0/3386686075 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00b4070 con 0x55d09121bde0 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- 172.21.15.119:0/3386686075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 msgr2=0x55d09121b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f12d000cba0 tx=0x7f12d0004920 comp rx=0 tx=0).stop 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- 172.21.15.119:0/3386686075 shutdown_connections 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d09121b740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d091213260 0x55d09121b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 --2- 172.21.15.119:0/3386686075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d0912128b0 0x55d09120c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- 172.21.15.119:0/3386686075 >> 172.21.15.119:0/3386686075 conn(0x55d09120cf80 msgr2=0x55d09120d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- 172.21.15.119:0/3386686075 shutdown_connections 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- 172.21.15.119:0/3386686075 wait complete. 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.451+0000 7f12e227d740 1 -- start start 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 Processor -- start 2024-03-28T18:44:59.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d0912222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81b06d0 con 0x55d0912128b0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81b0070 con 0x55d091213260 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d81ad570 con 0x55d09121bde0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12e227d740 10 client.0 _ll_register_callbacks cb 0x7f12d00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12dcc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12dcc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48948/0 (socket says 172.21.15.119:48948) 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12dcc00640 1 -- 172.21.15.119:0/3450987970 learned_addr learned my addr 172.21.15.119:0/3450987970 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12dd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.452+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d0912222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 -- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 msgr2=0x55d0912801c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912801c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 -- 172.21.15.119:0/3450987970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 msgr2=0x55d0912222d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d0912222d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 -- 172.21.15.119:0/3450987970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d091280800 con 0x55d0912128b0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dd600640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d0912222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12dcc00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f12cc009900 tx=0x7f12cc02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f280a554edce1474 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12be000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc0046c0 con 0x55d0912128b0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f12cc004860 con 0x55d0912128b0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.453+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f12d009bfb0 con 0x55d0912128b0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12cc061050 con 0x55d0912128b0 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 10 client.15609 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d091287bb0 con 0x55d0912128b0 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12d009bfb0 con 0x55d0912128b0 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f12cc0050e0 con 0x55d0912128b0 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 10 client.15609 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 10 client.15609 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 20 client.15609 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d0912882e0 0x55d09128a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 4 client.15609 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.454+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d09117db50 con 0x55d0912882e0 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d0912882e0 0x55d09128a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12d7e00640 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d0912882e0 0x55d09128a700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f52a04d55ea273d4 server_cookie=b4295a7f3d462d59 in_seq=0 out_seq=0 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12be000640 10 client.15609 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12be000640 1 -- 172.21.15.119:0/3450987970 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d09117db50 con 0x55d0912882e0 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12be000640 10 client.15609 handle_command_reply: tid=0 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12e227d740 1 client.15609 shutdown 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 msgr2=0x55d091221d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.455+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f12cc009900 tx=0x7f12cc02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 shutdown_connections 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d0912882e0 0x55d09128a700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d09121bde0 0x55d0912801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12be000640 0 client.15609 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d091213260 0x55d0912222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 --2- 172.21.15.119:0/3450987970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d0912128b0 0x55d091221d90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 >> 172.21.15.119:0/3450987970 conn(0x55d09120cf80 msgr2=0x55d09120eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 shutdown_connections 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 1 -- 172.21.15.119:0/3450987970 wait complete. 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12e227d740 20 client.15609 trim_cache size 0 max 16384 2024-03-28T18:44:59.460 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 msgr2=0x7f12c00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.456+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 0x7f12c00a4ec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12cc005ab0 tx=0x7f12cc02e040 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 msgr2=0x7f12d819e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f12d000efd0 tx=0x7f12d000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 shutdown_connections 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12c00a2a00 0x7f12c00a4ec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f12d810baf0 0x7f12d819e900 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f12d8102ff0 0x7f12d819a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 --2- 172.21.15.119:0/2277294693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12d81026d0 0x7f12d819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 >> 172.21.15.119:0/2277294693 conn(0x7f12d80fbef0 msgr2=0x7f12d80fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 shutdown_connections 2024-03-28T18:44:59.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.457+0000 7f12de000640 1 -- 172.21.15.119:0/2277294693 wait complete. 2024-03-28T18:44:59.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:44:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2197888821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:59.474 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:44:59.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:44:59.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 -- 172.21.15.119:0/202910454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28074370 msgr2=0x7f1f2810d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 --2- 172.21.15.119:0/202910454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28074370 0x7f1f2810d890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009900 tx=0x7f1f1402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 -- 172.21.15.119:0/202910454 shutdown_connections 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 --2- 172.21.15.119:0/202910454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 0x7f1f281101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 --2- 172.21.15.119:0/202910454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28074370 0x7f1f2810d890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 --2- 172.21.15.119:0/202910454 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28073a50 0x7f1f28073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 -- 172.21.15.119:0/202910454 >> 172.21.15.119:0/202910454 conn(0x7f1f280fd040 msgr2=0x7f1f280ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 -- 172.21.15.119:0/202910454 shutdown_connections 2024-03-28T18:44:59.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.679+0000 7f1f27200640 1 -- 172.21.15.119:0/202910454 wait complete. 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.680+0000 7f1f27200640 1 Processor -- start 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 -- start start 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 0x7f1f2819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 0x7f1f2819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 0x7f1f2819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28113b40 con 0x7f1f2810ddd0 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281139c0 con 0x7f1f28074370 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f27200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28113cc0 con 0x7f1f28073a50 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 0x7f1f2819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 0x7f1f2819a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52318/0 (socket says 172.21.15.119:52318) 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f25e00640 1 -- 172.21.15.119:0/463438037 learned_addr learned my addr 172.21.15.119:0/463438037 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f25400640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 0x7f1f2819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.681+0000 7f1f26800640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 0x7f1f2819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 -- 172.21.15.119:0/463438037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 msgr2=0x7f1f2819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 0x7f1f2819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 -- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 msgr2=0x7f1f2819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 0x7f1f2819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 -- 172.21.15.119:0/463438037 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2819f020 con 0x7f1f28074370 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f25400640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 0x7f1f2819a5c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1f1402c210 tx=0x7f1f14005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f1ea00640 1 -- 172.21.15.119:0/463438037 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14042020 con 0x7f1f28074370 2024-03-28T18:44:59.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f1ea00640 1 -- 172.21.15.119:0/463438037 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f1402ce00 con 0x7f1f28074370 2024-03-28T18:44:59.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2819f2b0 con 0x7f1f28074370 2024-03-28T18:44:59.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f1ea00640 1 -- 172.21.15.119:0/463438037 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14003a50 con 0x7f1f28074370 2024-03-28T18:44:59.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.682+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f281a6ba0 con 0x7f1f28074370 2024-03-28T18:44:59.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:44:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2197888821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:59.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.685+0000 7f1f1ea00640 1 -- 172.21.15.119:0/463438037 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f1f14045020 con 0x7f1f28074370 2024-03-28T18:44:59.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.685+0000 7f1f2f98e740 1 Processor -- start 2024-03-28T18:44:59.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 -- start start 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca1260 0x5577d3ca9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 0x5577d3cac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3c042c0 con 0x5577d3ca1260 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3bacc90 con 0x5577d3ca9de0 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3c00360 con 0x5577d3ca08b0 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f26800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f26800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52334/0 (socket says 172.21.15.119:52334) 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f26800640 1 -- 172.21.15.119:0/1133779012 learned_addr learned my addr 172.21.15.119:0/1133779012 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f25400640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 0x5577d3cac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.686+0000 7f1f25e00640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca1260 0x5577d3ca9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 -- 172.21.15.119:0/1133779012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 msgr2=0x5577d3cac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 0x5577d3cac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 -- 172.21.15.119:0/1133779012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca1260 msgr2=0x5577d3ca9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca1260 0x5577d3ca9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 -- 172.21.15.119:0/1133779012 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577d3bd1080 con 0x5577d3ca08b0 2024-03-28T18:44:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f25400640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 0x5577d3cac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:44:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f26800640 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009f50 tx=0x7f1f200118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ef60aa0da91e423 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f13000640 1 -- 172.21.15.119:0/1133779012 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20004d20 con 0x5577d3ca08b0 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f13000640 1 -- 172.21.15.119:0/1133779012 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f20004050 con 0x5577d3ca08b0 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f13000640 1 -- 172.21.15.119:0/1133779012 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20005220 con 0x5577d3ca08b0 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.687+0000 7f1f1ea00640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 0x7f1f0c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f1ea00640 1 -- 172.21.15.119:0/463438037 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f140f0a70 con 0x7f1f28074370 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f25e00640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 0x7f1f0c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1133779012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 msgr2=0x5577d3ca0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009f50 tx=0x7f1f200118d0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1133779012 shutdown_connections 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca9de0 0x5577d3cac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca1260 0x5577d3ca9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1133779012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca0c90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1133779012 >> 172.21.15.119:0/1133779012 conn(0x5577d3c9af80 msgr2=0x5577d3c9b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.688+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1133779012 shutdown_connections 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.689+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1133779012 wait complete. 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.689+0000 7f1f25e00640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 0x7f1f0c0a4ec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5577d3c9bff0 tx=0x7f1f18005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.689+0000 7f1f2f98e740 1 -- start start 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.689+0000 7f1f2f98e740 1 Processor -- start 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3cacd50 con 0x5577d3ca9de0 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3bc00d0 con 0x5577d3ca1260 2024-03-28T18:44:59.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577d3caced0 con 0x5577d3ca08b0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f2f98e740 10 client.0 _ll_register_callbacks cb 0x5577d3bc4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52344/0 (socket says 172.21.15.119:52344) 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25e00640 1 -- 172.21.15.119:0/2323044592 learned_addr learned my addr 172.21.15.119:0/2323044592 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57062/0 (socket says 172.21.15.119:57062) 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f26800640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 -- 172.21.15.119:0/2323044592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 msgr2=0x5577d3ca9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 -- 172.21.15.119:0/2323044592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 msgr2=0x5577d3ca70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.690+0000 7f1f25400640 1 -- 172.21.15.119:0/2323044592 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577d39183f0 con 0x5577d3ca1260 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f25400640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1f140ee150 tx=0x7f1f140b9250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58e54f0589ba09df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140f0e60 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f140bd210 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f20004050 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f140bc910 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f2f98e740 10 client.25748 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5577d39e27d0 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f20004050 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f140c52c0 con 0x5577d3ca1260 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.691+0000 7f1f2f98e740 10 client.25748 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f2f98e740 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f2f98e740 20 client.25748 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3cb9c00 0x5577d3cbc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f2f98e740 4 client.25748 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577d3c11800 con 0x5577d3cb9c00 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f25e00640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3cb9c00 0x5577d3cbc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f25e00640 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3cb9c00 0x5577d3cbc030 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46b9e95f942aa340 server_cookie=38bda7aff3a24660 in_seq=0 out_seq=0 2024-03-28T18:44:59.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.692+0000 7f1f07400640 10 client.25748 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:59.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:44:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2197888821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.693+0000 7f1f07400640 1 -- 172.21.15.119:0/2323044592 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5577d3c11800 con 0x5577d3cb9c00 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.693+0000 7f1f07400640 10 client.25748 handle_command_reply: tid=0 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.693+0000 7f1f2f98e740 1 client.25748 shutdown 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 msgr2=0x5577d3ca6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1f140ee150 tx=0x7f1f140b9250 comp rx=0 tx=0).stop 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 shutdown_connections 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3cb9c00 0x5577d3cbc030 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3ca6ab0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/2323044592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca08b0 0x5577d3ca9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 >> 172.21.15.119:0/2323044592 conn(0x5577d3c9af80 msgr2=0x5577d3c9ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f07400640 0 client.25748 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 shutdown_connections 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 1 -- 172.21.15.119:0/2323044592 wait complete. 2024-03-28T18:44:59.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.694+0000 7f1f2f98e740 20 client.25748 trim_cache size 0 max 16384 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 Processor -- start 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 -- start start 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca08b0 0x5577d3ca91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281af3e0 con 0x5577d3ca9de0 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281b0d60 con 0x5577d3ca08b0 2024-03-28T18:44:59.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.710+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281afa40 con 0x5577d3ca1260 2024-03-28T18:44:59.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52350/0 (socket says 172.21.15.119:52350) 2024-03-28T18:44:59.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 -- 172.21.15.119:0/1885399549 learned_addr learned my addr 172.21.15.119:0/1885399549 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25e00640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca08b0 0x5577d3ca91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f26800640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 -- 172.21.15.119:0/1885399549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca08b0 msgr2=0x5577d3ca91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca08b0 0x5577d3ca91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 -- 172.21.15.119:0/1885399549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 msgr2=0x5577d3ca3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 -- 172.21.15.119:0/1885399549 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577d3c9f940 con 0x5577d3ca1260 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f26800640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f25400640 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f140ee1e0 tx=0x7f1f140b9380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65a4f32eca5cc93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f12600640 1 -- 172.21.15.119:0/1885399549 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140bcd80 con 0x5577d3ca1260 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f12600640 1 -- 172.21.15.119:0/1885399549 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f140bd6b0 con 0x5577d3ca1260 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.711+0000 7f1f12600640 1 -- 172.21.15.119:0/1885399549 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140052c0 con 0x5577d3ca1260 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1885399549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 msgr2=0x5577d3ca9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1f140ee1e0 tx=0x7f1f140b9380 comp rx=0 tx=0).stop 2024-03-28T18:44:59.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1885399549 shutdown_connections 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca9de0 0x5577d3ca3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca1260 0x5577d3ca9720 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/1885399549 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca08b0 0x5577d3ca91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1885399549 >> 172.21.15.119:0/1885399549 conn(0x5577d3c9af80 msgr2=0x5577d3c9cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1885399549 shutdown_connections 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.712+0000 7f1f2f98e740 1 -- 172.21.15.119:0/1885399549 wait complete. 2024-03-28T18:44:59.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 -- start start 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 Processor -- start 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca08b0 0x5577d3cb1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca9de0 0x5577d3cb07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281b0700 con 0x5577d3ca08b0 2024-03-28T18:44:59.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281b00a0 con 0x5577d3ca1260 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281ad5a0 con 0x5577d3ca9de0 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.713+0000 7f1f2f98e740 10 client.0 _ll_register_callbacks cb 0x7f1f140edfe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57084/0 (socket says 172.21.15.119:57084) 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 -- 172.21.15.119:0/761144453 learned_addr learned my addr 172.21.15.119:0/761144453 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25e00640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca08b0 0x5577d3cb1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f26800640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca9de0 0x5577d3cb07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 -- 172.21.15.119:0/761144453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca9de0 msgr2=0x5577d3cb07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca9de0 0x5577d3cb07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 -- 172.21.15.119:0/761144453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca08b0 msgr2=0x5577d3cb1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:44:59.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca08b0 0x5577d3cb1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 -- 172.21.15.119:0/761144453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577d3cb0fe0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f25400640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1f140bcf40 tx=0x7f1f1402bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d47f3c4488b6a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f07400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f140bcd80 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f140bd6b0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.714+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f140c54c0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f140bcfb0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 10 client.25754 fetch_fsmap learned FSMap version 28 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5577d3cb0fe0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f140bd6b0 con 0x5577d3ca1260 2024-03-28T18:44:59.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f140b7b20 con 0x5577d3ca1260 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 10 client.25754 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 10 client.25754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 20 client.25754 populate_metadata read hostname 'smithi119' 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3d166f0 0x5577d3d18ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 4 client.25754 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5577d3c0bb50 con 0x5577d3d166f0 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.715+0000 7f1f25e00640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3d166f0 0x5577d3d18ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f25e00640 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3d166f0 0x5577d3d18ae0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0cfd34015517e server_cookie=efd4ba8baef716f7 in_seq=0 out_seq=0 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f07400640 10 client.25754 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f07400640 1 -- 172.21.15.119:0/761144453 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5577d3c0bb50 con 0x5577d3d166f0 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f07400640 10 client.25754 handle_command_reply: tid=0 2024-03-28T18:44:59.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 client.25754 shutdown 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 msgr2=0x5577d3cb0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1f140bcf40 tx=0x7f1f1402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 shutdown_connections 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5577d3d166f0 0x5577d3d18ae0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577d3ca9de0 0x5577d3cb07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577d3ca1260 0x5577d3cb0250 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f07400640 0 client.25754 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 --2- 172.21.15.119:0/761144453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577d3ca08b0 0x5577d3cb1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 >> 172.21.15.119:0/761144453 conn(0x5577d3c9af80 msgr2=0x5577d3c9c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 shutdown_connections 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.716+0000 7f1f2f98e740 1 -- 172.21.15.119:0/761144453 wait complete. 2024-03-28T18:44:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f2f98e740 20 client.25754 trim_cache size 0 max 16384 2024-03-28T18:44:59.723 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:44:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 msgr2=0x7f1f0c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 0x7f1f0c0a4ec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x5577d3c9bff0 tx=0x7f1f18005f90 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 msgr2=0x7f1f2819a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 0x7f1f2819a5c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1f1402c210 tx=0x7f1f14005e80 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 shutdown_connections 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f0c0a2a00 0x7f1f0c0a4ec0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f2810ddd0 0x7f1f2819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f28074370 0x7f1f2819a5c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 --2- 172.21.15.119:0/463438037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f28073a50 0x7f1f2819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.717+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 >> 172.21.15.119:0/463438037 conn(0x7f1f280fd040 msgr2=0x7f1f280fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.718+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 shutdown_connections 2024-03-28T18:44:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:44:59.718+0000 7f1f27200640 1 -- 172.21.15.119:0/463438037 wait complete. 2024-03-28T18:45:00.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-mon[31813]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:45:00.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:00 smithi196 ceph-mon[36300]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:45:00.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:00 smithi131 ceph-mon[36065]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-28T18:45:00.740 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:45:00.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:45:00.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:45:00.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 -- 172.21.15.119:0/1142994441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 msgr2=0x7fa640102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 --2- 172.21.15.119:0/1142994441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640102ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa630009860 tx=0x7fa63002bc20 comp rx=0 tx=0).stop 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 -- 172.21.15.119:0/1142994441 shutdown_connections 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 --2- 172.21.15.119:0/1142994441 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa64010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 --2- 172.21.15.119:0/1142994441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa64010b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 --2- 172.21.15.119:0/1142994441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640102ab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 -- 172.21.15.119:0/1142994441 >> 172.21.15.119:0/1142994441 conn(0x7fa6400fbef0 msgr2=0x7fa6400fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.972+0000 7fa645400640 1 -- 172.21.15.119:0/1142994441 shutdown_connections 2024-03-28T18:45:00.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.973+0000 7fa645400640 1 -- 172.21.15.119:0/1142994441 wait complete. 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.973+0000 7fa645400640 1 Processor -- start 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 -- start start 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa640100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6401118b0 con 0x7fa6401026d0 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640111730 con 0x7fa64010baf0 2024-03-28T18:45:00.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa645400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640111a30 con 0x7fa640102ff0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa63f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa63f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52398/0 (socket says 172.21.15.119:52398) 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.974+0000 7fa63f400640 1 -- 172.21.15.119:0/2273087817 learned_addr learned my addr 172.21.15.119:0/2273087817 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63fe00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa644a00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa640100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 -- 172.21.15.119:0/2273087817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 msgr2=0x7fa640100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa640100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 -- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 msgr2=0x7fa640100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 -- 172.21.15.119:0/2273087817 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6401a6370 con 0x7fa640102ff0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa644a00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa640100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63f400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e910 tx=0x7fa62c00edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa63cc00640 1 -- 172.21.15.119:0/2273087817 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00cd80 con 0x7fa640102ff0 2024-03-28T18:45:00.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.975+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6401a6660 con 0x7fa640102ff0 2024-03-28T18:45:00.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.976+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6401a6c00 con 0x7fa640102ff0 2024-03-28T18:45:00.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.976+0000 7fa63cc00640 1 -- 172.21.15.119:0/2273087817 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa62c009460 con 0x7fa640102ff0 2024-03-28T18:45:00.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.976+0000 7fa63cc00640 1 -- 172.21.15.119:0/2273087817 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c010430 con 0x7fa640102ff0 2024-03-28T18:45:00.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 Processor -- start 2024-03-28T18:45:00.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 -- start start 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd2591250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd2593b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd24eceb0 con 0x558bd2591790 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2495a60 con 0x558bd2588290 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa64977d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd24e8f50 con 0x558bd2588bb0 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.979+0000 7fa63cc00640 1 -- 172.21.15.119:0/2273087817 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7fa62c0035f0 con 0x7fa640102ff0 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57134/0 (socket says 172.21.15.119:57134) 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 -- 172.21.15.119:0/1257574110 learned_addr learned my addr 172.21.15.119:0/1257574110 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa63fe00640 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd2591250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 -- 172.21.15.119:0/1257574110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 msgr2=0x558bd2591250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:00.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd2591250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 -- 172.21.15.119:0/1257574110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 msgr2=0x558bd2593b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:00.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd2593b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 -- 172.21.15.119:0/1257574110 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd22c97e0 con 0x558bd2588290 2024-03-28T18:45:00.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa63fe00640 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd2591250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:45:00.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.980+0000 7fa644a00640 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa638009fd0 tx=0x7fa638010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e3eee69b231ebca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:00.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa635200640 1 -- 172.21.15.119:0/1257574110 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638011700 con 0x558bd2588290 2024-03-28T18:45:00.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa635200640 1 -- 172.21.15.119:0/1257574110 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa638011d80 con 0x558bd2588290 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa635200640 1 -- 172.21.15.119:0/1257574110 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63800e3c0 con 0x558bd2588290 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 -- 172.21.15.119:0/1257574110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 msgr2=0x558bd2588670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa638009fd0 tx=0x7fa638010860 comp rx=0 tx=0).stop 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 -- 172.21.15.119:0/1257574110 shutdown_connections 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd2593b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd2591250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 --2- 172.21.15.119:0/1257574110 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd2588670 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa64977d740 1 -- 172.21.15.119:0/1257574110 >> 172.21.15.119:0/1257574110 conn(0x558bd25829c0 msgr2=0x558bd2582da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.981+0000 7fa63cc00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 0x7fa6240a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.982+0000 7fa64977d740 1 -- 172.21.15.119:0/1257574110 shutdown_connections 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.982+0000 7fa63cc00640 1 -- 172.21.15.119:0/2273087817 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa62c0cf2a0 con 0x7fa640102ff0 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.982+0000 7fa64977d740 1 -- 172.21.15.119:0/1257574110 wait complete. 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.982+0000 7fa644a00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 0x7fa6240a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.982+0000 7fa64977d740 1 -- start start 2024-03-28T18:45:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 Processor -- start 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa644a00640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 0x7fa6240a4f40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa638005cc0 tx=0x7fa638013040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd258e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd258e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2594580 con 0x558bd2591790 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd24a8eb0 con 0x558bd2588290 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.983+0000 7fa64977d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bd2594700 con 0x558bd2588bb0 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa64977d740 10 client.0 _ll_register_callbacks cb 0x558bd249d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57144/0 (socket says 172.21.15.119:57144) 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 -- 172.21.15.119:0/127714103 learned_addr learned my addr 172.21.15.119:0/127714103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63f400640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd258e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa644a00640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd258e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 -- 172.21.15.119:0/127714103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 msgr2=0x558bd258e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd258e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 -- 172.21.15.119:0/127714103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 msgr2=0x558bd258e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd258e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 -- 172.21.15.119:0/127714103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd2329f70 con 0x558bd2588290 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa63fe00640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa630005b60 tx=0x7fa630004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a7739abc93287a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.984+0000 7fa629600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630039070 con 0x558bd2588290 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa638011d80 con 0x558bd2588290 2024-03-28T18:45:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa630034850 con 0x558bd2588290 2024-03-28T18:45:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa63004c050 con 0x558bd2588290 2024-03-28T18:45:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa64977d740 10 client.25772 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558bd21c1170 con 0x558bd2588290 2024-03-28T18:45:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.985+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa638011d80 con 0x558bd2588290 2024-03-28T18:45:00.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa630047030 con 0x558bd2588290 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 10 client.25772 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 20 client.25772 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd259a8e0 0x558bd259ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 4 client.25772 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558bd24fa3f0 con 0x558bd259a8e0 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.986+0000 7fa63f400640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd259a8e0 0x558bd259ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa63f400640 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd259a8e0 0x558bd259ccf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f88cdec96ebd53d5 server_cookie=7e895ddf5ec04b25 in_seq=0 out_seq=0 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa629600640 10 client.25772 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa629600640 1 -- 172.21.15.119:0/127714103 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558bd24fa3f0 con 0x558bd259a8e0 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa629600640 10 client.25772 handle_command_reply: tid=0 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa64977d740 1 client.25772 shutdown 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 msgr2=0x558bd25910d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.987+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa630005b60 tx=0x7fa630004270 comp rx=0 tx=0).stop 2024-03-28T18:45:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 shutdown_connections 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd259a8e0 0x558bd259ccf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2591790 0x558bd258e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2588bb0 0x558bd258e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa629600640 0 client.25772 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 --2- 172.21.15.119:0/127714103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588290 0x558bd25910d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 >> 172.21.15.119:0/127714103 conn(0x558bd25829c0 msgr2=0x558bd2584710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 shutdown_connections 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 1 -- 172.21.15.119:0/127714103 wait complete. 2024-03-28T18:45:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:00.988+0000 7fa64977d740 20 client.25772 trim_cache size 0 max 16384 2024-03-28T18:45:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.005+0000 7fa64977d740 1 Processor -- start 2024-03-28T18:45:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.005+0000 7fa64977d740 1 -- start start 2024-03-28T18:45:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd2597060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd2590b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000e720 con 0x558bd2588290 2024-03-28T18:45:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6200100a0 con 0x558bd2588bb0 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa64977d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000ed80 con 0x558bd2591790 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd2597060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49074/0 (socket says 172.21.15.119:49074) 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 -- 172.21.15.119:0/1650016408 learned_addr learned my addr 172.21.15.119:0/1650016408 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa644a00640 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd2590b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 -- 172.21.15.119:0/1650016408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 msgr2=0x558bd2590b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd2590b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 -- 172.21.15.119:0/1650016408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 msgr2=0x558bd2597060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd2597060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63fe00640 1 -- 172.21.15.119:0/1650016408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bd2583bf0 con 0x558bd2588290 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.006+0000 7fa63f400640 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd2597060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:01.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.007+0000 7fa63fe00640 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa63002cdd0 tx=0x7fa63002bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3008b507d72820e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.007+0000 7fa62be00640 1 -- 172.21.15.119:0/1650016408 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630039040 con 0x558bd2588290 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.007+0000 7fa62be00640 1 -- 172.21.15.119:0/1650016408 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa63003a040 con 0x558bd2588290 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.007+0000 7fa62be00640 1 -- 172.21.15.119:0/1650016408 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa630003ad0 con 0x558bd2588290 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- 172.21.15.119:0/1650016408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 msgr2=0x558bd25a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa63002cdd0 tx=0x7fa63002bc20 comp rx=0 tx=0).stop 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- 172.21.15.119:0/1650016408 shutdown_connections 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd2590b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd2597060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 --2- 172.21.15.119:0/1650016408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd25a5a60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- 172.21.15.119:0/1650016408 >> 172.21.15.119:0/1650016408 conn(0x558bd25829c0 msgr2=0x558bd2596950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- 172.21.15.119:0/1650016408 shutdown_connections 2024-03-28T18:45:01.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- 172.21.15.119:0/1650016408 wait complete. 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 -- start start 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.008+0000 7fa64977d740 1 Processor -- start 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd258b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd258b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000fa40 con 0x558bd2588290 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000f3e0 con 0x558bd2588bb0 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62000c8e0 con 0x558bd2591790 2024-03-28T18:45:01.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa64977d740 10 client.0 _ll_register_callbacks cb 0x7fa630037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:01.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa63fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49090/0 (socket says 172.21.15.119:49090) 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa63fe00640 1 -- 172.21.15.119:0/33193420 learned_addr learned my addr 172.21.15.119:0/33193420 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa63f400640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd258b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.009+0000 7fa644a00640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd258b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 -- 172.21.15.119:0/33193420 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 msgr2=0x558bd258b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd258b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 -- 172.21.15.119:0/33193420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 msgr2=0x558bd258acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 -- 172.21.15.119:0/33193420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bd260dd40 con 0x558bd2588bb0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63fe00640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa63f400640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd258b1f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0cc870 tx=0x7fa62c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0b00b5bc8cbb9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa629600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c0b3070 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa62c0999d0 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.010+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa63003a040 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa62c09b6d0 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 10 client.25778 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558bd260f0c0 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa63003a040 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa62c0ccb20 con 0x558bd2588bb0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 10 client.25778 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 10 client.25778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 20 client.25778 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd260f870 0x558bd2611c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 4 client.25778 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558bd24f4740 con 0x558bd260f870 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.011+0000 7fa63fe00640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd260f870 0x558bd2611c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.012+0000 7fa629600640 10 client.25778 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.012+0000 7fa63fe00640 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd260f870 0x558bd2611c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35b19322e3a02b7e server_cookie=c9af0638dcca56d8 in_seq=0 out_seq=0 2024-03-28T18:45:01.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa629600640 1 -- 172.21.15.119:0/33193420 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558bd24f4740 con 0x558bd260f870 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa629600640 10 client.25778 handle_command_reply: tid=0 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 client.25778 shutdown 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 msgr2=0x558bd258b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd258b1f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0cc870 tx=0x7fa62c004730 comp rx=0 tx=0).stop 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 shutdown_connections 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558bd260f870 0x558bd2611c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558bd2591790 0x558bd258b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558bd2588bb0 0x558bd258b1f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa629600640 0 client.25778 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 --2- 172.21.15.119:0/33193420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558bd2588290 0x558bd258acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 >> 172.21.15.119:0/33193420 conn(0x558bd25829c0 msgr2=0x558bd2581a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.013+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 shutdown_connections 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa64977d740 1 -- 172.21.15.119:0/33193420 wait complete. 2024-03-28T18:45:01.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa64977d740 20 client.25778 trim_cache size 0 max 16384 2024-03-28T18:45:01.016 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:45:01.016 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:45:01.016 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "dbac3cd3-4e83-4e35-91ee-8702c28eb0ad", 2024-03-28T18:45:01.016 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:45:01.016 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 msgr2=0x7fa6240a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 0x7fa6240a4f40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa638005cc0 tx=0x7fa638013040 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 msgr2=0x7fa640100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.014+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e910 tx=0x7fa62c00edd0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 shutdown_connections 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa6240a2a80 0x7fa6240a4f40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa64010baf0 0x7fa640100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640102ff0 0x7fa640100860 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 --2- 172.21.15.119:0/2273087817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6401026d0 0x7fa640100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 >> 172.21.15.119:0/2273087817 conn(0x7fa6400fbef0 msgr2=0x7fa6400fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 shutdown_connections 2024-03-28T18:45:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.015+0000 7fa645400640 1 -- 172.21.15.119:0/2273087817 wait complete. 2024-03-28T18:45:01.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:45:01.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:45:01.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:45:00.859+0000 7f09af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:45:01.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:45:00.859+0000 7f09af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:45:01.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:45:00.859+0000 7f09af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:45:01.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:45:00.859+0000 7f09af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:45:01.068 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:45:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:45:00.859+0000 7f09af600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T18:45:01.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 -- 172.21.15.119:0/1457842375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 msgr2=0x7f84c01008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 --2- 172.21.15.119:0/1457842375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c01008a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009900 tx=0x7f84b002bd10 comp rx=0 tx=0).stop 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 -- 172.21.15.119:0/1457842375 shutdown_connections 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 --2- 172.21.15.119:0/1457842375 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c01058c0 0x7f84c0105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 --2- 172.21.15.119:0/1457842375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c0100de0 0x7f84c01052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 --2- 172.21.15.119:0/1457842375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c01008a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 -- 172.21.15.119:0/1457842375 >> 172.21.15.119:0/1457842375 conn(0x7f84c00fbef0 msgr2=0x7f84c00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 -- 172.21.15.119:0/1457842375 shutdown_connections 2024-03-28T18:45:01.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.228+0000 7f84c5400640 1 -- 172.21.15.119:0/1457842375 wait complete. 2024-03-28T18:45:01.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.230+0000 7f84c5400640 1 Processor -- start 2024-03-28T18:45:01.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.230+0000 7f84c5400640 1 -- start start 2024-03-28T18:45:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.230+0000 7f84c5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.230+0000 7f84c5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 0x7f84c010d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.230+0000 7f84c5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 0x7f84c010dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84c5400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0117f50 con 0x7f84c01058c0 2024-03-28T18:45:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84c5400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c0117dd0 con 0x7f84c0100de0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84c5400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01180d0 con 0x7f84c01004c0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52460/0 (socket says 172.21.15.119:52460) 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 -- 172.21.15.119:0/294994728 learned_addr learned my addr 172.21.15.119:0/294994728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bf400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 0x7f84c010d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84c4a00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 0x7f84c010dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 -- 172.21.15.119:0/294994728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 msgr2=0x7f84c010d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 0x7f84c010d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 -- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 msgr2=0x7f84c010dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 0x7f84c010dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bfe00640 1 -- 172.21.15.119:0/294994728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c010e2a0 con 0x7f84c01004c0 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84bf400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 0x7f84c010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.231+0000 7f84c4a00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 0x7f84c010dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:45:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84bfe00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b0004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84bcc00640 1 -- 172.21.15.119:0/294994728 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0042020 con 0x7f84c01004c0 2024-03-28T18:45:01.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c010e530 con 0x7f84c01004c0 2024-03-28T18:45:01.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84bcc00640 1 -- 172.21.15.119:0/294994728 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b0005000 con 0x7f84c01004c0 2024-03-28T18:45:01.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c0112490 con 0x7f84c01004c0 2024-03-28T18:45:01.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.232+0000 7f84bcc00640 1 -- 172.21.15.119:0/294994728 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0003b20 con 0x7f84c01004c0 2024-03-28T18:45:01.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 Processor -- start 2024-03-28T18:45:01.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 -- start start 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84bcc00640 1 -- 172.21.15.119:0/294994728 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 25) v1 ==== 137467+0+0 (secure 0 0 0) 0x7f84b0045020 con 0x7f84c01004c0 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38d7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f383c250 con 0x5578f38d8dd0 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f37e4a30 con 0x5578f38e1460 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.237+0000 7f84c964c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f38382f0 con 0x5578f38d80f0 2024-03-28T18:45:01.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52464/0 (socket says 172.21.15.119:52464) 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 -- 172.21.15.119:0/2856255037 learned_addr learned my addr 172.21.15.119:0/2856255037 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84c4a00640 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38d7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 -- 172.21.15.119:0/2856255037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 msgr2=0x5578f38e3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 -- 172.21.15.119:0/2856255037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 msgr2=0x5578f38d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38d7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 -- 172.21.15.119:0/2856255037 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578f37cf690 con 0x5578f38d80f0 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84c4a00640 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38d7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:01.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.238+0000 7f84bfe00640 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee290 tx=0x7f84b00f4c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b829bd1a9b21e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84b5200640 1 -- 172.21.15.119:0/2856255037 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00f9ce0 con 0x5578f38d80f0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84bcc00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 0x7f84a40a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84b5200640 1 -- 172.21.15.119:0/2856255037 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b01038e0 con 0x5578f38d80f0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84b5200640 1 -- 172.21.15.119:0/2856255037 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00fb7a0 con 0x5578f38d80f0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84bcc00640 1 -- 172.21.15.119:0/294994728 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84b00f0bb0 con 0x7f84c01004c0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.239+0000 7f84c4a00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 0x7f84a40a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 -- 172.21.15.119:0/2856255037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 msgr2=0x5578f38d8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee290 tx=0x7f84b00f4c40 comp rx=0 tx=0).stop 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 -- 172.21.15.119:0/2856255037 shutdown_connections 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38d8570 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 --2- 172.21.15.119:0/2856255037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38d7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 -- 172.21.15.119:0/2856255037 >> 172.21.15.119:0/2856255037 conn(0x5578f389f200 msgr2=0x5578f38d2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c4a00640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 0x7f84a40a4f40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84b8005fd0 tx=0x7f84b8009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 -- 172.21.15.119:0/2856255037 shutdown_connections 2024-03-28T18:45:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.240+0000 7f84c964c740 1 -- 172.21.15.119:0/2856255037 wait complete. 2024-03-28T18:45:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.241+0000 7f84c964c740 1 -- start start 2024-03-28T18:45:01.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.241+0000 7f84c964c740 1 Processor -- start 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.241+0000 7f84c964c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.241+0000 7f84c964c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38ddf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c964c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38de450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c964c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f38e4090 con 0x5578f38e1460 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c964c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f37f7e80 con 0x5578f38d8dd0 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c964c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578f38e4210 con 0x5578f38d80f0 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c964c740 10 client.0 _ll_register_callbacks cb 0x5578f380a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:01.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52480/0 (socket says 172.21.15.119:52480) 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 -- 172.21.15.119:0/733470267 learned_addr learned my addr 172.21.15.119:0/733470267 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bf400640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38ddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84c4a00640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38de450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 -- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 msgr2=0x5578f38ddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38ddf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 -- 172.21.15.119:0/733470267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 msgr2=0x5578f38de450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38de450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 -- 172.21.15.119:0/733470267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578f361a7d0 con 0x5578f38d80f0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.242+0000 7f84bfe00640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee8a0 tx=0x7f84b0103f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3446fd5ff02e16ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84a9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00b93f0 con 0x5578f38d80f0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84b00051b0 con 0x5578f38d80f0 2024-03-28T18:45:01.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84b01038e0 con 0x5578f38d80f0 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b00bcab0 con 0x5578f38d80f0 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84c964c740 10 client.25810 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578f35503f0 con 0x5578f38d80f0 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.243+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b01038e0 con 0x5578f38d80f0 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f84b002cb10 con 0x5578f38d80f0 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 10 client.25810 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 10 client.25810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 20 client.25810 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f38f0f70 0x5578f38f3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 4 client.25810 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578f3849790 con 0x5578f38f0f70 2024-03-28T18:45:01.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.244+0000 7f84bf400640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f38f0f70 0x5578f38f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.245+0000 7f84bf400640 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f38f0f70 0x5578f38f3360 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1caf8bcc1f76132 server_cookie=9da8ceee74b249b4 in_seq=0 out_seq=0 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.245+0000 7f84a9600640 10 client.25810 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.245+0000 7f84a9600640 1 -- 172.21.15.119:0/733470267 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578f3849790 con 0x5578f38f0f70 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.245+0000 7f84a9600640 10 client.25810 handle_command_reply: tid=0 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.245+0000 7f84c964c740 1 client.25810 shutdown 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 msgr2=0x5578f38e0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee8a0 tx=0x7f84b0103f40 comp rx=0 tx=0).stop 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 shutdown_connections 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f38f0f70 0x5578f38f3360 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38de450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38ddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 --2- 172.21.15.119:0/733470267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0b80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84a9600640 0 client.25810 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 >> 172.21.15.119:0/733470267 conn(0x5578f389f200 msgr2=0x5578f38d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 shutdown_connections 2024-03-28T18:45:01.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 1 -- 172.21.15.119:0/733470267 wait complete. 2024-03-28T18:45:01.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.246+0000 7f84c964c740 20 client.25810 trim_cache size 0 max 16384 2024-03-28T18:45:01.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 Processor -- start 2024-03-28T18:45:01.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 -- start start 2024-03-28T18:45:01.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38e0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01bbf70 con 0x5578f38d8dd0 2024-03-28T18:45:01.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01bd8f0 con 0x5578f38e1460 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.279+0000 7f84c964c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01bc5d0 con 0x5578f38d80f0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84c4a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52494/0 (socket says 172.21.15.119:52494) 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 -- 172.21.15.119:0/450370200 learned_addr learned my addr 172.21.15.119:0/450370200 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bf400640 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38e0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84c4a00640 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57210/0 (socket says 172.21.15.119:57210) 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 -- 172.21.15.119:0/450370200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 msgr2=0x5578f38e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 -- 172.21.15.119:0/450370200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 msgr2=0x5578f38e0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38e0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 -- 172.21.15.119:0/450370200 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578f35100f0 con 0x5578f38d80f0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.280+0000 7f84bfe00640 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee380 tx=0x7f84b00c12f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2f2e1dce3ccda2a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84abe00640 1 -- 172.21.15.119:0/450370200 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00bcc90 con 0x5578f38d80f0 2024-03-28T18:45:01.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84abe00640 1 -- 172.21.15.119:0/450370200 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b00f4e90 con 0x5578f38d80f0 2024-03-28T18:45:01.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84abe00640 1 -- 172.21.15.119:0/450370200 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00bcf20 con 0x5578f38d80f0 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 -- 172.21.15.119:0/450370200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 msgr2=0x5578f38e0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ee380 tx=0x7f84b00c12f0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 -- 172.21.15.119:0/450370200 shutdown_connections 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38e1460 0x5578f38e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38d8dd0 0x5578f38e0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.281+0000 7f84c964c740 1 --2- 172.21.15.119:0/450370200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38e0640 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.282+0000 7f84c964c740 1 -- 172.21.15.119:0/450370200 >> 172.21.15.119:0/450370200 conn(0x5578f389f200 msgr2=0x5578f38d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.282+0000 7f84c964c740 1 -- 172.21.15.119:0/450370200 shutdown_connections 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.282+0000 7f84c964c740 1 -- 172.21.15.119:0/450370200 wait complete. 2024-03-28T18:45:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.282+0000 7f84c964c740 1 -- start start 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.282+0000 7f84c964c740 1 Processor -- start 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38db6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38dbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01bd290 con 0x5578f38e1460 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01bcc30 con 0x5578f38d8dd0 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c01ba130 con 0x5578f38d80f0 2024-03-28T18:45:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c964c740 10 client.0 _ll_register_callbacks cb 0x7f84c0069860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52508/0 (socket says 172.21.15.119:52508) 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84bfe00640 1 -- 172.21.15.119:0/591111138 learned_addr learned my addr 172.21.15.119:0/591111138 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84bf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38db6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.283+0000 7f84c4a00640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38dbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 -- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 msgr2=0x5578f38db6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38db6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 -- 172.21.15.119:0/591111138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 msgr2=0x5578f38dbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38dbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 -- 172.21.15.119:0/591111138 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578f38eb910 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84c4a00640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38dbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bf400640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38db6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84bfe00640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f84b0003630 tx=0x7f84b0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e57e15c65b9a5ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84a9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00b93f0 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84b00bb450 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84b00f4e90 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.284+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b00bd1a0 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 10 client.25816 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578f38eb910 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b00f4e90 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f84b00c5460 con 0x5578f38d80f0 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 10 client.25816 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:01.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 10 client.25816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 20 client.25816 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f394cb50 0x5578f394ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 4 client.25816 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5578f3843ae0 con 0x5578f394cb50 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.285+0000 7f84bf400640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f394cb50 0x5578f394ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84bf400640 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f394cb50 0x5578f394ef40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9fb60a4452096f4 server_cookie=b540da91149a2806 in_seq=0 out_seq=0 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84a9600640 10 client.25816 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84a9600640 1 -- 172.21.15.119:0/591111138 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5578f3843ae0 con 0x5578f394cb50 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84a9600640 10 client.25816 handle_command_reply: tid=0 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84c964c740 1 client.25816 shutdown 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 msgr2=0x5578f38db190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.286+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f84b0003630 tx=0x7f84b0005fd0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 shutdown_connections 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578f394cb50 0x5578f394ef40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578f38e1460 0x5578f38dbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578f38d8dd0 0x5578f38db6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84a9600640 0 client.25816 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 --2- 172.21.15.119:0/591111138 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578f38d80f0 0x5578f38db190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 >> 172.21.15.119:0/591111138 conn(0x5578f389f200 msgr2=0x5578f38dc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 shutdown_connections 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 1 -- 172.21.15.119:0/591111138 wait complete. 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.287+0000 7f84c964c740 20 client.25816 trim_cache size 0 max 16384 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "dbac3cd3-4e83-4e35-91ee-8702c28eb0ad": { 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "dbac3cd3-4e83-4e35-91ee-8702c28eb0ad", 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:45:01.290 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 msgr2=0x7f84a40a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 0x7f84a40a4f40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f84b8005fd0 tx=0x7f84b8009500 comp rx=0 tx=0).stop 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 msgr2=0x7f84c0114680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f84b0000c00 tx=0x7f84b0004360 comp rx=0 tx=0).stop 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 shutdown_connections 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84a40a2a80 0x7f84a40a4f40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84c01058c0 0x7f84c010dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84c0100de0 0x7f84c010d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 --2- 172.21.15.119:0/294994728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84c01004c0 0x7f84c0114680 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 >> 172.21.15.119:0/294994728 conn(0x7f84c00fbef0 msgr2=0x7f84c0103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 shutdown_connections 2024-03-28T18:45:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:01.288+0000 7f84c5400640 1 -- 172.21.15.119:0/294994728 wait complete. 2024-03-28T18:45:01.311 INFO:tasks.cephfs.filesystem:scrub status for tag:dbac3cd3-4e83-4e35-91ee-8702c28eb0ad - {'path': '/', 'tag': 'dbac3cd3-4e83-4e35-91ee-8702c28eb0ad', 'options': 'recursive,force'} 2024-03-28T18:45:02.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:45:02.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:45:02.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:45:02.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:02 smithi196 ceph-mon[36300]: mgrmap e26: x(active, since 58s), standbys: y, z 2024-03-28T18:45:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:45:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:45:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:45:02.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:02 smithi131 ceph-mon[36065]: mgrmap e26: x(active, since 58s), standbys: y, z 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T18:45:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:02 smithi119 ceph-mon[31813]: mgrmap e26: x(active, since 58s), standbys: y, z 2024-03-28T18:45:04.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:04 smithi196 ceph-mon[36300]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:04.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:04 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:45:04.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:04 smithi131 ceph-mon[36065]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:04.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:04 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:45:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:04 smithi119 ceph-mon[31813]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:04 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:45:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:06 smithi196 ceph-mon[36300]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:06 smithi131 ceph-mon[36065]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:06.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:06 smithi119 ceph-mon[31813]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:08.621 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:08 smithi119 ceph-mon[31813]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:09.266 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:08 smithi131 ceph-mon[36065]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:09.266 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:08 smithi196 ceph-mon[36300]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:10 smithi196 ceph-mon[36300]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T18:45:10.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:10 smithi131 ceph-mon[36065]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T18:45:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:10 smithi119 ceph-mon[31813]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-03-28T18:45:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:12 smithi196 ceph-mon[36300]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:45:12.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:12 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:12 smithi131 ceph-mon[36065]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:45:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:12 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:12 smithi119 ceph-mon[31813]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-28T18:45:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:12 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:45:13.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:13 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:13.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:13 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:13.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:13 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:45:13.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:13 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:13.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:13 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:13.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:13 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:45:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:13 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:45:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:13 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:45:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:13 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:45:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:14 smithi196 ceph-mon[36300]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:14 smithi131 ceph-mon[36065]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:14 smithi119 ceph-mon[31813]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T18:45:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:16 smithi196 ceph-mon[36300]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:16 smithi131 ceph-mon[36065]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:16 smithi119 ceph-mon[31813]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T18:45:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:18 smithi196 ceph-mon[36300]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:45:18.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:18 smithi131 ceph-mon[36065]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:45:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:18 smithi119 ceph-mon[31813]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:45:20.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:20 smithi196 ceph-mon[36300]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:45:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:20 smithi131 ceph-mon[36065]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:45:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:20 smithi119 ceph-mon[31813]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:45:22.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:22 smithi196 ceph-mon[36300]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:45:22.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:22 smithi131 ceph-mon[36065]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:45:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:22 smithi119 ceph-mon[31813]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T18:45:24.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:24 smithi196 ceph-mon[36300]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:24.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:24 smithi131 ceph-mon[36065]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:24 smithi119 ceph-mon[31813]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:26.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:26 smithi196 ceph-mon[36300]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:45:26.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:26 smithi131 ceph-mon[36065]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:45:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:26 smithi119 ceph-mon[31813]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:45:28.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:28 smithi196 ceph-mon[36300]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:28 smithi131 ceph-mon[36065]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:28 smithi119 ceph-mon[31813]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:30 smithi196 ceph-mon[36300]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T18:45:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:30 smithi131 ceph-mon[36065]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T18:45:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:30 smithi119 ceph-mon[31813]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-28T18:45:31.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:45:31.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:45:31.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 -- 172.21.15.119:0/2277361503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 msgr2=0x7f8f38103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/2277361503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f38103cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009930 tx=0x7f8f2402bc20 comp rx=0 tx=0).stop 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 -- 172.21.15.119:0/2277361503 shutdown_connections 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/2277361503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 0x7f8f38110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/2277361503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f38103cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/2277361503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 0x7f8f38109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 -- 172.21.15.119:0/2277361503 >> 172.21.15.119:0/2277361503 conn(0x7f8f380ff350 msgr2=0x7f8f38101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 -- 172.21.15.119:0/2277361503 shutdown_connections 2024-03-28T18:45:31.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.529+0000 7f8f3ce00640 1 -- 172.21.15.119:0/2277361503 wait complete. 2024-03-28T18:45:31.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.530+0000 7f8f3ce00640 1 Processor -- start 2024-03-28T18:45:31.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 -- start start 2024-03-28T18:45:31.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 0x7f8f3819a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 0x7f8f3819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38113c60 con 0x7f8f38109850 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38113ae0 con 0x7f8f38104200 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f3ce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38113de0 con 0x7f8f38103840 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40434/0 (socket says 172.21.15.119:40434) 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f35e00640 1 -- 172.21.15.119:0/742024553 learned_addr learned my addr 172.21.15.119:0/742024553 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f35400640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 0x7f8f3819a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.531+0000 7f8f36800640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 0x7f8f3819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 -- 172.21.15.119:0/742024553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 msgr2=0x7f8f3819a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 0x7f8f3819a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 -- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 msgr2=0x7f8f3819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 0x7f8f3819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 -- 172.21.15.119:0/742024553 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3819f1d0 con 0x7f8f38103840 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f35e00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e9f0 tx=0x7f8f2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f2ea00640 1 -- 172.21.15.119:0/742024553 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2800cd80 con 0x7f8f38103840 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f2ea00640 1 -- 172.21.15.119:0/742024553 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f28010910 con 0x7f8f38103840 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f2ea00640 1 -- 172.21.15.119:0/742024553 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28018990 con 0x7f8f38103840 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3819f4c0 con 0x7f8f38103840 2024-03-28T18:45:31.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.532+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f381a6d50 con 0x7f8f38103840 2024-03-28T18:45:31.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.535+0000 7f8f2ea00640 1 -- 172.21.15.119:0/742024553 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f8f28010ac0 con 0x7f8f38103840 2024-03-28T18:45:31.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.535+0000 7f8f3fb66740 1 Processor -- start 2024-03-28T18:45:31.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 -- start start 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf6818e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf6818f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 0x55cf6819a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf680f3250 con 0x55cf6818fc00 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6809ba30 con 0x55cf68198290 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf680ef2f0 con 0x55cf6818ef20 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf6818f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf6818f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40450/0 (socket says 172.21.15.119:40450) 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f35e00640 1 -- 172.21.15.119:0/4001176311 learned_addr learned my addr 172.21.15.119:0/4001176311 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.536+0000 7f8f35400640 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 0x55cf6819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 -- 172.21.15.119:0/4001176311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 msgr2=0x55cf6818f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf6818f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 -- 172.21.15.119:0/4001176311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 msgr2=0x55cf6818e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf6818e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 -- 172.21.15.119:0/4001176311 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf68086690 con 0x55cf68198290 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f2ea00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 0x7f8f1c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f2ea00640 1 -- 172.21.15.119:0/742024553 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8f280cf3f0 con 0x7f8f38103840 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f35400640 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 0x55cf6819a680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009230 tx=0x7f8f24009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147ad6eac77cb797 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f36800640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 0x7f8f1c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f23400640 1 -- 172.21.15.119:0/4001176311 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2402c870 con 0x55cf68198290 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f23400640 1 -- 172.21.15.119:0/4001176311 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f2402ca10 con 0x55cf68198290 2024-03-28T18:45:31.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.537+0000 7f8f23400640 1 -- 172.21.15.119:0/4001176311 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24004ed0 con 0x55cf68198290 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 -- 172.21.15.119:0/4001176311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 msgr2=0x55cf6819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 0x55cf6819a680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009230 tx=0x7f8f24009670 comp rx=0 tx=0).stop 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f36800640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 0x7f8f1c0a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f30005fd0 tx=0x7f8f30009b10 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 -- 172.21.15.119:0/4001176311 shutdown_connections 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf68198290 0x55cf6819a680 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf6818f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/4001176311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf6818e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 -- 172.21.15.119:0/4001176311 >> 172.21.15.119:0/4001176311 conn(0x55cf68156200 msgr2=0x55cf68189970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.538+0000 7f8f3fb66740 1 -- 172.21.15.119:0/4001176311 shutdown_connections 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.539+0000 7f8f3fb66740 1 -- 172.21.15.119:0/4001176311 wait complete. 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.539+0000 7f8f3fb66740 1 -- start start 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.539+0000 7f8f3fb66740 1 Processor -- start 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf68194d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 0x55cf68195240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6819b0a0 con 0x55cf6818fc00 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf680aee80 con 0x55cf6818ef20 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf6819b220 con 0x55cf68198290 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f3fb66740 10 client.0 _ll_register_callbacks cb 0x55cf680c1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:31.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44202/0 (socket says 172.21.15.119:44202) 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 -- 172.21.15.119:0/736699023 learned_addr learned my addr 172.21.15.119:0/736699023 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf68194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f36800640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 0x55cf68195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 -- 172.21.15.119:0/736699023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 msgr2=0x55cf68195240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 0x55cf68195240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 -- 172.21.15.119:0/736699023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 msgr2=0x55cf68194d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf68194d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f35e00640 1 -- 172.21.15.119:0/736699023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf67ed17d0 con 0x55cf6818ef20 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.540+0000 7f8f36800640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 0x55cf68195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f35e00640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f280981c0 tx=0x7f8f280d2a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c39ee466dada48b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2809b330 con 0x55cf6818ef20 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8f280ccbe0 con 0x55cf6818ef20 2024-03-28T18:45:31.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f2402ca10 con 0x55cf6818ef20 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f2809b150 con 0x55cf6818ef20 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 10 client.25805 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cf67e073f0 con 0x55cf6818ef20 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f2402ca10 con 0x55cf6818ef20 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8f28018bc0 con 0x55cf6818ef20 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 10 client.25805 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.541+0000 7f8f3fb66740 20 client.25805 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf681a8140 0x55cf681aa590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f3fb66740 4 client.25805 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf68100790 con 0x55cf681a8140 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f35400640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf681a8140 0x55cf681aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f35400640 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf681a8140 0x55cf681aa590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed0c4b59d031468 server_cookie=f2455d7106e1a04f in_seq=0 out_seq=0 2024-03-28T18:45:31.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.542+0000 7f8f17e00640 10 client.25805 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f17e00640 1 -- 172.21.15.119:0/736699023 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cf68100790 con 0x55cf681a8140 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f17e00640 10 client.25805 handle_command_reply: tid=0 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 client.25805 shutdown 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 msgr2=0x55cf68197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f280981c0 tx=0x7f8f280d2a90 comp rx=0 tx=0).stop 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 shutdown_connections 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf681a8140 0x55cf681aa590 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf68198290 0x55cf68195240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf6818fc00 0x55cf68194d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f17e00640 0 client.25805 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/736699023 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818ef20 0x55cf68197970 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 >> 172.21.15.119:0/736699023 conn(0x55cf68156200 msgr2=0x55cf6818b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 shutdown_connections 2024-03-28T18:45:31.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.543+0000 7f8f3fb66740 1 -- 172.21.15.119:0/736699023 wait complete. 2024-03-28T18:45:31.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.544+0000 7f8f3fb66740 20 client.25805 trim_cache size 0 max 16384 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 Processor -- start 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 -- start start 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf68197970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf68199680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381af570 con 0x55cf68198290 2024-03-28T18:45:31.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381b0ef0 con 0x55cf6818fc00 2024-03-28T18:45:31.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.561+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381afbd0 con 0x55cf6818ef20 2024-03-28T18:45:31.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf68197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40466/0 (socket says 172.21.15.119:40466) 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35e00640 1 -- 172.21.15.119:0/1887801068 learned_addr learned my addr 172.21.15.119:0/1887801068 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f36800640 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf68199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 -- 172.21.15.119:0/1887801068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 msgr2=0x55cf68197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 -- 172.21.15.119:0/1887801068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 msgr2=0x55cf68199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf68199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 -- 172.21.15.119:0/1887801068 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf67dc70f0 con 0x55cf6818fc00 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35e00640 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f35400640 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf68197970 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f24036040 tx=0x7f8f24004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2cd5e182699ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f22a00640 1 -- 172.21.15.119:0/1887801068 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24004b40 con 0x55cf6818fc00 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f22a00640 1 -- 172.21.15.119:0/1887801068 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f24004ce0 con 0x55cf6818fc00 2024-03-28T18:45:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.562+0000 7f8f22a00640 1 -- 172.21.15.119:0/1887801068 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24005290 con 0x55cf6818fc00 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 -- 172.21.15.119:0/1887801068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 msgr2=0x55cf68197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf68197970 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f24036040 tx=0x7f8f24004a30 comp rx=0 tx=0).stop 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 -- 172.21.15.119:0/1887801068 shutdown_connections 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf68199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf68197970 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/1887801068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 -- 172.21.15.119:0/1887801068 >> 172.21.15.119:0/1887801068 conn(0x55cf68156200 msgr2=0x55cf681894a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 -- 172.21.15.119:0/1887801068 shutdown_connections 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.563+0000 7f8f3fb66740 1 -- 172.21.15.119:0/1887801068 wait complete. 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 -- start start 2024-03-28T18:45:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 Processor -- start 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf681921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf681fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381b0890 con 0x55cf68198290 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381b0230 con 0x55cf6818fc00 2024-03-28T18:45:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.564+0000 7f8f3fb66740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f381ad730 con 0x55cf6818ef20 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f3fb66740 10 client.0 _ll_register_callbacks cb 0x7f8f28009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40478/0 (socket says 172.21.15.119:40478) 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 -- 172.21.15.119:0/3061924996 learned_addr learned my addr 172.21.15.119:0/3061924996 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f36800640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf681fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35400640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf681921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 -- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 msgr2=0x55cf681921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf681921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 -- 172.21.15.119:0/3061924996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 msgr2=0x55cf681fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf681fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 -- 172.21.15.119:0/3061924996 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf681fd110 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35400640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf681921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f35e00640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8f28098170 tx=0x7f8f280cc840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=227aea58dd73adbf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.565+0000 7f8f17e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280d2b70 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f24004ce0 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8f280ccdf0 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f2809b7f0 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 10 client.25834 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cf68204490 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f24004ce0 con 0x55cf6818ef20 2024-03-28T18:45:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8f28097540 con 0x55cf6818ef20 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 10 client.25834 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 10 client.25834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 20 client.25834 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf68204970 0x55cf68206d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 4 client.25834 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.566+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cf680faae0 con 0x55cf68204970 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f35400640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf68204970 0x55cf68206d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f17e00640 10 client.25834 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f35400640 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf68204970 0x55cf68206d60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a3e77bc8d567bbf server_cookie=bbca9f32780ba9ea in_seq=0 out_seq=0 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f17e00640 1 -- 172.21.15.119:0/3061924996 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cf680faae0 con 0x55cf68204970 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f17e00640 10 client.25834 handle_command_reply: tid=0 2024-03-28T18:45:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.567+0000 7f8f3fb66740 1 client.25834 shutdown 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 msgr2=0x55cf68191ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8f28098170 tx=0x7f8f280cc840 comp rx=0 tx=0).stop 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 shutdown_connections 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cf68204970 0x55cf68206d60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf68198290 0x55cf681fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cf6818fc00 0x55cf681921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f17e00640 0 client.25834 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 --2- 172.21.15.119:0/3061924996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf6818ef20 0x55cf68191ca0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 >> 172.21.15.119:0/3061924996 conn(0x55cf68156200 msgr2=0x55cf6818acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 shutdown_connections 2024-03-28T18:45:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 1 -- 172.21.15.119:0/3061924996 wait complete. 2024-03-28T18:45:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.568+0000 7f8f3fb66740 20 client.25834 trim_cache size 0 max 16384 2024-03-28T18:45:31.575 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:45:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:45:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:45:31.575 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:45:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 msgr2=0x7f8f1c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 0x7f8f1c0a4fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f30005fd0 tx=0x7f8f30009b10 comp rx=0 tx=0).stop 2024-03-28T18:45:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 msgr2=0x7f8f3819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e9f0 tx=0x7f8f2800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 shutdown_connections 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8f1c0a2b10 0x7f8f1c0a4fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f38109850 0x7f8f3819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8f38104200 0x7f8f3819a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 --2- 172.21.15.119:0/742024553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f38103840 0x7f8f3819a1e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 >> 172.21.15.119:0/742024553 conn(0x7f8f380ff350 msgr2=0x7f8f380ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 shutdown_connections 2024-03-28T18:45:31.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.569+0000 7f8f3ce00640 1 -- 172.21.15.119:0/742024553 wait complete. 2024-03-28T18:45:31.588 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:45:31.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:45:31.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 -- 172.21.15.119:0/553824770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600010bca0 msgr2=0x7f600010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 --2- 172.21.15.119:0/553824770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600010bca0 0x7f600010e0d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8009ca0 tx=0x7f5ff802c180 comp rx=0 tx=0).stop 2024-03-28T18:45:31.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 -- 172.21.15.119:0/553824770 shutdown_connections 2024-03-28T18:45:31.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 --2- 172.21.15.119:0/553824770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600010bca0 0x7f600010e0d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 --2- 172.21.15.119:0/553824770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600010b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 --2- 172.21.15.119:0/553824770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60001028d0 0x7f6000102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 -- 172.21.15.119:0/553824770 >> 172.21.15.119:0/553824770 conn(0x7f6000076920 msgr2=0x7f6000076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:31.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 -- 172.21.15.119:0/553824770 shutdown_connections 2024-03-28T18:45:31.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.784+0000 7f6007a00640 1 -- 172.21.15.119:0/553824770 wait complete. 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.785+0000 7f6007a00640 1 Processor -- start 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.785+0000 7f6007a00640 1 -- start start 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600010bca0 0x7f600019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000111a30 con 0x7f600010bca0 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60001118b0 con 0x7f60001031f0 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6007a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000111bb0 con 0x7f60001028d0 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6005000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6005000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40496/0 (socket says 172.21.15.119:40496) 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6005000640 1 -- 172.21.15.119:0/3239745826 learned_addr learned my addr 172.21.15.119:0/3239745826 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6005a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600010bca0 0x7f600019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 -- 172.21.15.119:0/3239745826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 msgr2=0x7f600019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 -- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600010bca0 msgr2=0x7f600019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600010bca0 0x7f600019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f5ff7e00640 1 -- 172.21.15.119:0/3239745826 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600019f120 con 0x7f60001031f0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.786+0000 7f6005000640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f5ff7e00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600019a670 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800e9f0 tx=0x7f5fe800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe800ccc0 con 0x7f60001031f0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe8005020 con 0x7f60001031f0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600019f410 con 0x7f60001031f0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80049e0 con 0x7f60001031f0 2024-03-28T18:45:31.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.787+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60001a6ca0 con 0x7f60001031f0 2024-03-28T18:45:31.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.788+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600006a4f0 con 0x7f60001031f0 2024-03-28T18:45:31.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.790+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f5fe8005560 con 0x7f60001031f0 2024-03-28T18:45:31.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.791+0000 7f5ffe000640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 0x7f5fe40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:31.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.791+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5fe80cf000 con 0x7f60001031f0 2024-03-28T18:45:31.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.792+0000 7f6005000640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 0x7f5fe40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:31.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.792+0000 7f6005000640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 0x7f5fe40a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0005fd0 tx=0x7f5ff000b480 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:31.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:31.798+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5fe80051d0 con 0x7f60001031f0 2024-03-28T18:45:32.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.038+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60000688e0 con 0x7f60001031f0 2024-03-28T18:45:32.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.040+0000 7f5ffe000640 1 -- 172.21.15.119:0/3239745826 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f60000688e0 con 0x7f60001031f0 2024-03-28T18:45:32.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:45:32.041 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:45:32.042 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 msgr2=0x7f5fe40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 0x7f5fe40a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0005fd0 tx=0x7f5ff000b480 comp rx=0 tx=0).stop 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 msgr2=0x7f600019a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600019a670 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800e9f0 tx=0x7f5fe800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 shutdown_connections 2024-03-28T18:45:32.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5fe40a2a90 0x7f5fe40a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600010bca0 0x7f600019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f60001031f0 0x7f600019a670 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 --2- 172.21.15.119:0/3239745826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60001028d0 0x7f600019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 >> 172.21.15.119:0/3239745826 conn(0x7f6000076920 msgr2=0x7f60000ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 shutdown_connections 2024-03-28T18:45:32.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.043+0000 7f6007a00640 1 -- 172.21.15.119:0/3239745826 wait complete. 2024-03-28T18:45:32.064 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:45:32.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 -- 172.21.15.119:0/3989506849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101610 msgr2=0x7fc4d8101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 --2- 172.21.15.119:0/3989506849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101610 0x7fc4d8101a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009930 tx=0x7fc4c002bc30 comp rx=0 tx=0).stop 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 -- 172.21.15.119:0/3989506849 shutdown_connections 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 --2- 172.21.15.119:0/3989506849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d8101fd0 0x7fc4d810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 --2- 172.21.15.119:0/3989506849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101610 0x7fc4d8101a90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 --2- 172.21.15.119:0/3989506849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d81076b0 0x7fc4d8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.256+0000 7fc4de600640 1 -- 172.21.15.119:0/3989506849 >> 172.21.15.119:0/3989506849 conn(0x7fc4d80fd100 msgr2=0x7fc4d80ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.257+0000 7fc4de600640 1 -- 172.21.15.119:0/3989506849 shutdown_connections 2024-03-28T18:45:32.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.257+0000 7fc4de600640 1 -- 172.21.15.119:0/3989506849 wait complete. 2024-03-28T18:45:32.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 Processor -- start 2024-03-28T18:45:32.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 -- start start 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101fd0 0x7fc4d819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 0x7fc4d819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8111a40 con 0x7fc4d81076b0 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81118c0 con 0x7fc4d8101610 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.258+0000 7fc4de600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8111bc0 con 0x7fc4d8101fd0 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4cfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101fd0 0x7fc4d819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44248/0 (socket says 172.21.15.119:44248) 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 -- 172.21.15.119:0/4280137200 learned_addr learned my addr 172.21.15.119:0/4280137200 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 0x7fc4d819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 -- 172.21.15.119:0/4280137200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101fd0 msgr2=0x7fc4d819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101fd0 0x7fc4d819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 -- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 msgr2=0x7fc4d819e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 0x7fc4d819e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 -- 172.21.15.119:0/4280137200 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d819f100 con 0x7fc4d8101610 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 0x7fc4d819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d7400640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e9c0 tx=0x7fc4c800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d4c00640 1 -- 172.21.15.119:0/4280137200 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c800cd80 con 0x7fc4d8101610 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4d4c00640 1 -- 172.21.15.119:0/4280137200 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c80035f0 con 0x7fc4d8101610 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.259+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d819f3f0 con 0x7fc4d8101610 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.260+0000 7fc4d4c00640 1 -- 172.21.15.119:0/4280137200 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8010690 con 0x7fc4d8101610 2024-03-28T18:45:32.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.260+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d81a6c80 con 0x7fc4d8101610 2024-03-28T18:45:32.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.262+0000 7fc4d4c00640 1 -- 172.21.15.119:0/4280137200 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7fc4c80037a0 con 0x7fc4d8101610 2024-03-28T18:45:32.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4d4c00640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 0x7fc4bc0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4e1216740 1 Processor -- start 2024-03-28T18:45:32.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4d4c00640 1 -- 172.21.15.119:0/4280137200 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc4c80cf630 con 0x7fc4d8101610 2024-03-28T18:45:32.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4cfe00640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 0x7fc4bc0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4e1216740 1 -- start start 2024-03-28T18:45:32.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.264+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 0x5595500d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4e1216740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955002d2c0 con 0x5595500d2c00 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4e1216740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55954ffd5c90 con 0x5595500ca080 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4e1216740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559550029360 con 0x5595500c96d0 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4cfe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 0x5595500d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40516/0 (socket says 172.21.15.119:40516) 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 -- 172.21.15.119:0/3638821357 learned_addr learned my addr 172.21.15.119:0/3638821357 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4cfe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 0x5595500d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44264/0 (socket says 172.21.15.119:44264) 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4cfe00640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 0x7fc4bc0a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009900 tx=0x7fc4c002df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 -- 172.21.15.119:0/3638821357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 msgr2=0x5595500d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 0x5595500d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 -- 172.21.15.119:0/3638821357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 msgr2=0x5595500d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500d5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 -- 172.21.15.119:0/3638821357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55954fffa080 con 0x5595500c96d0 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.265+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8097de0 tx=0x7fc4c809c080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=244c1336ebd319c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.266+0000 7fc4ccc00640 1 -- 172.21.15.119:0/3638821357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c80a4120 con 0x5595500c96d0 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.266+0000 7fc4ccc00640 1 -- 172.21.15.119:0/3638821357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c8003b30 con 0x5595500c96d0 2024-03-28T18:45:32.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.266+0000 7fc4ccc00640 1 -- 172.21.15.119:0/3638821357 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8086320 con 0x5595500c96d0 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- 172.21.15.119:0/3638821357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 msgr2=0x5595500c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8097de0 tx=0x7fc4c809c080 comp rx=0 tx=0).stop 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- 172.21.15.119:0/3638821357 shutdown_connections 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500d5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500ca080 0x5595500d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3638821357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500c96d0 0x5595500c9ab0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- 172.21.15.119:0/3638821357 >> 172.21.15.119:0/3638821357 conn(0x5595500c3da0 msgr2=0x5595500c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- 172.21.15.119:0/3638821357 shutdown_connections 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- 172.21.15.119:0/3638821357 wait complete. 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.267+0000 7fc4e1216740 1 -- start start 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 Processor -- start 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595500d5b70 con 0x5595500ca080 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55954ffe90d0 con 0x5595500c96d0 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595500d5cf0 con 0x5595500d2c00 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.268+0000 7fc4e1216740 10 client.0 _ll_register_callbacks cb 0x55954ffede80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44270/0 (socket says 172.21.15.119:44270) 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7400640 1 -- 172.21.15.119:0/2029052417 learned_addr learned my addr 172.21.15.119:0/2029052417 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4cfe00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 -- 172.21.15.119:0/2029052417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 msgr2=0x5595500d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 -- 172.21.15.119:0/2029052417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 msgr2=0x5595500cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 -- 172.21.15.119:0/2029052417 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55954fd413f0 con 0x5595500d2c00 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7400640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4cfe00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500cfdd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0005fd0 tx=0x7fc4d0011e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b206972dd5470119 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4c5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0010c00 con 0x5595500d2c00 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c8003b30 con 0x5595500d2c00 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.269+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc4d0010da0 con 0x5595500d2c00 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d0004b20 con 0x5595500d2c00 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 10 client.25852 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55954fe0b7d0 con 0x5595500d2c00 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c8003b30 con 0x5595500d2c00 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc4d001a410 con 0x5595500d2c00 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 10 client.25852 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 10 client.25852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 20 client.25852 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5595500e2aa0 0x5595500e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 4 client.25852 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.270+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55955003a800 con 0x5595500e2aa0 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.271+0000 7fc4d7400640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5595500e2aa0 0x5595500e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.271+0000 7fc4d7400640 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5595500e2aa0 0x5595500e4e90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d929e23934e2405e server_cookie=2409e3537a31e48 in_seq=0 out_seq=0 2024-03-28T18:45:32.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.271+0000 7fc4c5600640 10 client.25852 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:45:32.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4c5600640 1 -- 172.21.15.119:0/2029052417 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55955003a800 con 0x5595500e2aa0 2024-03-28T18:45:32.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4c5600640 10 client.25852 handle_command_reply: tid=0 2024-03-28T18:45:32.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 client.25852 shutdown 2024-03-28T18:45:32.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 msgr2=0x5595500cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500cfdd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0005fd0 tx=0x7fc4d0011e60 comp rx=0 tx=0).stop 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 shutdown_connections 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5595500e2aa0 0x5595500e4e90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500cfdd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 --2- 172.21.15.119:0/2029052417 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4c5600640 0 client.25852 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 >> 172.21.15.119:0/2029052417 conn(0x5595500c3da0 msgr2=0x5595500c5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 shutdown_connections 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 1 -- 172.21.15.119:0/2029052417 wait complete. 2024-03-28T18:45:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.272+0000 7fc4e1216740 20 client.25852 trim_cache size 0 max 16384 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.285+0000 7fc4e1216740 1 Processor -- start 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.285+0000 7fc4e1216740 1 -- start start 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.285+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500ca080 0x5595500d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4e1216740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81af650 con 0x5595500d2c00 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4e1216740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81b0fd0 con 0x5595500c96d0 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4e1216740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81afcb0 con 0x5595500ca080 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4cfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500ca080 0x5595500d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59576/0 (socket says 172.21.15.119:59576) 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 -- 172.21.15.119:0/1689393757 learned_addr learned my addr 172.21.15.119:0/1689393757 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 -- 172.21.15.119:0/1689393757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500ca080 msgr2=0x5595500d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500ca080 0x5595500d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 -- 172.21.15.119:0/1689393757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 msgr2=0x5595500d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7e00640 1 -- 172.21.15.119:0/1689393757 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595500d0140 con 0x5595500d2c00 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.286+0000 7fc4d7400640 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.287+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009810 tx=0x7fc4d001bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226fc0e1a401cc4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.287+0000 7fc4c7e00640 1 -- 172.21.15.119:0/1689393757 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0004cc0 con 0x5595500d2c00 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.287+0000 7fc4c7e00640 1 -- 172.21.15.119:0/1689393757 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4d001a410 con 0x5595500d2c00 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.287+0000 7fc4c7e00640 1 -- 172.21.15.119:0/1689393757 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0010e90 con 0x5595500d2c00 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- 172.21.15.119:0/1689393757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 msgr2=0x5595500c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009810 tx=0x7fc4d001bfa0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- 172.21.15.119:0/1689393757 shutdown_connections 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500d2c00 0x5595500c2d30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500ca080 0x5595500d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 --2- 172.21.15.119:0/1689393757 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- 172.21.15.119:0/1689393757 >> 172.21.15.119:0/1689393757 conn(0x5595500c3da0 msgr2=0x5595500c2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- 172.21.15.119:0/1689393757 shutdown_connections 2024-03-28T18:45:32.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- 172.21.15.119:0/1689393757 wait complete. 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 -- start start 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.288+0000 7fc4e1216740 1 Processor -- start 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81b0970 con 0x5595500ca080 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81b0310 con 0x5595500c96d0 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81ad810 con 0x5595500d2c00 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4e1216740 10 client.0 _ll_register_callbacks cb 0x7fc4d0009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4cfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40548/0 (socket says 172.21.15.119:40548) 2024-03-28T18:45:32.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 -- 172.21.15.119:0/3877167288 learned_addr learned my addr 172.21.15.119:0/3877167288 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44290/0 (socket says 172.21.15.119:44290) 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 -- 172.21.15.119:0/3877167288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 msgr2=0x5595500da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 -- 172.21.15.119:0/3877167288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 msgr2=0x5595500d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500d5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7e00640 1 -- 172.21.15.119:0/3877167288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595500d6980 con 0x5595500d2c00 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.289+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4d7e00640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00114a0 tx=0x7fc4d0010c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2468dc3daf03caec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4c5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0004cc0 con 0x5595500d2c00 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4d001a410 con 0x5595500d2c00 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc4d0022470 con 0x5595500d2c00 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d0022670 con 0x5595500d2c00 2024-03-28T18:45:32.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4e1216740 10 client.25858 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55955013e000 con 0x5595500d2c00 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.290+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4d001a410 con 0x5595500d2c00 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc4d002b020 con 0x5595500d2c00 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 10 client.25858 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 10 client.25858 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 20 client.25858 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55955013e560 0x559550140980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 4 client.25858 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559550034b50 con 0x55955013e560 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55955013e560 0x559550140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4d7400640 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55955013e560 0x559550140980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a57e98c19eed1fa server_cookie=bd043f68cfafde6d in_seq=0 out_seq=0 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4c5600640 10 client.25858 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4c5600640 1 -- 172.21.15.119:0/3877167288 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559550034b50 con 0x55955013e560 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4c5600640 10 client.25858 handle_command_reply: tid=0 2024-03-28T18:45:32.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.291+0000 7fc4e1216740 1 client.25858 shutdown 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 msgr2=0x5595500d6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00114a0 tx=0x7fc4d0010c00 comp rx=0 tx=0).stop 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 shutdown_connections 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55955013e560 0x559550140980 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5595500d2c00 0x5595500d6440 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5595500ca080 0x5595500d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 --2- 172.21.15.119:0/3877167288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5595500c96d0 0x5595500da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4c5600640 0 client.25858 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 >> 172.21.15.119:0/3877167288 conn(0x5595500c3da0 msgr2=0x5595500c2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 shutdown_connections 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 1 -- 172.21.15.119:0/3877167288 wait complete. 2024-03-28T18:45:32.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.292+0000 7fc4e1216740 20 client.25858 trim_cache size 0 max 16384 2024-03-28T18:45:32.298 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:45:32.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 msgr2=0x7fc4bc0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 0x7fc4bc0a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009900 tx=0x7fc4c002df90 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 msgr2=0x7fc4d819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800e9c0 tx=0x7fc4c800ee80 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 shutdown_connections 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4bc0a2a90 0x7fc4bc0a4f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4d81076b0 0x7fc4d819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d8101fd0 0x7fc4d819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 --2- 172.21.15.119:0/4280137200 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc4d8101610 0x7fc4d819a160 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 >> 172.21.15.119:0/4280137200 conn(0x7fc4d80fd100 msgr2=0x7fc4d80fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 shutdown_connections 2024-03-28T18:45:32.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.293+0000 7fc4de600640 1 -- 172.21.15.119:0/4280137200 wait complete. 2024-03-28T18:45:32.309 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:45:32.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:45:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 -- 172.21.15.119:0/2479452575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101630 msgr2=0x7f95f4101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 --2- 172.21.15.119:0/2479452575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101630 0x7f95f4101ab0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f95e0009930 tx=0x7f95e002bc30 comp rx=0 tx=0).stop 2024-03-28T18:45:32.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 -- 172.21.15.119:0/2479452575 shutdown_connections 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 --2- 172.21.15.119:0/2479452575 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101ff0 0x7f95f410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 --2- 172.21.15.119:0/2479452575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101630 0x7f95f4101ab0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 --2- 172.21.15.119:0/2479452575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 0x7f95f4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 -- 172.21.15.119:0/2479452575 >> 172.21.15.119:0/2479452575 conn(0x7f95f4078aa0 msgr2=0x7f95f4078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 -- 172.21.15.119:0/2479452575 shutdown_connections 2024-03-28T18:45:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.510+0000 7f95f9600640 1 -- 172.21.15.119:0/2479452575 wait complete. 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.511+0000 7f95f9600640 1 Processor -- start 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.511+0000 7f95f9600640 1 -- start start 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.511+0000 7f95f9600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.511+0000 7f95f9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101ff0 0x7f95f419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f9600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 0x7f95f419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f9600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4111aa0 con 0x7f95f4107640 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f9600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4111920 con 0x7f95f4101630 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f9600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4111c20 con 0x7f95f4101ff0 2024-03-28T18:45:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101ff0 0x7f95f419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44304/0 (socket says 172.21.15.119:44304) 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 0x7f95f419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 -- 172.21.15.119:0/1145390084 learned_addr learned my addr 172.21.15.119:0/1145390084 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 -- 172.21.15.119:0/1145390084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101ff0 msgr2=0x7f95f419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101ff0 0x7f95f419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 -- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 msgr2=0x7f95f419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 0x7f95f419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 -- 172.21.15.119:0/1145390084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f419f150 con 0x7f95f4101630 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.512+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95e400e9f0 tx=0x7f95e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.513+0000 7f95eb400640 1 -- 172.21.15.119:0/1145390084 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400cd80 con 0x7f95f4101630 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.513+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f419f440 con 0x7f95f4101630 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.513+0000 7f95eb400640 1 -- 172.21.15.119:0/1145390084 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e4010910 con 0x7f95f4101630 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.513+0000 7f95eb400640 1 -- 172.21.15.119:0/1145390084 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e40188f0 con 0x7f95f4101630 2024-03-28T18:45:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.513+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f41a6cd0 con 0x7f95f4101630 2024-03-28T18:45:32.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 Processor -- start 2024-03-28T18:45:32.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 -- start start 2024-03-28T18:45:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95eb400640 1 -- 172.21.15.119:0/1145390084 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f95e4010ac0 con 0x7f95f4101630 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf7302c0 con 0x558cdf7cc6d0 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf6d8c90 con 0x558cdf7d5c00 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.517+0000 7f95fc25d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf72c360 con 0x558cdf7cd080 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44314/0 (socket says 172.21.15.119:44314) 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59626/0 (socket says 172.21.15.119:59626) 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 -- 172.21.15.119:0/1314559857 learned_addr learned my addr 172.21.15.119:0/1314559857 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 -- 172.21.15.119:0/1314559857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 msgr2=0x558cdf7d5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 -- 172.21.15.119:0/1314559857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 msgr2=0x558cdf7ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 -- 172.21.15.119:0/1314559857 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cdf6fd080 con 0x558cdf7d5c00 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.518+0000 7f95f2000640 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95e0000c00 tx=0x7f95e0009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37546b86316da46f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95df400640 1 -- 172.21.15.119:0/1314559857 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0045070 con 0x558cdf7d5c00 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95df400640 1 -- 172.21.15.119:0/1314559857 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e002c650 con 0x558cdf7d5c00 2024-03-28T18:45:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95df400640 1 -- 172.21.15.119:0/1314559857 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0004aa0 con 0x558cdf7d5c00 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95eb400640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 0x7f95d80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95eb400640 1 -- 172.21.15.119:0/1145390084 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95e40cf4f0 con 0x7f95f4101630 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.519+0000 7f95f3400640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 0x7f95d80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- 172.21.15.119:0/1314559857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 msgr2=0x558cdf7d8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f95e0000c00 tx=0x7f95e0009560 comp rx=0 tx=0).stop 2024-03-28T18:45:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- 172.21.15.119:0/1314559857 shutdown_connections 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7d5c00 0x558cdf7d8050 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1314559857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- 172.21.15.119:0/1314559857 >> 172.21.15.119:0/1314559857 conn(0x558cdf7c6da0 msgr2=0x558cdf7c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- 172.21.15.119:0/1314559857 shutdown_connections 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95f3400640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 0x7f95d80a4f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f95ec005fd0 tx=0x7f95ec009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- 172.21.15.119:0/1314559857 wait complete. 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.520+0000 7f95fc25d740 1 -- start start 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 Processor -- start 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 0x558cdf7d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf7d8b10 con 0x558cdf7d5c00 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf6ec0d0 con 0x558cdf7cc6d0 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cdf7d8c90 con 0x558cdf7cd080 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95fc25d740 10 client.0 _ll_register_callbacks cb 0x558cdf6f0e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95f2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95f2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40570/0 (socket says 172.21.15.119:40570) 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95f2000640 1 -- 172.21.15.119:0/1058052116 learned_addr learned my addr 172.21.15.119:0/1058052116 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.521+0000 7f95f3400640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 0x558cdf7d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 -- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7cc6d0 msgr2=0x558cdf7d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 -- 172.21.15.119:0/1058052116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 msgr2=0x558cdf7d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 0x558cdf7d2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 -- 172.21.15.119:0/1058052116 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558cdf4443f0 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f3400640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 0x558cdf7d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95f2000640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f95e0005900 tx=0x7f95e0004dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80cfd2e495003339 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0045040 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95e002c650 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95e0004ee0 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.522+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95e0059050 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 10 client.25870 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558cdf50e7d0 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e002c650 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f95e004ac90 con 0x558cdf7cd080 2024-03-28T18:45:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 10 client.25870 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 10 client.25870 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 20 client.25870 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7e5820 0x558cdf7e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 4 client.25870 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558cdf73d800 con 0x558cdf7e5820 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.523+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7e5820 0x558cdf7e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.524+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7e5820 0x558cdf7e7c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=714c4c2f4b0c10b1 server_cookie=2eb380addabcef80 in_seq=0 out_seq=0 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.524+0000 7f95d3e00640 10 client.25870 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.524+0000 7f95d3e00640 1 -- 172.21.15.119:0/1058052116 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558cdf73d800 con 0x558cdf7e5820 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.524+0000 7f95d3e00640 10 client.25870 handle_command_reply: tid=0 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.524+0000 7f95fc25d740 1 client.25870 shutdown 2024-03-28T18:45:32.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 msgr2=0x558cdf7d2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f95e0005900 tx=0x7f95e0004dc0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 shutdown_connections 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7e5820 0x558cdf7e7c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7d5c00 0x558cdf7d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7d2810 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1058052116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95d3e00640 0 client.25870 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 >> 172.21.15.119:0/1058052116 conn(0x558cdf7c6da0 msgr2=0x558cdf7c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 shutdown_connections 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 1 -- 172.21.15.119:0/1058052116 wait complete. 2024-03-28T18:45:32.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.525+0000 7f95fc25d740 20 client.25870 trim_cache size 0 max 16384 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.538+0000 7f95fc25d740 1 Processor -- start 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.538+0000 7f95fc25d740 1 -- start start 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41af540 con 0x558cdf7cc6d0 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41b0ec0 con 0x558cdf7c6da0 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95fc25d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41afba0 con 0x558cdf7cd080 2024-03-28T18:45:32.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44318/0 (socket says 172.21.15.119:44318) 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 -- 172.21.15.119:0/1498209432 learned_addr learned my addr 172.21.15.119:0/1498209432 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2000640 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 -- 172.21.15.119:0/1498209432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 msgr2=0x558cdf7cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 -- 172.21.15.119:0/1498209432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 msgr2=0x558cdf7d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.539+0000 7f95f2a00640 1 -- 172.21.15.119:0/1498209432 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cdf7d3350 con 0x558cdf7c6da0 2024-03-28T18:45:32.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.540+0000 7f95f2a00640 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95e409b880 tx=0x7f95e40d1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d678dceed8019692 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.540+0000 7f95dea00640 1 -- 172.21.15.119:0/1498209432 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e409b8f0 con 0x558cdf7c6da0 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.540+0000 7f95dea00640 1 -- 172.21.15.119:0/1498209432 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e409f990 con 0x558cdf7c6da0 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.540+0000 7f95dea00640 1 -- 172.21.15.119:0/1498209432 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e409cb70 con 0x558cdf7c6da0 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- 172.21.15.119:0/1498209432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 msgr2=0x558cdf7d4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f95e409b880 tx=0x7f95e40d1040 comp rx=0 tx=0).stop 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- 172.21.15.119:0/1498209432 shutdown_connections 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf7cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7d5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 --2- 172.21.15.119:0/1498209432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d4f40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- 172.21.15.119:0/1498209432 >> 172.21.15.119:0/1498209432 conn(0x558cdf7d2eb0 msgr2=0x558cdf7c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- 172.21.15.119:0/1498209432 shutdown_connections 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- 172.21.15.119:0/1498209432 wait complete. 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.541+0000 7f95fc25d740 1 -- start start 2024-03-28T18:45:32.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 Processor -- start 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7da300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf838df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41b0860 con 0x558cdf7cc6d0 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41b0200 con 0x558cdf7c6da0 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f41ad700 con 0x558cdf7cd080 2024-03-28T18:45:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.542+0000 7f95fc25d740 10 client.0 _ll_register_callbacks cb 0x7f95ec009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44322/0 (socket says 172.21.15.119:44322) 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 -- 172.21.15.119:0/2155049705 learned_addr learned my addr 172.21.15.119:0/2155049705 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f3400640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf838df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2000640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 -- 172.21.15.119:0/2155049705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 msgr2=0x558cdf838df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf838df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 -- 172.21.15.119:0/2155049705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 msgr2=0x558cdf7da300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7da300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 -- 172.21.15.119:0/2155049705 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558cdf839490 con 0x558cdf7c6da0 2024-03-28T18:45:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95f2a00640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f95e4099ba0 tx=0x7f95e4005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8742533089c033ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e40af060 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95e40a3c30 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.543+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95e409f990 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95e409b8f0 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 10 client.25853 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558cdf8419b0 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95e409f990 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f95e409add0 con 0x558cdf7c6da0 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 10 client.25853 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 10 client.25853 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 20 client.25853 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7d5c00 0x558cdf8440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 4 client.25853 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558cdf737b50 con 0x558cdf7d5c00 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.544+0000 7f95f2000640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7d5c00 0x558cdf8440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95f2000640 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7d5c00 0x558cdf8440d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d9f545be839a14a0 server_cookie=f1004c855142386f in_seq=0 out_seq=0 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95d3e00640 10 client.25853 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95d3e00640 1 -- 172.21.15.119:0/2155049705 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558cdf737b50 con 0x558cdf7d5c00 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95d3e00640 10 client.25853 handle_command_reply: tid=0 2024-03-28T18:45:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 client.25853 shutdown 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 msgr2=0x558cdf7d9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f95e4099ba0 tx=0x7f95e4005c60 comp rx=0 tx=0).stop 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 shutdown_connections 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cdf7cd080 0x558cdf838df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558cdf7d5c00 0x558cdf8440d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558cdf7cc6d0 0x558cdf7da300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 --2- 172.21.15.119:0/2155049705 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558cdf7c6da0 0x558cdf7d9dc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95d3e00640 0 client.25853 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 >> 172.21.15.119:0/2155049705 conn(0x558cdf7d2eb0 msgr2=0x558cdf7c5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 shutdown_connections 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 1 -- 172.21.15.119:0/2155049705 wait complete. 2024-03-28T18:45:32.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.545+0000 7f95fc25d740 20 client.25853 trim_cache size 0 max 16384 2024-03-28T18:45:32.552 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 msgr2=0x7f95d80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 0x7f95d80a4f50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f95ec005fd0 tx=0x7f95ec009500 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 msgr2=0x7f95f419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95e400e9f0 tx=0x7f95e400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 shutdown_connections 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f95d80a2a90 0x7f95d80a4f50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4107640 0x7f95f419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95f4101ff0 0x7f95f419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 --2- 172.21.15.119:0/1145390084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95f4101630 0x7f95f419a160 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 >> 172.21.15.119:0/1145390084 conn(0x7f95f4078aa0 msgr2=0x7f95f410cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 shutdown_connections 2024-03-28T18:45:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.546+0000 7f95f9600640 1 -- 172.21.15.119:0/1145390084 wait complete. 2024-03-28T18:45:32.563 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:45:32.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:45:32.633 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:32 smithi119 ceph-mon[31813]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:45:32.633 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:32 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3239745826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:45:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:32 smithi196 ceph-mon[36300]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:45:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:32 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3239745826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:45:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:32 smithi131 ceph-mon[36065]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-28T18:45:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:32 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3239745826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:45:32.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 -- 172.21.15.119:0/4010130252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c1124e0 msgr2=0x7f4b2c1148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 --2- 172.21.15.119:0/4010130252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1148d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009cd0 tx=0x7f4b2402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 -- 172.21.15.119:0/4010130252 shutdown_connections 2024-03-28T18:45:32.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 --2- 172.21.15.119:0/4010130252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1148d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 --2- 172.21.15.119:0/4010130252 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 --2- 172.21.15.119:0/4010130252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 -- 172.21.15.119:0/4010130252 >> 172.21.15.119:0/4010130252 conn(0x7f4b2c0ff350 msgr2=0x7f4b2c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 -- 172.21.15.119:0/4010130252 shutdown_connections 2024-03-28T18:45:32.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.763+0000 7f4b31400640 1 -- 172.21.15.119:0/4010130252 wait complete. 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.764+0000 7f4b31400640 1 Processor -- start 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.764+0000 7f4b31400640 1 -- start start 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.764+0000 7f4b31400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b31400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b31400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b31400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c118040 con 0x7f4b2c1124e0 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b31400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c117ec0 con 0x7f4b2c0764f0 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b31400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1181c0 con 0x7f4b2c075bd0 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40608/0 (socket says 172.21.15.119:40608) 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b2a800640 1 -- 172.21.15.119:0/3806264356 learned_addr learned my addr 172.21.15.119:0/3806264356 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b2b200640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 -- 172.21.15.119:0/3806264356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 msgr2=0x7f4b2c19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 -- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c1124e0 msgr2=0x7f4b2c1a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 -- 172.21.15.119:0/3806264356 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c1a35b0 con 0x7f4b2c0764f0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.765+0000 7f4b29e00640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c19eb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9f0 tx=0x7f4b1800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.766+0000 7f4b23400640 1 -- 172.21.15.119:0/3806264356 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800ccc0 con 0x7f4b2c0764f0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.766+0000 7f4b23400640 1 -- 172.21.15.119:0/3806264356 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b18010910 con 0x7f4b2c0764f0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.766+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c1a38a0 con 0x7f4b2c0764f0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.766+0000 7f4b23400640 1 -- 172.21.15.119:0/3806264356 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18018960 con 0x7f4b2c0764f0 2024-03-28T18:45:32.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.766+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c1ab130 con 0x7f4b2c0764f0 2024-03-28T18:45:32.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 Processor -- start 2024-03-28T18:45:32.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 -- start start 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff92ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff9e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1fef62c0 con 0x557b1ff93080 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1fe9ec90 con 0x557b1ff926d0 2024-03-28T18:45:32.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b341c0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1fef2360 con 0x557b1ff9bc00 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b23400640 1 -- 172.21.15.119:0/3806264356 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f4b18010ac0 con 0x7f4b2c0764f0 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59656/0 (socket says 172.21.15.119:59656) 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 -- 172.21.15.119:0/4192494589 learned_addr learned my addr 172.21.15.119:0/4192494589 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.769+0000 7f4b2b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff92ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b29e00640 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff9e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 -- 172.21.15.119:0/4192494589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 msgr2=0x557b1ff9e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff9e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 -- 172.21.15.119:0/4192494589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 msgr2=0x557b1ff92ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff92ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 -- 172.21.15.119:0/4192494589 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b1fec3080 con 0x557b1ff93080 2024-03-28T18:45:32.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2b200640 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff92ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.770+0000 7f4b2a800640 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009dc0 tx=0x7f4b1c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf35bc12c4c8e7d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b17400640 1 -- 172.21.15.119:0/4192494589 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c005220 con 0x557b1ff93080 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b17400640 1 -- 172.21.15.119:0/4192494589 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b1c004c80 con 0x557b1ff93080 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b17400640 1 -- 172.21.15.119:0/4192494589 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c019970 con 0x557b1ff93080 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b23400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 0x7f4b100a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b341c0740 1 -- 172.21.15.119:0/4192494589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 msgr2=0x557b1ff9b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009dc0 tx=0x7f4b1c011810 comp rx=0 tx=0).stop 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b23400640 1 -- 172.21.15.119:0/3806264356 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4b180cf490 con 0x7f4b2c0764f0 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.771+0000 7f4b2b200640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 0x7f4b100a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 -- 172.21.15.119:0/4192494589 shutdown_connections 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff9e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff9b630 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4192494589 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff92ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 -- 172.21.15.119:0/4192494589 >> 172.21.15.119:0/4192494589 conn(0x557b1ff8cda0 msgr2=0x557b1ff8d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 -- 172.21.15.119:0/4192494589 shutdown_connections 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 -- 172.21.15.119:0/4192494589 wait complete. 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b2b200640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 0x7f4b100a4fd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009bf0 tx=0x7f4b24005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 -- start start 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.772+0000 7f4b341c0740 1 Processor -- start 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff98910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1ff9eb70 con 0x557b1ff93080 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1feb20d0 con 0x557b1ff926d0 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b1ff9ecf0 con 0x557b1ff9bc00 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b341c0740 10 client.0 _ll_register_callbacks cb 0x557b1feb6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b2b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b29e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff98910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44356/0 (socket says 172.21.15.119:44356) 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b2b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40620/0 (socket says 172.21.15.119:40620) 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.773+0000 7f4b2b200640 1 -- 172.21.15.119:0/833650105 learned_addr learned my addr 172.21.15.119:0/833650105 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 -- 172.21.15.119:0/833650105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 msgr2=0x557b1ff9b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 -- 172.21.15.119:0/833650105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 msgr2=0x557b1ff98910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff98910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 -- 172.21.15.119:0/833650105 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b1fc0a3f0 con 0x557b1ff9bc00 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2a800640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b29e00640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff98910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b2b200640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b24004af0 tx=0x7f4b240095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8df1c701c0e5332 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b0be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24039040 con 0x557b1ff9bc00 2024-03-28T18:45:32.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b1c004c80 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4b24003b70 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b2405a050 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b341c0740 10 client.25888 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.774+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557b1fcd47d0 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b1c004c80 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4b2404c480 con 0x557b1ff9bc00 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 10 client.25888 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 10 client.25888 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 20 client.25888 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b1ffabb30 0x557b1ffadf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 4 client.25888 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b1ff03800 con 0x557b1ffabb30 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.775+0000 7f4b2a800640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b1ffabb30 0x557b1ffadf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.776+0000 7f4b2a800640 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b1ffabb30 0x557b1ffadf20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a8e8672dc00759b server_cookie=ab5fbf518896bf09 in_seq=0 out_seq=0 2024-03-28T18:45:32.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.776+0000 7f4b0be00640 10 client.25888 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.776+0000 7f4b0be00640 1 -- 172.21.15.119:0/833650105 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557b1ff03800 con 0x557b1ffabb30 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.776+0000 7f4b0be00640 10 client.25888 handle_command_reply: tid=0 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.776+0000 7f4b341c0740 1 client.25888 shutdown 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 msgr2=0x557b1ff98f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b24004af0 tx=0x7f4b240095c0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 shutdown_connections 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b1ffabb30 0x557b1ffadf20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ff98f20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ff98910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b0be00640 0 client.25888 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 --2- 172.21.15.119:0/833650105 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ff9b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 >> 172.21.15.119:0/833650105 conn(0x557b1ff8cda0 msgr2=0x557b1ff8eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 shutdown_connections 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 1 -- 172.21.15.119:0/833650105 wait complete. 2024-03-28T18:45:32.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.777+0000 7f4b341c0740 20 client.25888 trim_cache size 0 max 16384 2024-03-28T18:45:32.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 Processor -- start 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 -- start start 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff93080 0x557b1ff9b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff9bc00 0x557b1ff95b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b3970 con 0x557b1ff9bc00 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b52f0 con 0x557b1ff93080 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b341c0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b3fd0 con 0x557b1ff926d0 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40634/0 (socket says 172.21.15.119:40634) 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b29e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff93080 0x557b1ff9b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b2a800640 1 -- 172.21.15.119:0/2041126247 learned_addr learned my addr 172.21.15.119:0/2041126247 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.790+0000 7f4b2b200640 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff9bc00 0x557b1ff95b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 -- 172.21.15.119:0/2041126247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff93080 msgr2=0x557b1ff9b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff93080 0x557b1ff9b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 -- 172.21.15.119:0/2041126247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff9bc00 msgr2=0x557b1ff95b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff9bc00 0x557b1ff95b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 -- 172.21.15.119:0/2041126247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b1ffae580 con 0x557b1ff926d0 2024-03-28T18:45:32.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b2a800640 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c01b040 tx=0x7f4b1c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16222cd9e2d7af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b16a00640 1 -- 172.21.15.119:0/2041126247 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c02f020 con 0x557b1ff926d0 2024-03-28T18:45:32.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b16a00640 1 -- 172.21.15.119:0/2041126247 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b1c02b2c0 con 0x557b1ff926d0 2024-03-28T18:45:32.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.791+0000 7f4b16a00640 1 -- 172.21.15.119:0/2041126247 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c02b520 con 0x557b1ff926d0 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- 172.21.15.119:0/2041126247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 msgr2=0x557b1ff9b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c01b040 tx=0x7f4b1c011810 comp rx=0 tx=0).stop 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- 172.21.15.119:0/2041126247 shutdown_connections 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff9bc00 0x557b1ff95b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff93080 0x557b1ff9b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 --2- 172.21.15.119:0/2041126247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff926d0 0x557b1ff9b040 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- 172.21.15.119:0/2041126247 >> 172.21.15.119:0/2041126247 conn(0x557b1ff8cda0 msgr2=0x557b1ff8beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- 172.21.15.119:0/2041126247 shutdown_connections 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- 172.21.15.119:0/2041126247 wait complete. 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.792+0000 7f4b341c0740 1 -- start start 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 Processor -- start 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ffa13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ffa18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b4c90 con 0x557b1ff93080 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b4630 con 0x557b1ff926d0 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2c1b1b30 con 0x557b1ff9bc00 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b341c0740 10 client.0 _ll_register_callbacks cb 0x7f4b24043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ffa13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b2b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b29e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ffa18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b2b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40642/0 (socket says 172.21.15.119:40642) 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b2b200640 1 -- 172.21.15.119:0/4282003648 learned_addr learned my addr 172.21.15.119:0/4282003648 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.793+0000 7f4b2a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ffa13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44384/0 (socket says 172.21.15.119:44384) 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 -- 172.21.15.119:0/4282003648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 msgr2=0x557b1ffa13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ffa13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 -- 172.21.15.119:0/4282003648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 msgr2=0x557b1ffa18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ffa18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 -- 172.21.15.119:0/4282003648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b20006380 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b2b200640 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4b2402c630 tx=0x7f4b24009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ea303cae611b369 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b0be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24063060 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4b24003b70 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b1c02b2c0 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b24039070 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b341c0740 10 client.25894 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.794+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557b20007650 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b1c02b2c0 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4b24034440 con 0x557b1ff9bc00 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 10 client.25894 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 10 client.25894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 20 client.25894 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:32.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b20007ca0 0x557b2000a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 4 client.25894 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557b1fefdb50 con 0x557b20007ca0 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.795+0000 7f4b2a800640 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b20007ca0 0x557b2000a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b2a800640 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b20007ca0 0x557b2000a0c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9ce2f6f17e23e74 server_cookie=f052db41de29efba in_seq=0 out_seq=0 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b0be00640 10 client.25894 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b0be00640 1 -- 172.21.15.119:0/4282003648 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557b1fefdb50 con 0x557b20007ca0 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b0be00640 10 client.25894 handle_command_reply: tid=0 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 client.25894 shutdown 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 msgr2=0x557b1ffa1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4b2402c630 tx=0x7f4b24009800 comp rx=0 tx=0).stop 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 shutdown_connections 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x557b20007ca0 0x557b2000a0c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557b1ff9bc00 0x557b1ffa1e30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557b1ff93080 0x557b1ffa18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b0be00640 0 client.25894 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:45:32.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 --2- 172.21.15.119:0/4282003648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557b1ff926d0 0x557b1ffa13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 >> 172.21.15.119:0/4282003648 conn(0x557b1ff8cda0 msgr2=0x557b1ff8b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 shutdown_connections 2024-03-28T18:45:32.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 1 -- 172.21.15.119:0/4282003648 wait complete. 2024-03-28T18:45:32.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.796+0000 7f4b341c0740 20 client.25894 trim_cache size 0 max 16384 2024-03-28T18:45:32.802 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 msgr2=0x7f4b100a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 0x7f4b100a4fd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009bf0 tx=0x7f4b24005d80 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 msgr2=0x7f4b2c19eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c19eb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9f0 tx=0x7f4b1800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 shutdown_connections 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4b100a2b10 0x7f4b100a4fd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b2c1124e0 0x7f4b2c1a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4b2c0764f0 0x7f4b2c19eb00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 --2- 172.21.15.119:0/3806264356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b2c075bd0 0x7f4b2c19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 >> 172.21.15.119:0/3806264356 conn(0x7f4b2c0ff350 msgr2=0x7f4b2c113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:32.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 shutdown_connections 2024-03-28T18:45:32.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:32.797+0000 7f4b31400640 1 -- 172.21.15.119:0/3806264356 wait complete. 2024-03-28T18:45:32.813 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:45:32.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.030+0000 7f2897600640 1 -- 172.21.15.119:0/3274917462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28900688a0 msgr2=0x7f2890068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.030+0000 7f2897600640 1 --2- 172.21.15.119:0/3274917462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28900688a0 0x7f2890068c80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2884009900 tx=0x7f288402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 -- 172.21.15.119:0/3274917462 shutdown_connections 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 --2- 172.21.15.119:0/3274917462 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f289010e040 0x7f2890110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 --2- 172.21.15.119:0/3274917462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900691c0 0x7f289010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 --2- 172.21.15.119:0/3274917462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28900688a0 0x7f2890068c80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 -- 172.21.15.119:0/3274917462 >> 172.21.15.119:0/3274917462 conn(0x7f28900776a0 msgr2=0x7f2890077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 -- 172.21.15.119:0/3274917462 shutdown_connections 2024-03-28T18:45:33.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.031+0000 7f2897600640 1 -- 172.21.15.119:0/3274917462 wait complete. 2024-03-28T18:45:33.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.032+0000 7f2897600640 1 Processor -- start 2024-03-28T18:45:33.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.032+0000 7f2897600640 1 -- start start 2024-03-28T18:45:33.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900688a0 0x7f289019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f289010e040 0x7f289019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2890113c60 con 0x7f28900688a0 2024-03-28T18:45:33.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2890113ae0 con 0x7f28900691c0 2024-03-28T18:45:33.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2897600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2890113de0 con 0x7f289010e040 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2894c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900688a0 0x7f289019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44388/0 (socket says 172.21.15.119:44388) 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 -- 172.21.15.119:0/3800690689 learned_addr learned my addr 172.21.15.119:0/3800690689 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f2895600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f289010e040 0x7f289019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 -- 172.21.15.119:0/3800690689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f289010e040 msgr2=0x7f289019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f289010e040 0x7f289019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 -- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900688a0 msgr2=0x7f289019a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900688a0 0x7f289019a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.033+0000 7f288fe00640 1 -- 172.21.15.119:0/3800690689 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289019f0b0 con 0x7f28900691c0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f288fe00640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f287c00e9c0 tx=0x7f287c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f288d600640 1 -- 172.21.15.119:0/3800690689 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c00ccc0 con 0x7f28900691c0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f288d600640 1 -- 172.21.15.119:0/3800690689 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f287c0035f0 con 0x7f28900691c0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289019f3a0 con 0x7f28900691c0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f288d600640 1 -- 172.21.15.119:0/3800690689 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c00ce60 con 0x7f28900691c0 2024-03-28T18:45:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.034+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28901a6c30 con 0x7f28900691c0 2024-03-28T18:45:33.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.037+0000 7f289a205740 1 Processor -- start 2024-03-28T18:45:33.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.037+0000 7f289a205740 1 -- start start 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e8443fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e8444b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e843a32c0 con 0x555e84448c00 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8434bc90 con 0x555e8443f6d0 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f289a205740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8439f360 con 0x555e84440080 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2895600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e8443fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2895600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e8443fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44394/0 (socket says 172.21.15.119:44394) 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2895600640 1 -- 172.21.15.119:0/3094127710 learned_addr learned my addr 172.21.15.119:0/3094127710 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 -- 172.21.15.119:0/3094127710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 msgr2=0x555e8443fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e8443fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 -- 172.21.15.119:0/3094127710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 msgr2=0x555e8444b050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e8444b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.038+0000 7f2894c00640 1 -- 172.21.15.119:0/3094127710 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e84370080 con 0x555e84440080 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.039+0000 7f2894c00640 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f288402c210 tx=0x7f2884009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a895859abe996846 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.039+0000 7f288d600640 1 -- 172.21.15.119:0/3800690689 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f287c0037a0 con 0x7f28900691c0 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.039+0000 7f2881c00640 1 -- 172.21.15.119:0/3094127710 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884043070 con 0x555e84440080 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.039+0000 7f2881c00640 1 -- 172.21.15.119:0/3094127710 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288402c6f0 con 0x555e84440080 2024-03-28T18:45:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.039+0000 7f2881c00640 1 -- 172.21.15.119:0/3094127710 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28840049f0 con 0x555e84440080 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 -- 172.21.15.119:0/3094127710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 msgr2=0x555e84448630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f288402c210 tx=0x7f2884009560 comp rx=0 tx=0).stop 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 -- 172.21.15.119:0/3094127710 shutdown_connections 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e8444b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 --2- 172.21.15.119:0/3094127710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e8443fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 -- 172.21.15.119:0/3094127710 >> 172.21.15.119:0/3094127710 conn(0x555e84439da0 msgr2=0x555e8443a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f288d600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 0x7f28780a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 -- 172.21.15.119:0/3094127710 shutdown_connections 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f288d600640 1 -- 172.21.15.119:0/3800690689 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f287c0cf4f0 con 0x7f28900691c0 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f289a205740 1 -- 172.21.15.119:0/3094127710 wait complete. 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.040+0000 7f2894c00640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 0x7f28780a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.041+0000 7f289a205740 1 -- start start 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.041+0000 7f2894c00640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 0x7f28780a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ab0 tx=0x7f2884005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.041+0000 7f289a205740 1 Processor -- start 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.041+0000 7f289a205740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 0x555e84448490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e84445d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8444bb10 con 0x555e84448c00 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8435f0d0 con 0x555e84440080 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8444bc90 con 0x555e8443f6d0 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f289a205740 10 client.0 _ll_register_callbacks cb 0x555e84363e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f2894c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 0x555e84448490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44410/0 (socket says 172.21.15.119:44410) 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 -- 172.21.15.119:0/2062259400 learned_addr learned my addr 172.21.15.119:0/2062259400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f2894c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 0x555e84448490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40668/0 (socket says 172.21.15.119:40668) 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f2895600640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e84445d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 -- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 msgr2=0x555e84448490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 0x555e84448490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 -- 172.21.15.119:0/2062259400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 msgr2=0x555e84445d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e84445d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 -- 172.21.15.119:0/2062259400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e840b73f0 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f2895600640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e84445d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f288fe00640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f287c0058c0 tx=0x7f287c09b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6da5fcd1a286102 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.042+0000 7f2876000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c0b3020 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f287c09bd00 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f288402c6f0 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f287c0ba170 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 10 client.25883 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555e841817d0 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288402c6f0 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f287c0ccd40 con 0x555e84440080 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 10 client.25883 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 10 client.25883 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 20 client.25883 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:33.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844589e0 0x555e8445ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 4 client.25883 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:33.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.043+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e843b0800 con 0x555e844589e0 2024-03-28T18:45:33.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.044+0000 7f2894c00640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844589e0 0x555e8445ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.044+0000 7f2894c00640 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844589e0 0x555e8445ae00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=253a1935fdb29518 server_cookie=2dbe22750498870a in_seq=0 out_seq=0 2024-03-28T18:45:33.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.044+0000 7f2876000640 10 client.25883 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f2876000640 1 -- 172.21.15.119:0/2062259400 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555e843b0800 con 0x555e844589e0 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f2876000640 10 client.25883 handle_command_reply: tid=0 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 client.25883 shutdown 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 msgr2=0x555e84445820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f287c0058c0 tx=0x7f287c09b2c0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 shutdown_connections 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844589e0 0x555e8445ae00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84448c00 0x555e84445d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84440080 0x555e84445820 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f2876000640 0 client.25883 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 --2- 172.21.15.119:0/2062259400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e8443f6d0 0x555e84448490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 >> 172.21.15.119:0/2062259400 conn(0x555e84439da0 msgr2=0x555e8443baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 shutdown_connections 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.045+0000 7f289a205740 1 -- 172.21.15.119:0/2062259400 wait complete. 2024-03-28T18:45:33.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.046+0000 7f289a205740 20 client.25883 trim_cache size 0 max 16384 2024-03-28T18:45:33.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.070+0000 7f289a205740 1 Processor -- start 2024-03-28T18:45:33.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 -- start start 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 0x555e84447f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84448c00 0x555e84442aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901af3e0 con 0x555e8443f6d0 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901b0d60 con 0x555e84448c00 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f289a205740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901afa40 con 0x555e84440080 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f288fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f288fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40684/0 (socket says 172.21.15.119:40684) 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f288fe00640 1 -- 172.21.15.119:0/2405009771 learned_addr learned my addr 172.21.15.119:0/2405009771 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f2895600640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84448c00 0x555e84442aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.071+0000 7f2894c00640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 0x555e84447f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 -- 172.21.15.119:0/2405009771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84448c00 msgr2=0x555e84442aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84448c00 0x555e84442aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 -- 172.21.15.119:0/2405009771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 msgr2=0x555e84447f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 0x555e84447f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 -- 172.21.15.119:0/2405009771 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e8443ab30 con 0x555e84440080 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f2894c00640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 0x555e84447f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f288fe00640 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f287c003ee0 tx=0x7f287c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d32264fda1a252d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f2881200640 1 -- 172.21.15.119:0/2405009771 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c09b670 con 0x555e84440080 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f2881200640 1 -- 172.21.15.119:0/2405009771 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f287c0ba170 con 0x555e84440080 2024-03-28T18:45:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.072+0000 7f2881200640 1 -- 172.21.15.119:0/2405009771 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c0a4040 con 0x555e84440080 2024-03-28T18:45:33.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 -- 172.21.15.119:0/2405009771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 msgr2=0x555e84448490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f287c003ee0 tx=0x7f287c000bc0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 -- 172.21.15.119:0/2405009771 shutdown_connections 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e84448c00 0x555e84442aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84440080 0x555e84448490 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 --2- 172.21.15.119:0/2405009771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e8443f6d0 0x555e84447f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 -- 172.21.15.119:0/2405009771 >> 172.21.15.119:0/2405009771 conn(0x555e84439da0 msgr2=0x555e8443ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.073+0000 7f289a205740 1 -- 172.21.15.119:0/2405009771 shutdown_connections 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.074+0000 7f289a205740 1 -- 172.21.15.119:0/2405009771 wait complete. 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.074+0000 7f289a205740 1 -- start start 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.074+0000 7f289a205740 1 Processor -- start 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84440080 0x555e84450e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84448c00 0x555e8444ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901b0700 con 0x555e84440080 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901b00a0 con 0x555e8443f6d0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28901ad5a0 con 0x555e84448c00 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f289a205740 10 client.0 _ll_register_callbacks cb 0x7f287c003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44432/0 (socket says 172.21.15.119:44432) 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 -- 172.21.15.119:0/1212422557 learned_addr learned my addr 172.21.15.119:0/1212422557 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2895600640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84448c00 0x555e8444ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f288fe00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84440080 0x555e84450e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 -- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84448c00 msgr2=0x555e8444ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84448c00 0x555e8444ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 -- 172.21.15.119:0/1212422557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84440080 msgr2=0x555e84450e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84440080 0x555e84450e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.075+0000 7f2894c00640 1 -- 172.21.15.119:0/1212422557 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e8444d640 con 0x555e8443f6d0 2024-03-28T18:45:33.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2894c00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ec0 tx=0x7f2884000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc00995566a6855f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2876000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884004c00 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f287c0ba170 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f288405d070 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f288406a050 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f289a205740 10 client.25889 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555e844b4ed0 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.076+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f287c0ba170 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f28840035f0 con 0x555e8443f6d0 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 10 client.25889 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 10 client.25889 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 20 client.25889 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844b51c0 0x555e844b75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 4 client.25889 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:45:33.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e843aab50 con 0x555e844b51c0 2024-03-28T18:45:33.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f288fe00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844b51c0 0x555e844b75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f288fe00640 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844b51c0 0x555e844b75b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=333eb6dc67b4691a server_cookie=4a3cbb76897d2680 in_seq=0 out_seq=0 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.077+0000 7f2876000640 10 client.25889 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f2876000640 1 -- 172.21.15.119:0/1212422557 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e843aab50 con 0x555e844b51c0 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f2876000640 10 client.25889 handle_command_reply: tid=0 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 client.25889 shutdown 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 msgr2=0x555e844508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ec0 tx=0x7f2884000bc0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 shutdown_connections 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x555e844b51c0 0x555e844b75b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555e84448c00 0x555e8444ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555e84440080 0x555e84450e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f2876000640 0 client.25889 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 --2- 172.21.15.119:0/1212422557 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555e8443f6d0 0x555e844508c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 >> 172.21.15.119:0/1212422557 conn(0x555e84439da0 msgr2=0x555e8443b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 shutdown_connections 2024-03-28T18:45:33.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 1 -- 172.21.15.119:0/1212422557 wait complete. 2024-03-28T18:45:33.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.078+0000 7f289a205740 20 client.25889 trim_cache size 0 max 16384 2024-03-28T18:45:33.083 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 msgr2=0x7f28780a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 0x7f28780a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2884005ab0 tx=0x7f2884005fd0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 msgr2=0x7f289019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f287c00e9c0 tx=0x7f287c00edc0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 shutdown_connections 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f28780a2a90 0x7f28780a4f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f289010e040 0x7f289019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f28900691c0 0x7f289019a600 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 --2- 172.21.15.119:0/3800690689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28900688a0 0x7f289019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 >> 172.21.15.119:0/3800690689 conn(0x7f28900776a0 msgr2=0x7f289010f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 shutdown_connections 2024-03-28T18:45:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.079+0000 7f2897600640 1 -- 172.21.15.119:0/3800690689 wait complete. 2024-03-28T18:45:33.096 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:45:33.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:45:33.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 -- 172.21.15.119:0/1902600509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40101ff0 msgr2=0x7f1f4010e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 --2- 172.21.15.119:0/1902600509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40101ff0 0x7f1f4010e500 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009cd0 tx=0x7f1f3802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 -- 172.21.15.119:0/1902600509 shutdown_connections 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 --2- 172.21.15.119:0/1902600509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40101ff0 0x7f1f4010e500 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 --2- 172.21.15.119:0/1902600509 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f40101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 --2- 172.21.15.119:0/1902600509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40107640 0x7f1f40107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 -- 172.21.15.119:0/1902600509 >> 172.21.15.119:0/1902600509 conn(0x7f1f40078aa0 msgr2=0x7f1f40078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 -- 172.21.15.119:0/1902600509 shutdown_connections 2024-03-28T18:45:33.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.343+0000 7f1f48400640 1 -- 172.21.15.119:0/1902600509 wait complete. 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.344+0000 7f1f48400640 1 Processor -- start 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.344+0000 7f1f48400640 1 -- start start 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40101ff0 0x7f1f4019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 0x7f1f4019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40111a50 con 0x7f1f40101ff0 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401118d0 con 0x7f1f40101630 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f48400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f40111bd0 con 0x7f1f40107640 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44452/0 (socket says 172.21.15.119:44452) 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 -- 172.21.15.119:0/2362594025 learned_addr learned my addr 172.21.15.119:0/2362594025 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f46400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 0x7f1f4019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45000640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40101ff0 0x7f1f4019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 -- 172.21.15.119:0/2362594025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 msgr2=0x7f1f4019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 0x7f1f4019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 -- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40101ff0 msgr2=0x7f1f4019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40101ff0 0x7f1f4019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f45a00640 1 -- 172.21.15.119:0/2362594025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4019f280 con 0x7f1f40101630 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.345+0000 7f1f46400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 0x7f1f4019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400e9f0 tx=0x7f1f3400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f3e000640 1 -- 172.21.15.119:0/2362594025 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400cd80 con 0x7f1f40101630 2024-03-28T18:45:33.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4019f570 con 0x7f1f40101630 2024-03-28T18:45:33.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f3e000640 1 -- 172.21.15.119:0/2362594025 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f34010910 con 0x7f1f40101630 2024-03-28T18:45:33.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f3e000640 1 -- 172.21.15.119:0/2362594025 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f340188f0 con 0x7f1f40101630 2024-03-28T18:45:33.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.346+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f401a6e00 con 0x7f1f40101630 2024-03-28T18:45:33.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.349+0000 7f1f4b1dd740 1 Processor -- start 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.349+0000 7f1f4b1dd740 1 -- start start 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.349+0000 7f1f3e000640 1 -- 172.21.15.119:0/2362594025 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f1f34018af0 con 0x7f1f40101630 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.349+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e42ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e4b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6da62c0 con 0x55bcc6e426d0 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6d4ec90 con 0x55bcc6e4bc00 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6da2360 con 0x55bcc6e43080 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f46400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e42ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e4b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44464/0 (socket says 172.21.15.119:44464) 2024-03-28T18:45:33.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 -- 172.21.15.119:0/1236407661 learned_addr learned my addr 172.21.15.119:0/1236407661 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 -- 172.21.15.119:0/1236407661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 msgr2=0x55bcc6e4b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e4b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 -- 172.21.15.119:0/1236407661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 msgr2=0x55bcc6e42ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e42ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.350+0000 7f1f45000640 1 -- 172.21.15.119:0/1236407661 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcc6d73080 con 0x55bcc6e4bc00 2024-03-28T18:45:33.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f45000640 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009ca0 tx=0x7f1f2c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63c49c34f2c5f4e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f32a00640 1 -- 172.21.15.119:0/1236407661 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c01e070 con 0x55bcc6e4bc00 2024-03-28T18:45:33.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f3e000640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 0x7f1f280a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f32a00640 1 -- 172.21.15.119:0/1236407661 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f2c019b10 con 0x55bcc6e4bc00 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f32a00640 1 -- 172.21.15.119:0/1236407661 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00ea30 con 0x55bcc6e4bc00 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f3e000640 1 -- 172.21.15.119:0/2362594025 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f340cf4f0 con 0x7f1f40101630 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.351+0000 7f1f46400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 0x7f1f280a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1236407661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 msgr2=0x55bcc6e4e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009ca0 tx=0x7f1f2c011810 comp rx=0 tx=0).stop 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1236407661 shutdown_connections 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4e050 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e4b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1236407661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e42ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1236407661 >> 172.21.15.119:0/1236407661 conn(0x55bcc6e3cda0 msgr2=0x55bcc6e3d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f46400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 0x7f1f280a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f3802c630 tx=0x7f1f38005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1236407661 shutdown_connections 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1236407661 wait complete. 2024-03-28T18:45:33.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.352+0000 7f1f4b1dd740 1 -- start start 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 Processor -- start 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e48850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e48db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6e4eb70 con 0x55bcc6e426d0 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6d620d0 con 0x55bcc6e43080 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcc6e4ecf0 con 0x55bcc6e4bc00 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f4b1dd740 10 client.0 _ll_register_callbacks cb 0x55bcc6d66e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f46400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e48db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f45000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e48850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59770/0 (socket says 172.21.15.119:59770) 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f46400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e48db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40716/0 (socket says 172.21.15.119:40716) 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.353+0000 7f1f45a00640 1 -- 172.21.15.119:0/3790470063 learned_addr learned my addr 172.21.15.119:0/3790470063 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 -- 172.21.15.119:0/3790470063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 msgr2=0x55bcc6e48db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e48db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 -- 172.21.15.119:0/3790470063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 msgr2=0x55bcc6e4b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 -- 172.21.15.119:0/3790470063 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcc6aba3f0 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45a00640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f45000640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e48850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009c70 tx=0x7f1f2c00e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fc6a842f788a744 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f27400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c028020 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f2c019b10 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.354+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f2c019cc0 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f2c033050 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 10 client.25901 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bcc6b847d0 con 0x55bcc6e43080 2024-03-28T18:45:33.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f2c019b10 con 0x55bcc6e43080 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f2c00ece0 con 0x55bcc6e43080 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 10 client.25901 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 10 client.25901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 20 client.25901 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6e5b8d0 0x55bcc6e5dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 4 client.25901 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcc6db3800 con 0x55bcc6e5b8d0 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.355+0000 7f1f45a00640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6e5b8d0 0x55bcc6e5dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.356+0000 7f1f27400640 10 client.25901 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.356+0000 7f1f45a00640 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6e5b8d0 0x55bcc6e5dcc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c3e71813e238990 server_cookie=1f9483e1a06b2760 in_seq=0 out_seq=0 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f27400640 1 -- 172.21.15.119:0/3790470063 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bcc6db3800 con 0x55bcc6e5b8d0 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f27400640 10 client.25901 handle_command_reply: tid=0 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 client.25901 shutdown 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 msgr2=0x55bcc6e48850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e48850 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009c70 tx=0x7f1f2c00e3b0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 shutdown_connections 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6e5b8d0 0x55bcc6e5dcc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e48db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e48850 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f27400640 0 client.25901 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/3790470063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 >> 172.21.15.119:0/3790470063 conn(0x55bcc6e3cda0 msgr2=0x55bcc6e3eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 shutdown_connections 2024-03-28T18:45:33.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/3790470063 wait complete. 2024-03-28T18:45:33.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.357+0000 7f1f4b1dd740 20 client.25901 trim_cache size 0 max 16384 2024-03-28T18:45:33.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 Processor -- start 2024-03-28T18:45:33.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 -- start start 2024-03-28T18:45:33.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e46ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e4af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401af5e0 con 0x55bcc6e426d0 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401b0f60 con 0x55bcc6e43080 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401afc40 con 0x55bcc6e4bc00 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f46400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f46400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40732/0 (socket says 172.21.15.119:40732) 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f46400640 1 -- 172.21.15.119:0/1910536879 learned_addr learned my addr 172.21.15.119:0/1910536879 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f45a00640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e46ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.371+0000 7f1f45000640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e4af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 -- 172.21.15.119:0/1910536879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 msgr2=0x55bcc6e4af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e4af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 -- 172.21.15.119:0/1910536879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 msgr2=0x55bcc6e46ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e46ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 -- 172.21.15.119:0/1910536879 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcc6e4ea50 con 0x55bcc6e4bc00 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f45000640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e4af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:33.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f46400640 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009ca0 tx=0x7f1f38041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29067c28ad694a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f32000640 1 -- 172.21.15.119:0/1910536879 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38039040 con 0x55bcc6e4bc00 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f32000640 1 -- 172.21.15.119:0/1910536879 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f38003e50 con 0x55bcc6e4bc00 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.372+0000 7f1f32000640 1 -- 172.21.15.119:0/1910536879 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38004040 con 0x55bcc6e4bc00 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1910536879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 msgr2=0x55bcc6e4b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009ca0 tx=0x7f1f38041000 comp rx=0 tx=0).stop 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1910536879 shutdown_connections 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e4bc00 0x55bcc6e4b4c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e43080 0x55bcc6e4af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/1910536879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e46ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1910536879 >> 172.21.15.119:0/1910536879 conn(0x55bcc6e3cda0 msgr2=0x55bcc6e3e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1910536879 shutdown_connections 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/1910536879 wait complete. 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.373+0000 7f1f4b1dd740 1 -- start start 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 Processor -- start 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6eaffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401b0900 con 0x55bcc6e426d0 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401b02a0 con 0x55bcc6e4bc00 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f401ad7a0 con 0x55bcc6e43080 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f4b1dd740 10 client.0 _ll_register_callbacks cb 0x7f1f2c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f45000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f45000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40746/0 (socket says 172.21.15.119:40746) 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f45000640 1 -- 172.21.15.119:0/2004382520 learned_addr learned my addr 172.21.15.119:0/2004382520 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.374+0000 7f1f45a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59796/0 (socket says 172.21.15.119:59796) 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f46400640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6eaffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 -- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 msgr2=0x55bcc6eaffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6eaffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 -- 172.21.15.119:0/2004382520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 msgr2=0x55bcc6e4fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 -- 172.21.15.119:0/2004382520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcc6eb05a0 con 0x55bcc6e43080 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f46400640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6eaffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f45000640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0198c0 tx=0x7f1f2c0048b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e299167156c00a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f27400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c028070 con 0x55bcc6e43080 2024-03-28T18:45:33.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f38003e50 con 0x55bcc6e43080 2024-03-28T18:45:33.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.375+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1f2c004a30 con 0x55bcc6e43080 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.378+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f2c03f050 con 0x55bcc6e43080 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.378+0000 7f1f4b1dd740 10 client.25930 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.378+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bcc6eb7950 con 0x55bcc6e43080 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.378+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f38003e50 con 0x55bcc6e43080 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1f2c038470 con 0x55bcc6e43080 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 10 client.25930 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 10 client.25930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 20 client.25930 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6eb8050 0x55bcc6eba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 4 client.25930 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:45:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bcc6dadb50 con 0x55bcc6eb8050 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6eb8050 0x55bcc6eba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f45a00640 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6eb8050 0x55bcc6eba470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=190f64671ed457c7 server_cookie=1c462ce11540bc57 in_seq=0 out_seq=0 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.379+0000 7f1f27400640 10 client.25930 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f27400640 1 -- 172.21.15.119:0/2004382520 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bcc6dadb50 con 0x55bcc6eb8050 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f27400640 10 client.25930 handle_command_reply: tid=0 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 client.25930 shutdown 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 msgr2=0x55bcc6e50130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0198c0 tx=0x7f1f2c0048b0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 shutdown_connections 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bcc6eb8050 0x55bcc6eba470 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bcc6e4bc00 0x55bcc6eaffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bcc6e43080 0x55bcc6e50130 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f27400640 0 client.25930 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 --2- 172.21.15.119:0/2004382520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcc6e426d0 0x55bcc6e4fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 >> 172.21.15.119:0/2004382520 conn(0x55bcc6e3cda0 msgr2=0x55bcc6e3e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 shutdown_connections 2024-03-28T18:45:33.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 1 -- 172.21.15.119:0/2004382520 wait complete. 2024-03-28T18:45:33.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.380+0000 7f1f4b1dd740 20 client.25930 trim_cache size 0 max 16384 2024-03-28T18:45:33.383 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 msgr2=0x7f1f280a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 0x7f1f280a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1f3802c630 tx=0x7f1f38005d80 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 msgr2=0x7f1f4019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400e9f0 tx=0x7f1f3400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 shutdown_connections 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1f280a2a90 0x7f1f280a4f50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f40107640 0x7f1f4019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f40101ff0 0x7f1f4019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 --2- 172.21.15.119:0/2362594025 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1f40101630 0x7f1f4019a290 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 >> 172.21.15.119:0/2362594025 conn(0x7f1f40078aa0 msgr2=0x7f1f4010ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:33.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 shutdown_connections 2024-03-28T18:45:33.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:33.381+0000 7f1f48400640 1 -- 172.21.15.119:0/2362594025 wait complete. 2024-03-28T18:45:33.562 INFO:tasks.workunit.client.0.smithi119.stderr:Updating files: 60% (7413/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 99% (12189/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.164 INFO:tasks.workunit.client.0.smithi119.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: git switch -c 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr:Or undo this operation with: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: git switch - 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-03-28T18:45:34.165 INFO:tasks.workunit.client.0.smithi119.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T18:45:34.173 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T18:45:34.283 INFO:tasks.workunit.client.0.smithi119.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T18:45:34.291 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:45:34.291 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T18:45:34.403 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:45:34.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:45:34.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.628+0000 7f8984a00640 1 -- 172.21.15.119:0/167677107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8980102930 msgr2=0x7f8980102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.628+0000 7f8984a00640 1 --2- 172.21.15.119:0/167677107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8980102930 0x7f8980102d10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8970009900 tx=0x7f897002bcd0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 -- 172.21.15.119:0/167677107 shutdown_connections 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 --2- 172.21.15.119:0/167677107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f898010bc70 0x7f898010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 --2- 172.21.15.119:0/167677107 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 0x7f898010b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 --2- 172.21.15.119:0/167677107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8980102930 0x7f8980102d10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 -- 172.21.15.119:0/167677107 >> 172.21.15.119:0/167677107 conn(0x7f8980077700 msgr2=0x7f8980077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 -- 172.21.15.119:0/167677107 shutdown_connections 2024-03-28T18:45:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.629+0000 7f8984a00640 1 -- 172.21.15.119:0/167677107 wait complete. 2024-03-28T18:45:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 Processor -- start 2024-03-28T18:45:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 -- start start 2024-03-28T18:45:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 0x7f8980100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f898010bc70 0x7f8980101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980111a40 con 0x7f898010bc70 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801118c0 con 0x7f8980103250 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f8984a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980111bc0 con 0x7f8980102930 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f897de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f897de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40762/0 (socket says 172.21.15.119:40762) 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f897de00640 1 -- 172.21.15.119:0/3562714376 learned_addr learned my addr 172.21.15.119:0/3562714376 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f897d400640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 0x7f8980100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.630+0000 7f897e800640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f898010bc70 0x7f8980101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 -- 172.21.15.119:0/3562714376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 msgr2=0x7f8980100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 0x7f8980100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 -- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f898010bc70 msgr2=0x7f8980101000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f898010bc70 0x7f8980101000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 -- 172.21.15.119:0/3562714376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89801a6460 con 0x7f8980102930 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f897de00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8970005940 tx=0x7f897002cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f8976a00640 1 -- 172.21.15.119:0/3562714376 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8970041070 con 0x7f8980102930 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f8976a00640 1 -- 172.21.15.119:0/3562714376 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8970004450 con 0x7f8980102930 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89801a66f0 con 0x7f8980102930 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f8976a00640 1 -- 172.21.15.119:0/3562714376 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8970003d80 con 0x7f8980102930 2024-03-28T18:45:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.631+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89801a6be0 con 0x7f8980102930 2024-03-28T18:45:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.634+0000 7f8976a00640 1 -- 172.21.15.119:0/3562714376 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f8970044070 con 0x7f8980102930 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.634+0000 7f8987792740 1 Processor -- start 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 -- start start 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4ead040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4e08eb0 con 0x55afc4ea49a0 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4db1a60 con 0x55afc4ead580 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f8987792740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4e04f50 con 0x55afc4ea4080 2024-03-28T18:45:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f897e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f897de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4ead040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f897e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40776/0 (socket says 172.21.15.119:40776) 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f897e800640 1 -- 172.21.15.119:0/3332857615 learned_addr learned my addr 172.21.15.119:0/3332857615 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.635+0000 7f897d400640 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 -- 172.21.15.119:0/3332857615 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 msgr2=0x55afc4eaf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 -- 172.21.15.119:0/3332857615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 msgr2=0x55afc4ead040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4ead040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 -- 172.21.15.119:0/3332857615 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afc4be57e0 con 0x55afc4ea4080 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f8976a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 0x7f89640a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f8976a00640 1 -- 172.21.15.119:0/3562714376 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f89700f0bd0 con 0x7f8980102930 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897e800640 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8978009f40 tx=0x7f89780118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c5bd9ea3328fa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f896b000640 1 -- 172.21.15.119:0/3332857615 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897801e070 con 0x55afc4ea4080 2024-03-28T18:45:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f897de00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 0x7f89640a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f896b000640 1 -- 172.21.15.119:0/3332857615 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8978003c30 con 0x55afc4ea4080 2024-03-28T18:45:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.636+0000 7f896b000640 1 -- 172.21.15.119:0/3332857615 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800f990 con 0x55afc4ea4080 2024-03-28T18:45:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 -- 172.21.15.119:0/3332857615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 msgr2=0x55afc4ea4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f897de00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 0x7f89640a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89700ee250 tx=0x7f89700f4660 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8978009f40 tx=0x7f89780118d0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 -- 172.21.15.119:0/3332857615 shutdown_connections 2024-03-28T18:45:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4ead040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 --2- 172.21.15.119:0/3332857615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4ea4460 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 -- 172.21.15.119:0/3332857615 >> 172.21.15.119:0/3332857615 conn(0x55afc4e9e7b0 msgr2=0x55afc4e9eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 -- 172.21.15.119:0/3332857615 shutdown_connections 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.637+0000 7f8987792740 1 -- 172.21.15.119:0/3332857615 wait complete. 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 -- start start 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 Processor -- start 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4eacf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4eb0370 con 0x55afc4ea49a0 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4dc4eb0 con 0x55afc4ead580 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.638+0000 7f8987792740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4eb04f0 con 0x55afc4ea4080 2024-03-28T18:45:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f8987792740 10 client.0 _ll_register_callbacks cb 0x55afc4db9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4eacf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59828/0 (socket says 172.21.15.119:59828) 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 -- 172.21.15.119:0/1075963109 learned_addr learned my addr 172.21.15.119:0/1075963109 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897e800640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 -- 172.21.15.119:0/1075963109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 msgr2=0x55afc4eacf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4eacf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 -- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 msgr2=0x55afc4eaa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 -- 172.21.15.119:0/1075963109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afc4c45f70 con 0x55afc4ea49a0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897e800640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f897d400640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f896c012700 tx=0x7f896c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f8bda3b6f217a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f895f400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.639+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c00f8c0 con 0x55afc4ea49a0 2024-03-28T18:45:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f896c00fa60 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8978003c30 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f896c00fc60 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 10 client.15777 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55afc4add170 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8978003c30 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f896c01c420 con 0x55afc4ea49a0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 10 client.15777 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 20 client.15777 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.640+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4ebd650 0x55afc4ebfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.641+0000 7f8987792740 4 client.15777 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.641+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afc4e163f0 con 0x55afc4ebd650 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.641+0000 7f897e800640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4ebd650 0x55afc4ebfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.641+0000 7f897e800640 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4ebd650 0x55afc4ebfa40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7654ff23b35b171 server_cookie=c88f17b7b2831fc1 in_seq=0 out_seq=0 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.641+0000 7f895f400640 10 client.15777 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f895f400640 1 -- 172.21.15.119:0/1075963109 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55afc4e163f0 con 0x55afc4ebd650 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f895f400640 10 client.15777 handle_command_reply: tid=0 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 client.15777 shutdown 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 msgr2=0x55afc4eaa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f896c012700 tx=0x7f896c012bc0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 shutdown_connections 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4ebd650 0x55afc4ebfa40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ead580 0x55afc4eaa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eaa290 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 --2- 172.21.15.119:0/1075963109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ea4080 0x55afc4eacf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f895f400640 0 client.15777 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 >> 172.21.15.119:0/1075963109 conn(0x55afc4e9e7b0 msgr2=0x55afc4ea0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 shutdown_connections 2024-03-28T18:45:34.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.642+0000 7f8987792740 1 -- 172.21.15.119:0/1075963109 wait complete. 2024-03-28T18:45:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.643+0000 7f8987792740 20 client.15777 trim_cache size 0 max 16384 2024-03-28T18:45:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.659+0000 7f8987792740 1 Processor -- start 2024-03-28T18:45:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.659+0000 7f8987792740 1 -- start start 2024-03-28T18:45:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eacf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4eb8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801af5c0 con 0x55afc4ea49a0 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801b0f40 con 0x55afc4ea4080 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f8987792740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801afc20 con 0x55afc4ead580 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44536/0 (socket says 172.21.15.119:44536) 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 -- 172.21.15.119:0/2340459613 learned_addr learned my addr 172.21.15.119:0/2340459613 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897d400640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eacf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4eb8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 -- 172.21.15.119:0/2340459613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 msgr2=0x55afc4eb8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4eb8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 -- 172.21.15.119:0/2340459613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 msgr2=0x55afc4eacf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eacf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 -- 172.21.15.119:0/2340459613 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afc4ea0190 con 0x55afc4ea4080 2024-03-28T18:45:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897d400640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eacf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.660+0000 7f897e800640 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f897800f560 tx=0x7f89780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0f9d5d8e0bac32 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f896a600640 1 -- 172.21.15.119:0/2340459613 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897801e040 con 0x55afc4ea4080 2024-03-28T18:45:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f896a600640 1 -- 172.21.15.119:0/2340459613 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8978004e30 con 0x55afc4ea4080 2024-03-28T18:45:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f896a600640 1 -- 172.21.15.119:0/2340459613 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978004260 con 0x55afc4ea4080 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 -- 172.21.15.119:0/2340459613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 msgr2=0x55afc4eac9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f897800f560 tx=0x7f89780118d0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 -- 172.21.15.119:0/2340459613 shutdown_connections 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4eb8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea49a0 0x55afc4eacf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 --2- 172.21.15.119:0/2340459613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea4080 0x55afc4eac9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 -- 172.21.15.119:0/2340459613 >> 172.21.15.119:0/2340459613 conn(0x55afc4e9e7b0 msgr2=0x55afc4e9d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.661+0000 7f8987792740 1 -- 172.21.15.119:0/2340459613 shutdown_connections 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 -- 172.21.15.119:0/2340459613 wait complete. 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 -- start start 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 Processor -- start 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 0x55afc4eb2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4f18ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801b08e0 con 0x55afc4ea4080 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801b0280 con 0x55afc4ea49a0 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89801ad780 con 0x55afc4ead580 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.662+0000 7f8987792740 10 client.0 _ll_register_callbacks cb 0x7f896c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44542/0 (socket says 172.21.15.119:44542) 2024-03-28T18:45:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 -- 172.21.15.119:0/1451893343 learned_addr learned my addr 172.21.15.119:0/1451893343 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897e800640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 0x55afc4eb2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897e800640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4f18ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 -- 172.21.15.119:0/1451893343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 msgr2=0x55afc4f18ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4f18ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 -- 172.21.15.119:0/1451893343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 msgr2=0x55afc4eb2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 0x55afc4eb2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 -- 172.21.15.119:0/1451893343 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afc4eaab80 con 0x55afc4ea49a0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897e800640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 0x55afc4eb2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f897d400640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55afc4eab860 tx=0x7f896c00fef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24119449ebf7220a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f895f400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c00f790 con 0x55afc4ea49a0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8978004e30 con 0x55afc4ea49a0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.663+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f896c00f930 con 0x55afc4ea49a0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f896c032050 con 0x55afc4ea49a0 2024-03-28T18:45:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 10 client.25925 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55afc4f1a410 con 0x55afc4ea49a0 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8978004e30 con 0x55afc4ea49a0 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f896c02b060 con 0x55afc4ea49a0 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 10 client.25925 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 20 client.25925 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4f1ab00 0x55afc4f1cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 4 client.25925 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.664+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55afc4e10740 con 0x55afc4f1ab00 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.665+0000 7f897e800640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4f1ab00 0x55afc4f1cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.665+0000 7f897e800640 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4f1ab00 0x55afc4f1cf20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7338c9b45da36a1c server_cookie=8304808cad17600c in_seq=0 out_seq=0 2024-03-28T18:45:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.665+0000 7f895f400640 10 client.25925 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.666+0000 7f895f400640 1 -- 172.21.15.119:0/1451893343 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55afc4e10740 con 0x55afc4f1ab00 2024-03-28T18:45:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.666+0000 7f895f400640 10 client.25925 handle_command_reply: tid=0 2024-03-28T18:45:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.666+0000 7f8987792740 1 client.25925 shutdown 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 msgr2=0x55afc4eb2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55afc4eab860 tx=0x7f896c00fef0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 shutdown_connections 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55afc4f1ab00 0x55afc4f1cf20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afc4ead580 0x55afc4f18ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55afc4ea49a0 0x55afc4eb2c50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 --2- 172.21.15.119:0/1451893343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55afc4ea4080 0x55afc4eb2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f895f400640 0 client.25925 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 >> 172.21.15.119:0/1451893343 conn(0x55afc4e9e7b0 msgr2=0x55afc4ea4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 shutdown_connections 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 1 -- 172.21.15.119:0/1451893343 wait complete. 2024-03-28T18:45:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.667+0000 7f8987792740 20 client.25925 trim_cache size 0 max 16384 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "ab139c3c-177d-442f-85c8-9472b6e3eb63", 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 msgr2=0x7f89640a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 0x7f89640a4f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89700ee250 tx=0x7f89700f4660 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 msgr2=0x7f8980100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8970005940 tx=0x7f897002cbd0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 shutdown_connections 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f89640a2a90 0x7f89640a4f50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f898010bc70 0x7f8980101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8980103250 0x7f8980100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 --2- 172.21.15.119:0/3562714376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8980102930 0x7f8980100580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 >> 172.21.15.119:0/3562714376 conn(0x7f8980077700 msgr2=0x7f89800fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 shutdown_connections 2024-03-28T18:45:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.668+0000 7f8984a00640 1 -- 172.21.15.119:0/3562714376 wait complete. 2024-03-28T18:45:34.682 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:34 smithi119 ceph-mon[31813]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:34.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:45:34.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:45:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:34 smithi196 ceph-mon[36300]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:34 smithi131 ceph-mon[36065]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:45:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 -- 172.21.15.119:0/1584969299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc109850 msgr2=0x7f57dc109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 --2- 172.21.15.119:0/1584969299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc109850 0x7f57dc109c30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009900 tx=0x7f57cc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 -- 172.21.15.119:0/1584969299 shutdown_connections 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 --2- 172.21.15.119:0/1584969299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc104200 0x7f57dc110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 --2- 172.21.15.119:0/1584969299 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc103840 0x7f57dc103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 --2- 172.21.15.119:0/1584969299 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc109850 0x7f57dc109c30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 -- 172.21.15.119:0/1584969299 >> 172.21.15.119:0/1584969299 conn(0x7f57dc0ff350 msgr2=0x7f57dc101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 -- 172.21.15.119:0/1584969299 shutdown_connections 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.886+0000 7f57e1800640 1 -- 172.21.15.119:0/1584969299 wait complete. 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 Processor -- start 2024-03-28T18:45:34.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 -- start start 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 0x7f57dc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 0x7f57dc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 0x7f57dc19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc113cb0 con 0x7f57dc103840 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc113b30 con 0x7f57dc104200 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57e1800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc113e30 con 0x7f57dc109850 2024-03-28T18:45:34.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57daa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 0x7f57dc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57db400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 0x7f57dc19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57da000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 0x7f57dc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57daa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 0x7f57dc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59860/0 (socket says 172.21.15.119:59860) 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.887+0000 7f57daa00640 1 -- 172.21.15.119:0/3282369512 learned_addr learned my addr 172.21.15.119:0/3282369512 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 -- 172.21.15.119:0/3282369512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 msgr2=0x7f57dc19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 0x7f57dc19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 -- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 msgr2=0x7f57dc19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 0x7f57dc19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 -- 172.21.15.119:0/3282369512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc19f150 con 0x7f57dc109850 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57da000640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 0x7f57dc19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57db400640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 0x7f57dc19ea20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f57d400da10 tx=0x7f57d400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57d3400640 1 -- 172.21.15.119:0/3282369512 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400bd70 con 0x7f57dc109850 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57dc19f440 con 0x7f57dc109850 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57d3400640 1 -- 172.21.15.119:0/3282369512 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d4003e50 con 0x7f57dc109850 2024-03-28T18:45:34.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57d3400640 1 -- 172.21.15.119:0/3282369512 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d40106a0 con 0x7f57dc109850 2024-03-28T18:45:34.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.888+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57dc1a6cd0 con 0x7f57dc109850 2024-03-28T18:45:34.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.891+0000 7f57d3400640 1 -- 172.21.15.119:0/3282369512 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 26) v1 ==== 137524+0+0 (secure 0 0 0) 0x7f57d4003970 con 0x7f57dc109850 2024-03-28T18:45:34.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.891+0000 7f57e45d5740 1 Processor -- start 2024-03-28T18:45:34.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 -- start start 2024-03-28T18:45:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae161680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae0ba250 con 0x5579ae15f290 2024-03-28T18:45:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae062a30 con 0x5579ae155f20 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57e45d5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae0b62f0 con 0x5579ae156c00 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57daa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57daa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44570/0 (socket says 172.21.15.119:44570) 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57daa00640 1 -- 172.21.15.119:0/4157493130 learned_addr learned my addr 172.21.15.119:0/4157493130 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57da000640 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.892+0000 7f57da000640 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 -- 172.21.15.119:0/4157493130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 msgr2=0x5579ae155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 -- 172.21.15.119:0/4157493130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 msgr2=0x5579ae161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae161680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 -- 172.21.15.119:0/4157493130 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579ae04d690 con 0x5579ae155f20 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57d3400640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 0x7f57c00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57daa00640 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036040 tx=0x7f57cc009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503e40da0d2fa5ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57d3400640 1 -- 172.21.15.119:0/3282369512 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f57d40cf510 con 0x7f57dc109850 2024-03-28T18:45:34.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57da000640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 0x7f57c00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57c7e00640 1 -- 172.21.15.119:0/4157493130 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc042070 con 0x5579ae155f20 2024-03-28T18:45:34.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57c7e00640 1 -- 172.21.15.119:0/4157493130 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57cc004240 con 0x5579ae155f20 2024-03-28T18:45:34.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.893+0000 7f57c7e00640 1 -- 172.21.15.119:0/4157493130 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc005120 con 0x5579ae155f20 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 -- 172.21.15.119:0/4157493130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 msgr2=0x5579ae1563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57da000640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 0x7f57c00a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f57c80058c0 tx=0x7f57c8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036040 tx=0x7f57cc009610 comp rx=0 tx=0).stop 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 -- 172.21.15.119:0/4157493130 shutdown_connections 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae161680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae1563a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4157493130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 -- 172.21.15.119:0/4157493130 >> 172.21.15.119:0/4157493130 conn(0x5579ae11d200 msgr2=0x5579ae150970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 -- 172.21.15.119:0/4157493130 shutdown_connections 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.894+0000 7f57e45d5740 1 -- 172.21.15.119:0/4157493130 wait complete. 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 -- start start 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 Processor -- start 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae15bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae15c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae1620a0 con 0x5579ae15f290 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae075e80 con 0x5579ae155f20 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579ae162220 con 0x5579ae156c00 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57e45d5740 10 client.0 _ll_register_callbacks cb 0x5579ae088130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57daa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57daa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44572/0 (socket says 172.21.15.119:44572) 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57daa00640 1 -- 172.21.15.119:0/642320022 learned_addr learned my addr 172.21.15.119:0/642320022 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57da000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae15bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.895+0000 7f57db400640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae15c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 -- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 msgr2=0x5579ae15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 -- 172.21.15.119:0/642320022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 msgr2=0x5579ae15c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae15c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 -- 172.21.15.119:0/642320022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579ade987d0 con 0x5579ae156c00 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57db400640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae15c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57daa00640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57da000640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae15bcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f57c80038a0 tx=0x7f57c8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f0c3af7bbb65530 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c802a440 con 0x5579ae156c00 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57cc004240 con 0x5579ae156c00 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.896+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f57c8020440 con 0x5579ae156c00 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57c8020640 con 0x5579ae156c00 2024-03-28T18:45:34.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 10 client.25957 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5579addce3f0 con 0x5579ae156c00 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57cc004240 con 0x5579ae156c00 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f57c8028020 con 0x5579ae156c00 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 10 client.25957 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 10 client.25957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 20 client.25957 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae16f110 0x5579ae171500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 4 client.25957 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579ae0c7790 con 0x5579ae16f110 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.897+0000 7f57daa00640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae16f110 0x5579ae171500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.898+0000 7f57daa00640 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae16f110 0x5579ae171500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a97ee21eec698c2 server_cookie=9f211bd3f0d1f60d in_seq=0 out_seq=0 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.898+0000 7f57c4c00640 10 client.25957 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57c4c00640 1 -- 172.21.15.119:0/642320022 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5579ae0c7790 con 0x5579ae16f110 2024-03-28T18:45:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57c4c00640 10 client.25957 handle_command_reply: tid=0 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 client.25957 shutdown 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 msgr2=0x5579ae15bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae15bcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f57c80038a0 tx=0x7f57c8005850 comp rx=0 tx=0).stop 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 shutdown_connections 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae16f110 0x5579ae171500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae15f290 0x5579ae15c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57c4c00640 0 client.25957 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae156c00 0x5579ae15bcd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 --2- 172.21.15.119:0/642320022 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae155f20 0x5579ae15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 >> 172.21.15.119:0/642320022 conn(0x5579ae11d200 msgr2=0x5579ae1526c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 shutdown_connections 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 1 -- 172.21.15.119:0/642320022 wait complete. 2024-03-28T18:45:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.899+0000 7f57e45d5740 20 client.25957 trim_cache size 0 max 16384 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 Processor -- start 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 -- start start 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae15e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae160680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1af560 con 0x5579ae156c00 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1b0ee0 con 0x5579ae15f290 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57e45d5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1afbc0 con 0x5579ae155f20 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.916+0000 7f57daa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40878/0 (socket says 172.21.15.119:40878) 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 -- 172.21.15.119:0/4002850008 learned_addr learned my addr 172.21.15.119:0/4002850008 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57db400640 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57da000640 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 -- 172.21.15.119:0/4002850008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 msgr2=0x5579ae160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 -- 172.21.15.119:0/4002850008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 msgr2=0x5579ae15e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae15e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 -- 172.21.15.119:0/4002850008 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579ae171870 con 0x5579ae155f20 2024-03-28T18:45:34.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57daa00640 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f57cc02c240 tx=0x7f57cc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c81d1498b9f55b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57c7400640 1 -- 172.21.15.119:0/4002850008 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc042040 con 0x5579ae155f20 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57c7400640 1 -- 172.21.15.119:0/4002850008 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57cc004810 con 0x5579ae155f20 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.917+0000 7f57c7400640 1 -- 172.21.15.119:0/4002850008 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc0035f0 con 0x5579ae155f20 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 -- 172.21.15.119:0/4002850008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 msgr2=0x5579ae15e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f57cc02c240 tx=0x7f57cc02bc30 comp rx=0 tx=0).stop 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 -- 172.21.15.119:0/4002850008 shutdown_connections 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae15e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 --2- 172.21.15.119:0/4002850008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae15e400 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 -- 172.21.15.119:0/4002850008 >> 172.21.15.119:0/4002850008 conn(0x5579ae11d200 msgr2=0x5579ae1504a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 -- 172.21.15.119:0/4002850008 shutdown_connections 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.918+0000 7f57e45d5740 1 -- 172.21.15.119:0/4002850008 wait complete. 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 -- start start 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 Processor -- start 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae1673a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae158f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1b0880 con 0x5579ae156c00 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1b0220 con 0x5579ae15f290 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1ad720 con 0x5579ae155f20 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57e45d5740 10 client.0 _ll_register_callbacks cb 0x7f57c8009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:45:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57daa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57daa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40886/0 (socket says 172.21.15.119:40886) 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57daa00640 1 -- 172.21.15.119:0/2084798693 learned_addr learned my addr 172.21.15.119:0/2084798693 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57da000640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae1673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.919+0000 7f57db400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 -- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 msgr2=0x5579ae158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 -- 172.21.15.119:0/2084798693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 msgr2=0x5579ae1673a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae1673a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 -- 172.21.15.119:0/2084798693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579add8e0f0 con 0x5579ae155f20 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57db400640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57daa00640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036040 tx=0x7f57cc009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbc84df047cb66d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc058050 con 0x5579ae155f20 2024-03-28T18:45:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f57cc004040 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.920+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57cc004810 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57cc042040 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 10 client.25963 fetch_fsmap learned FSMap version 28 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5579ae1cb3f0 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57cc004810 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f57cc0035f0 con 0x5579ae155f20 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 10 client.25963 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 20 client.25963 populate_metadata read hostname 'smithi119' 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae1cb670 0x5579ae1cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 4 client.25963 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5579ae0c1ae0 con 0x5579ae1cb670 2024-03-28T18:45:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.921+0000 7f57da000640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae1cb670 0x5579ae1cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57da000640 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae1cb670 0x5579ae1cda60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=826505aa631bf26f server_cookie=910f10c1a3f03cc5 in_seq=0 out_seq=0 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57c4c00640 10 client.25963 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57c4c00640 1 -- 172.21.15.119:0/2084798693 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5579ae0c1ae0 con 0x5579ae1cb670 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57c4c00640 10 client.25963 handle_command_reply: tid=0 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 client.25963 shutdown 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 msgr2=0x5579ae166e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f57cc036040 tx=0x7f57cc009610 comp rx=0 tx=0).stop 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 shutdown_connections 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5579ae1cb670 0x5579ae1cda60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5579ae15f290 0x5579ae158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ae156c00 0x5579ae1673a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57c4c00640 0 client.25963 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 --2- 172.21.15.119:0/2084798693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5579ae155f20 0x5579ae166e60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 >> 172.21.15.119:0/2084798693 conn(0x5579ae11d200 msgr2=0x5579ae14f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 shutdown_connections 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.922+0000 7f57e45d5740 1 -- 172.21.15.119:0/2084798693 wait complete. 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e45d5740 20 client.25963 trim_cache size 0 max 16384 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:45:34.928 INFO:teuthology.orchestra.run.smithi119.stdout: "ab139c3c-177d-442f-85c8-9472b6e3eb63": { 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "ab139c3c-177d-442f-85c8-9472b6e3eb63", 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:45:34.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 msgr2=0x7f57c00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 0x7f57c00a4f50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f57c80058c0 tx=0x7f57c8005f90 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 msgr2=0x7f57dc19ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 0x7f57dc19ea20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f57d400da10 tx=0x7f57d400ded0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 shutdown_connections 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f57c00a2a90 0x7f57c00a4f50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57dc109850 0x7f57dc19ea20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f57dc104200 0x7f57dc19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 --2- 172.21.15.119:0/3282369512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57dc103840 0x7f57dc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 >> 172.21.15.119:0/3282369512 conn(0x7f57dc0ff350 msgr2=0x7f57dc0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 shutdown_connections 2024-03-28T18:45:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:45:34.923+0000 7f57e1800640 1 -- 172.21.15.119:0/3282369512 wait complete. 2024-03-28T18:45:34.941 INFO:tasks.cephfs.filesystem:scrub status for tag:ab139c3c-177d-442f-85c8-9472b6e3eb63 - {'path': '/', 'tag': 'ab139c3c-177d-442f-85c8-9472b6e3eb63', 'options': 'recursive,force'} 2024-03-28T18:45:35.192 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T18:45:35.319 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T18:45:35.412 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T18:45:35.417 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:45:35.417 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T18:45:35.492 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T18:45:35.501 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:45:35.501 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T18:45:35.546 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-03-28T18:45:35.548 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-03-28T18:45:35.548 DEBUG:teuthology.orchestra.run.smithi119:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-28T18:45:35.664 INFO:tasks.workunit.client.0.smithi119.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T18:45:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:35 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:45:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:35 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:35 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:45:35.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:35 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:35 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:45:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:35 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:35 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:45:35.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:35 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:35.770 INFO:tasks.workunit.client.0.smithi119.stdout: Iozone: Performance Test of File I/O 2024-03-28T18:45:35.770 INFO:tasks.workunit.client.0.smithi119.stdout: Version $Revision: 3.494 $ 2024-03-28T18:45:35.770 INFO:tasks.workunit.client.0.smithi119.stdout: Compiled for 64 bit mode. 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Build: linux-AMD64 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Run began: Thu Mar 28 18:45:35 2024 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:35.771 INFO:tasks.workunit.client.0.smithi119.stdout: Include close in write timing 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Include fsync in write timing 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: File size set to 1048576 kB 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Record Size 16 kB 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Output is in kBytes/sec 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache line size set to 32 bytes. 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: File stride size set to 17 * record size. 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Throughput test with 1 process 2024-03-28T18:45:35.772 INFO:tasks.workunit.client.0.smithi119.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-03-28T18:45:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:35 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:45:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:35 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:35 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:45:35.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:35 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:45:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:36 smithi196 ceph-mon[36300]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-03-28T18:45:36.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:36 smithi196 ceph-mon[36300]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:45:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:36 smithi131 ceph-mon[36065]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-03-28T18:45:36.698 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:36 smithi131 ceph-mon[36065]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:45:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:36 smithi119 ceph-mon[31813]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 14 op/s 2024-03-28T18:45:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:36 smithi119 ceph-mon[31813]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-28T18:45:38.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:38 smithi196 ceph-mon[36300]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T18:45:38.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:38 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:45:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:38 smithi131 ceph-mon[36065]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T18:45:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:38 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:45:38.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:38 smithi119 ceph-mon[31813]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-28T18:45:38.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:38 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 initial writers = 283816.19 kB/sec 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 initial writers = 283761.29 kB/sec 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 283816.19 kB/sec 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 283816.19 kB/sec 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 283816.19 kB/sec 2024-03-28T18:45:39.471 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:45:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:40 smithi196 ceph-mon[36300]: pgmap v51: 97 pgs: 97 active+clean; 709 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 59 MiB/s wr, 30 op/s 2024-03-28T18:45:40.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:40 smithi131 ceph-mon[36065]: pgmap v51: 97 pgs: 97 active+clean; 709 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 59 MiB/s wr, 30 op/s 2024-03-28T18:45:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:40 smithi119 ceph-mon[31813]: pgmap v51: 97 pgs: 97 active+clean; 709 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 59 MiB/s wr, 30 op/s 2024-03-28T18:45:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:42 smithi196 ceph-mon[36300]: pgmap v52: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:42 smithi131 ceph-mon[36065]: pgmap v52: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:42 smithi119 ceph-mon[31813]: pgmap v52: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:44 smithi196 ceph-mon[36300]: pgmap v53: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:44 smithi131 ceph-mon[36065]: pgmap v53: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:44 smithi119 ceph-mon[31813]: pgmap v53: 97 pgs: 97 active+clean; 897 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 75 MiB/s wr, 30 op/s 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 rewriters = 351824.88 kB/sec 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 rewriters = 351738.71 kB/sec 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 351824.88 kB/sec 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 351824.88 kB/sec 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 351824.88 kB/sec 2024-03-28T18:45:46.565 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:45:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:46 smithi196 ceph-mon[36300]: pgmap v54: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 37 op/s 2024-03-28T18:45:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:46 smithi131 ceph-mon[36065]: pgmap v54: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 37 op/s 2024-03-28T18:45:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:46 smithi119 ceph-mon[31813]: pgmap v54: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 85 MiB/s wr, 37 op/s 2024-03-28T18:45:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:48 smithi196 ceph-mon[36300]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-28T18:45:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:48 smithi131 ceph-mon[36065]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-28T18:45:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:48 smithi119 ceph-mon[31813]: pgmap v55: 97 pgs: 97 active+clean; 1.0 GiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-28T18:45:49.323 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:49.323 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 readers = 3592231.00 kB/sec 2024-03-28T18:45:49.323 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 readers = 3583111.50 kB/sec 2024-03-28T18:45:49.323 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 3592231.00 kB/sec 2024-03-28T18:45:49.323 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 3592231.00 kB/sec 2024-03-28T18:45:49.324 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 3592231.00 kB/sec 2024-03-28T18:45:49.324 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:45:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:50 smithi196 ceph-mon[36300]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T18:45:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:50 smithi131 ceph-mon[36065]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T18:45:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:50 smithi119 ceph-mon[31813]: pgmap v56: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-03-28T18:45:51.793 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:51.793 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 re-readers = 2701221.00 kB/sec 2024-03-28T18:45:51.794 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 re-readers = 2696524.13 kB/sec 2024-03-28T18:45:51.794 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 2701221.00 kB/sec 2024-03-28T18:45:51.794 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 2701221.00 kB/sec 2024-03-28T18:45:51.794 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 2701221.00 kB/sec 2024-03-28T18:45:51.794 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:45:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:52 smithi196 ceph-mon[36300]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 112 MiB/s wr, 37 op/s 2024-03-28T18:45:52.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:52 smithi131 ceph-mon[36065]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 112 MiB/s wr, 37 op/s 2024-03-28T18:45:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:52 smithi119 ceph-mon[31813]: pgmap v57: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 112 MiB/s wr, 37 op/s 2024-03-28T18:45:54.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:54 smithi196 ceph-mon[36300]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 96 MiB/s wr, 32 op/s 2024-03-28T18:45:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:54 smithi131 ceph-mon[36065]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 96 MiB/s wr, 32 op/s 2024-03-28T18:45:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:54 smithi119 ceph-mon[31813]: pgmap v58: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 96 MiB/s wr, 32 op/s 2024-03-28T18:45:56.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:56 smithi196 ceph-mon[36300]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 96 MiB/s wr, 37 op/s 2024-03-28T18:45:56.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:56 smithi131 ceph-mon[36065]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 96 MiB/s wr, 37 op/s 2024-03-28T18:45:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:56 smithi119 ceph-mon[31813]: pgmap v59: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 96 MiB/s wr, 37 op/s 2024-03-28T18:45:58.478 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.478 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.479 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.479 INFO:tasks.workunit.client.0.smithi119.stdout:iozone test complete. 2024-03-28T18:45:58.485 INFO:tasks.workunit.client.0.smithi119.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T18:45:58.577 INFO:tasks.workunit.client.0.smithi119.stdout: Iozone: Performance Test of File I/O 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Version $Revision: 3.494 $ 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Compiled for 64 bit mode. 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Build: linux-AMD64 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T18:45:58.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Run began: Thu Mar 28 18:45:58 2024 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Include close in write timing 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Include fsync in write timing 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: File size set to 1048576 kB 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Record Size 1024 kB 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Output is in kBytes/sec 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache line size set to 32 bytes. 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: File stride size set to 17 * record size. 2024-03-28T18:45:58.579 INFO:tasks.workunit.client.0.smithi119.stdout: Throughput test with 1 process 2024-03-28T18:45:58.580 INFO:tasks.workunit.client.0.smithi119.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-03-28T18:45:58.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:45:58 smithi131 ceph-mon[36065]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:45:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:45:58 smithi119 ceph-mon[31813]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:45:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:45:58 smithi196 ceph-mon[36300]: pgmap v60: 97 pgs: 97 active+clean; 1.0 GiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:46:00.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:00 smithi131 ceph-mon[36065]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 85 MiB/s wr, 57 op/s 2024-03-28T18:46:00.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:00 smithi119 ceph-mon[31813]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 85 MiB/s wr, 57 op/s 2024-03-28T18:46:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:00 smithi196 ceph-mon[36300]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 85 MiB/s wr, 57 op/s 2024-03-28T18:46:02.083 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:02.083 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 initial writers = 302195.66 kB/sec 2024-03-28T18:46:02.083 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 initial writers = 302126.70 kB/sec 2024-03-28T18:46:02.083 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 302195.66 kB/sec 2024-03-28T18:46:02.084 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 302195.66 kB/sec 2024-03-28T18:46:02.084 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 302195.66 kB/sec 2024-03-28T18:46:02.084 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:46:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:02 smithi131 ceph-mon[36065]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:02 smithi119 ceph-mon[31813]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:02 smithi196 ceph-mon[36300]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:04 smithi119 ceph-mon[31813]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:04 smithi196 ceph-mon[36300]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:04.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:46:04.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:46:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:04 smithi131 ceph-mon[36065]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.9 KiB/s wr, 32 op/s 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 -- 172.21.15.119:0/3253391151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd04101530 msgr2=0x7fbd041019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3253391151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd04101530 0x7fbd041019b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009930 tx=0x7fbcf402bc20 comp rx=0 tx=0).stop 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 -- 172.21.15.119:0/3253391151 shutdown_connections 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3253391151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101ef0 0x7fbd0410e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3253391151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd04101530 0x7fbd041019b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3253391151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd041075f0 0x7fbd041079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 -- 172.21.15.119:0/3253391151 >> 172.21.15.119:0/3253391151 conn(0x7fbd040fd000 msgr2=0x7fbd040ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 -- 172.21.15.119:0/3253391151 shutdown_connections 2024-03-28T18:46:05.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.169+0000 7fbd0a000640 1 -- 172.21.15.119:0/3253391151 wait complete. 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 Processor -- start 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 -- start start 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 0x7fbd041982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd041075f0 0x7fbd0419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041118b0 con 0x7fbd04101530 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04111730 con 0x7fbd04101ef0 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.170+0000 7fbd0a000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04111a30 con 0x7fbd041075f0 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 0x7fbd041982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50740/0 (socket says 172.21.15.119:50740) 2024-03-28T18:46:05.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbd08c00640 1 -- 172.21.15.119:0/3315167511 learned_addr learned my addr 172.21.15.119:0/3315167511 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbd09600640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd041075f0 0x7fbd0419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3315167511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd041075f0 msgr2=0x7fbd0419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd041075f0 0x7fbd0419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 msgr2=0x7fbd04197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3315167511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0419cd80 con 0x7fbd04101ef0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.171+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 0x7fbd041982d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009230 tx=0x7fbcf402cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.172+0000 7fbd02000640 1 -- 172.21.15.119:0/3315167511 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4041070 con 0x7fbd04101ef0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.172+0000 7fbd02000640 1 -- 172.21.15.119:0/3315167511 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcf4004450 con 0x7fbd04101ef0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.172+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0419d010 con 0x7fbd04101ef0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.172+0000 7fbd02000640 1 -- 172.21.15.119:0/3315167511 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4003c60 con 0x7fbd04101ef0 2024-03-28T18:46:05.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.172+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd041a4900 con 0x7fbd04101ef0 2024-03-28T18:46:05.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 Processor -- start 2024-03-28T18:46:05.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd02000640 1 -- 172.21.15.119:0/3315167511 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbcf4044070 con 0x7fbd04101ef0 2024-03-28T18:46:05.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 -- start start 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb9870 0x559d4febbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4fe10250 con 0x559d4feb0500 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4fdb8a30 con 0x559d4feb9870 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.175+0000 7fbd0e207740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4fe0c2f0 con 0x559d4feb11e0 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59126/0 (socket says 172.21.15.119:59126) 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 -- 172.21.15.119:0/3944285572 learned_addr learned my addr 172.21.15.119:0/3944285572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb9870 0x559d4febbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 -- 172.21.15.119:0/3944285572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb9870 msgr2=0x559d4febbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb9870 0x559d4febbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 -- 172.21.15.119:0/3944285572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 msgr2=0x559d4feb0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 -- 172.21.15.119:0/3944285572 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d4fda3690 con 0x559d4feb11e0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd09600640 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009fd0 tx=0x7fbcfc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be010c0af579ec44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbd02000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 0x7fbce40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.176+0000 7fbcf9c00640 1 -- 172.21.15.119:0/3944285572 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004d20 con 0x559d4feb11e0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbcf9c00640 1 -- 172.21.15.119:0/3944285572 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc004050 con 0x559d4feb11e0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbcf9c00640 1 -- 172.21.15.119:0/3944285572 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc005280 con 0x559d4feb11e0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd02000640 1 -- 172.21.15.119:0/3315167511 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbcf40f0730 con 0x7fbd04101ef0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 0x7fbce40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 -- 172.21.15.119:0/3944285572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 msgr2=0x559d4feaff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009fd0 tx=0x7fbcfc0118d0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 -- 172.21.15.119:0/3944285572 shutdown_connections 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb9870 0x559d4febbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3944285572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feaff30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 -- 172.21.15.119:0/3944285572 >> 172.21.15.119:0/3944285572 conn(0x559d4fe73200 msgr2=0x559d4feaaf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 -- 172.21.15.119:0/3944285572 shutdown_connections 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd0e207740 1 -- 172.21.15.119:0/3944285572 wait complete. 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.177+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 0x7fbce40a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x559d4feb1730 tx=0x7fbcec005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.178+0000 7fbd0e207740 1 -- start start 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.178+0000 7fbd0e207740 1 Processor -- start 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4feb67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4febc6e0 con 0x559d4feb9870 2024-03-28T18:46:05.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4fdcbe80 con 0x559d4feb0500 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4febc860 con 0x559d4feb11e0 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd0e207740 10 client.0 _ll_register_callbacks cb 0x559d4fdde130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37046/0 (socket says 172.21.15.119:37046) 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 -- 172.21.15.119:0/1348724374 learned_addr learned my addr 172.21.15.119:0/1348724374 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd09600640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4feb67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 -- 172.21.15.119:0/1348724374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 msgr2=0x559d4feb62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 -- 172.21.15.119:0/1348724374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 msgr2=0x559d4feb67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4feb67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbd08c00640 1 -- 172.21.15.119:0/1348724374 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d4fbee7d0 con 0x559d4feb0500 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.179+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:05.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbd08c00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00f840 tx=0x7fbcec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0559ee3c9126291 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbcf2000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0119c0 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcfc004050 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbcec011b60 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcec011d60 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbd0e207740 10 client.25955 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d4fb243f0 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.180+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc004050 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbcec027070 con 0x559d4feb0500 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 10 client.25955 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 10 client.25955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 20 client.25955 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4fec9760 0x559d4fecbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 4 client.25955 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d4fe1d790 con 0x559d4fec9760 2024-03-28T18:46:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4fec9760 0x559d4fecbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbcf2000640 10 client.25955 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.181+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4fec9760 0x559d4fecbb90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5955a5d7a6a0daac server_cookie=9f7806f88445de19 in_seq=0 out_seq=0 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbcf2000640 1 -- 172.21.15.119:0/1348724374 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d4fe1d790 con 0x559d4fec9760 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbcf2000640 10 client.25955 handle_command_reply: tid=0 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 client.25955 shutdown 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 msgr2=0x559d4feb8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00f840 tx=0x7fbcec004730 comp rx=0 tx=0).stop 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 shutdown_connections 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4fec9760 0x559d4fecbb90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4feb67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbcf2000640 0 client.25955 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 --2- 172.21.15.119:0/1348724374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb8f10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.182+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 >> 172.21.15.119:0/1348724374 conn(0x559d4fe73200 msgr2=0x559d4feacca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.183+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 shutdown_connections 2024-03-28T18:46:05.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.183+0000 7fbd0e207740 1 -- 172.21.15.119:0/1348724374 wait complete. 2024-03-28T18:46:05.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.183+0000 7fbd0e207740 20 client.25955 trim_cache size 0 max 16384 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 Processor -- start 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 -- start start 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4febac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041aaf40 con 0x559d4feb9870 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ac8c0 con 0x559d4feb0500 2024-03-28T18:46:05.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.200+0000 7fbd0e207740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ab5a0 con 0x559d4feb11e0 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbd09600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4febac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37060/0 (socket says 172.21.15.119:37060) 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59132/0 (socket says 172.21.15.119:59132) 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbd08c00640 1 -- 172.21.15.119:0/790524374 learned_addr learned my addr 172.21.15.119:0/790524374 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 -- 172.21.15.119:0/790524374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 msgr2=0x559d4feb89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 -- 172.21.15.119:0/790524374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 msgr2=0x559d4febac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4febac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 -- 172.21.15.119:0/790524374 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d4fae40f0 con 0x559d4feb11e0 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbd08c00640 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40eded0 tx=0x7fbcf40f2b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8d0bb46dc2e945a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.201+0000 7fbcf9200640 1 -- 172.21.15.119:0/790524374 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf404b040 con 0x559d4feb11e0 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbcf9200640 1 -- 172.21.15.119:0/790524374 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcf4004c80 con 0x559d4feb11e0 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbcf9200640 1 -- 172.21.15.119:0/790524374 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf40c5580 con 0x559d4feb11e0 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 -- 172.21.15.119:0/790524374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 msgr2=0x559d4feb8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40eded0 tx=0x7fbcf40f2b80 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 -- 172.21.15.119:0/790524374 shutdown_connections 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb9870 0x559d4febac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb11e0 0x559d4feb8f10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 --2- 172.21.15.119:0/790524374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb0500 0x559d4feb89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 -- 172.21.15.119:0/790524374 >> 172.21.15.119:0/790524374 conn(0x559d4fe73200 msgr2=0x559d4feaaa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 -- 172.21.15.119:0/790524374 shutdown_connections 2024-03-28T18:46:05.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.202+0000 7fbd0e207740 1 -- 172.21.15.119:0/790524374 wait complete. 2024-03-28T18:46:05.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 -- start start 2024-03-28T18:46:05.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 Processor -- start 2024-03-28T18:46:05.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 0x559d4feb25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb9870 0x559d4feb2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ac260 con 0x559d4feb0500 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041abc00 con 0x559d4feb11e0 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041a9100 con 0x559d4feb9870 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd0e207740 10 client.0 _ll_register_callbacks cb 0x7fbcec012050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.203+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 0x559d4feb25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbd09600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb9870 0x559d4feb2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbd08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50782/0 (socket says 172.21.15.119:50782) 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbd08c00640 1 -- 172.21.15.119:0/3309852856 learned_addr learned my addr 172.21.15.119:0/3309852856 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3309852856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb9870 msgr2=0x559d4feb2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb9870 0x559d4feb2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3309852856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 msgr2=0x559d4feb3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 -- 172.21.15.119:0/3309852856 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d4feb31e0 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcfbe00640 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 0x559d4feb25b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40b92a0 tx=0x7fbcf40f2bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58c8f5687fe0da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcf2000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf404b040 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbcf40ee040 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcf4004c80 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcf40bc9d0 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.204+0000 7fbd0e207740 10 client.25961 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d4ff24f10 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcf4004c80 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbcf40bc2e0 con 0x559d4feb11e0 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 10 client.25961 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 20 client.25961 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4ff25600 0x559d4ff27a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 4 client.25961 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559d4fe17ae0 con 0x559d4ff25600 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.205+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4ff25600 0x559d4ff27a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbcf2000640 10 client.25961 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd08c00640 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4ff25600 0x559d4ff27a20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eed32b09183049b server_cookie=386dec370fa72a14 in_seq=0 out_seq=0 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbcf2000640 1 -- 172.21.15.119:0/3309852856 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559d4fe17ae0 con 0x559d4ff25600 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbcf2000640 10 client.25961 handle_command_reply: tid=0 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 client.25961 shutdown 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 msgr2=0x559d4feb25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 0x559d4feb25b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40b92a0 tx=0x7fbcf40f2bf0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 shutdown_connections 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559d4ff25600 0x559d4ff27a20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d4feb9870 0x559d4feb2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d4feb11e0 0x559d4feb25b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbcf2000640 0 client.25961 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 --2- 172.21.15.119:0/3309852856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d4feb0500 0x559d4feb3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 >> 172.21.15.119:0/3309852856 conn(0x559d4fe73200 msgr2=0x559d4feb4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 shutdown_connections 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 1 -- 172.21.15.119:0/3309852856 wait complete. 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.206+0000 7fbd0e207740 20 client.25961 trim_cache size 0 max 16384 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:46:05.211 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 msgr2=0x7fbce40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 0x7fbce40a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x559d4feb1730 tx=0x7fbcec005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 msgr2=0x7fbd041982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 0x7fbd041982d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009230 tx=0x7fbcf402cb20 comp rx=0 tx=0).stop 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 shutdown_connections 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbce40a29f0 0x7fbce40a4eb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd041075f0 0x7fbd0419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbd04101ef0 0x7fbd041982d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 --2- 172.21.15.119:0/3315167511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd04101530 0x7fbd04197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 >> 172.21.15.119:0/3315167511 conn(0x7fbd040fd000 msgr2=0x7fbd040fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 shutdown_connections 2024-03-28T18:46:05.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.207+0000 7fbd0a000640 1 -- 172.21.15.119:0/3315167511 wait complete. 2024-03-28T18:46:05.225 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:46:05.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:46:05.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 -- 172.21.15.119:0/2719450198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1016c0 msgr2=0x7f0a5c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 --2- 172.21.15.119:0/2719450198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c101b40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009930 tx=0x7f0a4402bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 -- 172.21.15.119:0/2719450198 shutdown_connections 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 --2- 172.21.15.119:0/2719450198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c102080 0x7f0a5c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 --2- 172.21.15.119:0/2719450198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c101b40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 --2- 172.21.15.119:0/2719450198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 -- 172.21.15.119:0/2719450198 >> 172.21.15.119:0/2719450198 conn(0x7f0a5c0fd170 msgr2=0x7f0a5c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 -- 172.21.15.119:0/2719450198 shutdown_connections 2024-03-28T18:46:05.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.442+0000 7f0a62200640 1 -- 172.21.15.119:0/2719450198 wait complete. 2024-03-28T18:46:05.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.443+0000 7f0a62200640 1 Processor -- start 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.443+0000 7f0a62200640 1 -- start start 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c102080 0x7f0a5c073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c111b30 con 0x7f0a5c102080 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c1119b0 con 0x7f0a5c1016c0 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a62200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c111cb0 con 0x7f0a5c1076d0 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a5b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a5be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59158/0 (socket says 172.21.15.119:59158) 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a5be00640 1 -- 172.21.15.119:0/876166850 learned_addr learned my addr 172.21.15.119:0/876166850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a5b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37086/0 (socket says 172.21.15.119:37086) 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.444+0000 7f0a53e00640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c102080 0x7f0a5c073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 -- 172.21.15.119:0/876166850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 msgr2=0x7f0a5c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 -- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c102080 msgr2=0x7f0a5c073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c102080 0x7f0a5c073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 -- 172.21.15.119:0/876166850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c1a64d0 con 0x7f0a5c1016c0 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5be00640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:05.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a5b400640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005fd0 tx=0x7f0a4c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c009c30 con 0x7f0a5c1016c0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a4c009dd0 con 0x7f0a5c1016c0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5c1a67c0 con 0x7f0a5c1016c0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c010460 con 0x7f0a5c1016c0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.445+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5c1a6d30 con 0x7f0a5c1016c0 2024-03-28T18:46:05.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.447+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5c067c00 con 0x7f0a5c1016c0 2024-03-28T18:46:05.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.448+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f0a4c003f50 con 0x7f0a5c1016c0 2024-03-28T18:46:05.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.451+0000 7f0a58c00640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 0x7f0a400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:05.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.451+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0a4c0ce970 con 0x7f0a5c1016c0 2024-03-28T18:46:05.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.451+0000 7f0a53e00640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 0x7f0a400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:05.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.452+0000 7f0a53e00640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 0x7f0a400a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009900 tx=0x7f0a4402df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:05.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.455+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a4c017020 con 0x7f0a5c1016c0 2024-03-28T18:46:05.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.716+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a5c109c30 con 0x7f0a5c1016c0 2024-03-28T18:46:05.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.717+0000 7f0a58c00640 1 -- 172.21.15.119:0/876166850 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f0a4c09afc0 con 0x7f0a5c1016c0 2024-03-28T18:46:05.717 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:46:05.717 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:46:05.717 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:46:05.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.719+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 msgr2=0x7f0a400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.719+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 0x7f0a400a4eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a44009900 tx=0x7f0a4402df90 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.719+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 msgr2=0x7f0a5c0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.719+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005fd0 tx=0x7f0a4c00dde0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 shutdown_connections 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0a400a29f0 0x7f0a400a4eb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a5c1076d0 0x7f0a5c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a5c102080 0x7f0a5c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 --2- 172.21.15.119:0/876166850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0a5c1016c0 0x7f0a5c0739a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 >> 172.21.15.119:0/876166850 conn(0x7f0a5c0fd170 msgr2=0x7f0a5c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 shutdown_connections 2024-03-28T18:46:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:05.720+0000 7f0a62200640 1 -- 172.21.15.119:0/876166850 wait complete. 2024-03-28T18:46:05.738 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:05.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 -- 172.21.15.119:0/336600228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900107640 msgr2=0x7f4900107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 --2- 172.21.15.119:0/336600228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900107640 0x7f4900107a20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009900 tx=0x7f48f402bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 -- 172.21.15.119:0/336600228 shutdown_connections 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 --2- 172.21.15.119:0/336600228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900101ff0 0x7f490010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 --2- 172.21.15.119:0/336600228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101630 0x7f4900101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 --2- 172.21.15.119:0/336600228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900107640 0x7f4900107a20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 -- 172.21.15.119:0/336600228 >> 172.21.15.119:0/336600228 conn(0x7f4900078aa0 msgr2=0x7f4900078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 -- 172.21.15.119:0/336600228 shutdown_connections 2024-03-28T18:46:06.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.043+0000 7f4907600640 1 -- 172.21.15.119:0/336600228 wait complete. 2024-03-28T18:46:06.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.044+0000 7f4907600640 1 Processor -- start 2024-03-28T18:46:06.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.044+0000 7f4907600640 1 -- start start 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 0x7f490019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900107640 0x7f490019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4900111a50 con 0x7f4900107640 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001118d0 con 0x7f4900101630 2024-03-28T18:46:06.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4907600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4900111bd0 con 0x7f4900101ff0 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4904c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4904c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37118/0 (socket says 172.21.15.119:37118) 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4904c00640 1 -- 172.21.15.119:0/2090052039 learned_addr learned my addr 172.21.15.119:0/2090052039 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f4905600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900107640 0x7f490019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.045+0000 7f48ffe00640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 0x7f490019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 -- 172.21.15.119:0/2090052039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 msgr2=0x7f490019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 0x7f490019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 -- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900107640 msgr2=0x7f490019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900107640 0x7f490019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 -- 172.21.15.119:0/2090052039 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490019f0e0 con 0x7f4900101630 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f48ffe00640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 0x7f490019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.046+0000 7f4904c00640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f48f4005940 tx=0x7f48f40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.047+0000 7f48fd600640 1 -- 172.21.15.119:0/2090052039 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4041070 con 0x7f4900101630 2024-03-28T18:46:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.047+0000 7f48fd600640 1 -- 172.21.15.119:0/2090052039 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f402ccc0 con 0x7f4900101630 2024-03-28T18:46:06.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.047+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490019f370 con 0x7f4900101630 2024-03-28T18:46:06.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.047+0000 7f48fd600640 1 -- 172.21.15.119:0/2090052039 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4003d80 con 0x7f4900101630 2024-03-28T18:46:06.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.047+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49001a6c60 con 0x7f4900101630 2024-03-28T18:46:06.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.052+0000 7f490a39c740 1 Processor -- start 2024-03-28T18:46:06.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 -- start start 2024-03-28T18:46:06.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68deab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b68422c0 con 0x55f5b68df080 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b67eac90 con 0x55f5b68de6d0 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.053+0000 7f490a39c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b683e360 con 0x55f5b68e7c00 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.054+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.054+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59196/0 (socket says 172.21.15.119:59196) 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.054+0000 7f48ffe00640 1 -- 172.21.15.119:0/1187246150 learned_addr learned my addr 172.21.15.119:0/1187246150 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.054+0000 7f4905600640 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68deab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.055+0000 7f48ffe00640 1 -- 172.21.15.119:0/1187246150 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 msgr2=0x55f5b68deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.055+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68deab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.055+0000 7f48ffe00640 1 -- 172.21.15.119:0/1187246150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 msgr2=0x55f5b68e7630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:06.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.055+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.055+0000 7f48ffe00640 1 -- 172.21.15.119:0/1187246150 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5b680f080 con 0x55f5b68e7c00 2024-03-28T18:46:06.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.057+0000 7f48fd600640 1 -- 172.21.15.119:0/2090052039 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f48f4044070 con 0x7f4900101630 2024-03-28T18:46:06.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.057+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009c00 tx=0x7f48ec010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c30d834e4c365f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.061+0000 7f48fd600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 0x7f48e80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.061+0000 7f48f1c00640 1 -- 172.21.15.119:0/1187246150 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec004030 con 0x55f5b68e7c00 2024-03-28T18:46:06.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.061+0000 7f48f1c00640 1 -- 172.21.15.119:0/1187246150 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48ec0041d0 con 0x55f5b68e7c00 2024-03-28T18:46:06.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.061+0000 7f48f1c00640 1 -- 172.21.15.119:0/1187246150 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec012670 con 0x55f5b68e7c00 2024-03-28T18:46:06.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.062+0000 7f48fd600640 1 -- 172.21.15.119:0/2090052039 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f48f40f05b0 con 0x7f4900101630 2024-03-28T18:46:06.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.062+0000 7f4905600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 0x7f48e80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 -- 172.21.15.119:0/1187246150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 msgr2=0x55f5b68ea050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009c00 tx=0x7f48ec010740 comp rx=0 tx=0).stop 2024-03-28T18:46:06.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f4905600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 0x7f48e80a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48f80092f0 tx=0x7f48f8009280 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 -- 172.21.15.119:0/1187246150 shutdown_connections 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68ea050 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 --2- 172.21.15.119:0/1187246150 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68deab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.065+0000 7f490a39c740 1 -- 172.21.15.119:0/1187246150 >> 172.21.15.119:0/1187246150 conn(0x55f5b68d8da0 msgr2=0x55f5b68d9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.066+0000 7f490a39c740 1 -- 172.21.15.119:0/1187246150 shutdown_connections 2024-03-28T18:46:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.066+0000 7f490a39c740 1 -- 172.21.15.119:0/1187246150 wait complete. 2024-03-28T18:46:06.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.066+0000 7f490a39c740 1 -- start start 2024-03-28T18:46:06.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.067+0000 7f490a39c740 1 Processor -- start 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.069+0000 7f490a39c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.069+0000 7f490a39c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68df080 0x55f5b68e4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.069+0000 7f490a39c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f4904c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f4904c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59198/0 (socket says 172.21.15.119:59198) 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f490a39c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b68eabc0 con 0x55f5b68e7c00 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f490a39c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b67fe0d0 con 0x55f5b68df080 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f490a39c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5b68ead40 con 0x55f5b68de6d0 2024-03-28T18:46:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f4904c00640 1 -- 172.21.15.119:0/292189849 learned_addr learned my addr 172.21.15.119:0/292189849 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.070+0000 7f490a39c740 10 client.0 _ll_register_callbacks cb 0x55f5b6802e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:06.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.071+0000 7f48ffe00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68df080 0x55f5b68e4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.071+0000 7f4905600640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4904c00640 1 -- 172.21.15.119:0/292189849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68df080 msgr2=0x55f5b68e4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4904c00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68df080 0x55f5b68e4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4904c00640 1 -- 172.21.15.119:0/292189849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 msgr2=0x55f5b68e4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4904c00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68e4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4904c00640 1 -- 172.21.15.119:0/292189849 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5b65563f0 con 0x55f5b68de6d0 2024-03-28T18:46:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.072+0000 7f4905600640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.073+0000 7f4904c00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f48f4000c00 tx=0x7f48f40b9020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ad9d6a27be6d21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.073+0000 7f48e6000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:06.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.074+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f404b040 con 0x55f5b68de6d0 2024-03-28T18:46:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.074+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f48f40c4fd0 con 0x55f5b68de6d0 2024-03-28T18:46:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.074+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48ec0041d0 con 0x55f5b68de6d0 2024-03-28T18:46:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.075+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f4004c80 con 0x55f5b68de6d0 2024-03-28T18:46:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.075+0000 7f490a39c740 10 client.25996 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.075+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f5b66207d0 con 0x55f5b68de6d0 2024-03-28T18:46:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.077+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48ec0041d0 con 0x55f5b68de6d0 2024-03-28T18:46:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f48f40db290 con 0x55f5b68de6d0 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 10 client.25996 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 10 client.25996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 20 client.25996 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b68f9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 4 client.25996 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.078+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5b684f800 con 0x55f5b68f7a30 2024-03-28T18:46:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.079+0000 7f48ffe00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b68f9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.079+0000 7f48e6000640 10 client.25996 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.079+0000 7f48ffe00640 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b68f9e40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9cc68432e70abc19 server_cookie=8d619f0d529e3ef8 in_seq=0 out_seq=0 2024-03-28T18:46:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.081+0000 7f48e6000640 1 -- 172.21.15.119:0/292189849 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5b684f800 con 0x55f5b68f7a30 2024-03-28T18:46:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.081+0000 7f48e6000640 10 client.25996 handle_command_reply: tid=0 2024-03-28T18:46:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.081+0000 7f490a39c740 1 client.25996 shutdown 2024-03-28T18:46:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.082+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 msgr2=0x55f5b68e7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.082+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f48f4000c00 tx=0x7f48f40b9020 comp rx=0 tx=0).stop 2024-03-28T18:46:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 shutdown_connections 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b68f9e40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f48e6000640 0 client.25996 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68df080 0x55f5b68e4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 --2- 172.21.15.119:0/292189849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7580 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 >> 172.21.15.119:0/292189849 conn(0x55f5b68d8da0 msgr2=0x55f5b68daaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 shutdown_connections 2024-03-28T18:46:06.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.083+0000 7f490a39c740 1 -- 172.21.15.119:0/292189849 wait complete. 2024-03-28T18:46:06.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.084+0000 7f490a39c740 20 client.25996 trim_cache size 0 max 16384 2024-03-28T18:46:06.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.113+0000 7f490a39c740 1 Processor -- start 2024-03-28T18:46:06.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 -- start start 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001af5c0 con 0x55f5b68df080 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001b0f40 con 0x55f5b68de6d0 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.114+0000 7f490a39c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001afc20 con 0x55f5b68e7c00 2024-03-28T18:46:06.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.115+0000 7f4904c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.115+0000 7f4905600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.116+0000 7f4905600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59214/0 (socket says 172.21.15.119:59214) 2024-03-28T18:46:06.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.116+0000 7f4905600640 1 -- 172.21.15.119:0/4068247752 learned_addr learned my addr 172.21.15.119:0/4068247752 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.116+0000 7f4904c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37136/0 (socket says 172.21.15.119:37136) 2024-03-28T18:46:06.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.117+0000 7f4905600640 1 -- 172.21.15.119:0/4068247752 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 msgr2=0x55f5b68e7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.117+0000 7f48ffe00640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f4905600640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f4905600640 1 -- 172.21.15.119:0/4068247752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 msgr2=0x55f5b68e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f4905600640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f4905600640 1 -- 172.21.15.119:0/4068247752 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5b68eace0 con 0x55f5b68e7c00 2024-03-28T18:46:06.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f48ffe00640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:06.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.119+0000 7f4904c00640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:06.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.120+0000 7f4905600640 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009470 tx=0x7f48f8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0ca3c13373c945 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.121+0000 7f48f1200640 1 -- 172.21.15.119:0/4068247752 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f801e770 con 0x55f5b68e7c00 2024-03-28T18:46:06.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.121+0000 7f48f1200640 1 -- 172.21.15.119:0/4068247752 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f801edf0 con 0x55f5b68e7c00 2024-03-28T18:46:06.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.121+0000 7f48f1200640 1 -- 172.21.15.119:0/4068247752 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f801c410 con 0x55f5b68e7c00 2024-03-28T18:46:06.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.122+0000 7f490a39c740 1 -- 172.21.15.119:0/4068247752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 msgr2=0x55f5b68e1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.122+0000 7f490a39c740 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009470 tx=0x7f48f8014000 comp rx=0 tx=0).stop 2024-03-28T18:46:06.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 -- 172.21.15.119:0/4068247752 shutdown_connections 2024-03-28T18:46:06.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68e7c00 0x55f5b68e1b70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68df080 0x55f5b68e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 --2- 172.21.15.119:0/4068247752 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b68e7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 -- 172.21.15.119:0/4068247752 >> 172.21.15.119:0/4068247752 conn(0x55f5b68d8da0 msgr2=0x55f5b68d7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 -- 172.21.15.119:0/4068247752 shutdown_connections 2024-03-28T18:46:06.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.123+0000 7f490a39c740 1 -- 172.21.15.119:0/4068247752 wait complete. 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.124+0000 7f490a39c740 1 -- start start 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.124+0000 7f490a39c740 1 Processor -- start 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.124+0000 7f490a39c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b694bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.125+0000 7f490a39c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.125+0000 7f490a39c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68ee140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.125+0000 7f490a39c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001b08e0 con 0x55f5b68e7c00 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.125+0000 7f490a39c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001b0280 con 0x55f5b68de6d0 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.125+0000 7f490a39c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001ad780 con 0x55f5b68df080 2024-03-28T18:46:06.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f48ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59222/0 (socket says 172.21.15.119:59222) 2024-03-28T18:46:06.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f48ffe00640 1 -- 172.21.15.119:0/1458506118 learned_addr learned my addr 172.21.15.119:0/1458506118 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f4904c00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b694bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f4905600640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68ee140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.126+0000 7f490a39c740 10 client.0 _ll_register_callbacks cb 0x7f48f4003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 -- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 msgr2=0x55f5b694c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 -- 172.21.15.119:0/1458506118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 msgr2=0x55f5b68ee140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68ee140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 -- 172.21.15.119:0/1458506118 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5b68ee780 con 0x55f5b68de6d0 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f4904c00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b694bf10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f48f40bc9c0 tx=0x7f48f402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b0aa510c38bc74e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.127+0000 7f48e6000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40a0060 con 0x55f5b68de6d0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f48f40c4fd0 con 0x55f5b68de6d0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f801edf0 con 0x55f5b68de6d0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f4004c80 con 0x55f5b68de6d0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f490a39c740 10 client.25985 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.128+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f5b6964b10 con 0x55f5b68de6d0 2024-03-28T18:46:06.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f801edf0 con 0x55f5b68de6d0 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f48f409bae0 con 0x55f5b68de6d0 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 10 client.25985 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 10 client.25985 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 20 client.25985 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b6967040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 4 client.25985 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.129+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f5b6849b50 con 0x55f5b68f7a30 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.130+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b6967040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.130+0000 7f48ffe00640 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b6967040 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6e860ade8a50b96e server_cookie=6e9a592f7442cfed in_seq=0 out_seq=0 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.130+0000 7f48e6000640 10 client.25985 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f48e6000640 1 -- 172.21.15.119:0/1458506118 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f5b6849b50 con 0x55f5b68f7a30 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f48e6000640 10 client.25985 handle_command_reply: tid=0 2024-03-28T18:46:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 client.25985 shutdown 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 msgr2=0x55f5b694bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b694bf10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f48f40bc9c0 tx=0x7f48f402bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 shutdown_connections 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f5b68f7a30 0x55f5b6967040 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5b68e7c00 0x55f5b68ee140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f5b68df080 0x55f5b694c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 --2- 172.21.15.119:0/1458506118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f5b68de6d0 0x55f5b694bf10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f48e6000640 0 client.25985 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:06.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.131+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 >> 172.21.15.119:0/1458506118 conn(0x55f5b68d8da0 msgr2=0x55f5b68df460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.132+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 shutdown_connections 2024-03-28T18:46:06.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.132+0000 7f490a39c740 1 -- 172.21.15.119:0/1458506118 wait complete. 2024-03-28T18:46:06.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.132+0000 7f490a39c740 20 client.25985 trim_cache size 0 max 16384 2024-03-28T18:46:06.133 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 msgr2=0x7f48e80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 0x7f48e80a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f48f80092f0 tx=0x7f48f8009280 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 msgr2=0x7f490019a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f48f4005940 tx=0x7f48f40042f0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 shutdown_connections 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48e80a29f0 0x7f48e80a4eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4900107640 0x7f490019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4900101ff0 0x7f490019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 --2- 172.21.15.119:0/2090052039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4900101630 0x7f490019a140 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.133+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 >> 172.21.15.119:0/2090052039 conn(0x7f4900078aa0 msgr2=0x7f4900103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.134+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 shutdown_connections 2024-03-28T18:46:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.134+0000 7f4907600640 1 -- 172.21.15.119:0/2090052039 wait complete. 2024-03-28T18:46:06.153 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:06.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:46:06.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 -- 172.21.15.119:0/937647091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac201084e0 msgr2=0x7fac20108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 --2- 172.21.15.119:0/937647091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac201084e0 0x7fac20108960 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009860 tx=0x7fac0c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 -- 172.21.15.119:0/937647091 shutdown_connections 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 --2- 172.21.15.119:0/937647091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac20114ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 --2- 172.21.15.119:0/937647091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac201084e0 0x7fac20108960 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 --2- 172.21.15.119:0/937647091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2010dde0 0x7fac2010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.528+0000 7fac27e00640 1 -- 172.21.15.119:0/937647091 >> 172.21.15.119:0/937647091 conn(0x7fac20077400 msgr2=0x7fac20079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.529+0000 7fac27e00640 1 -- 172.21.15.119:0/937647091 shutdown_connections 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.529+0000 7fac27e00640 1 -- 172.21.15.119:0/937647091 wait complete. 2024-03-28T18:46:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 Processor -- start 2024-03-28T18:46:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 -- start start 2024-03-28T18:46:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac201a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac2010dde0 0x7fac201a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 0x7fac2019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20118130 con 0x7fac20108ea0 2024-03-28T18:46:06.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac20117fb0 con 0x7fac2019e6e0 2024-03-28T18:46:06.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.530+0000 7fac27e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201182b0 con 0x7fac2010dde0 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 0x7fac2019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 0x7fac2019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37168/0 (socket says 172.21.15.119:37168) 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25e00640 1 -- 172.21.15.119:0/1694536394 learned_addr learned my addr 172.21.15.119:0/1694536394 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac201a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac24a00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac2010dde0 0x7fac201a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 -- 172.21.15.119:0/1694536394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac2010dde0 msgr2=0x7fac201a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac2010dde0 0x7fac201a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 -- 172.21.15.119:0/1694536394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 msgr2=0x7fac2019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 0x7fac2019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.531+0000 7fac25400640 1 -- 172.21.15.119:0/1694536394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2019f2e0 con 0x7fac20108ea0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.532+0000 7fac25400640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac201a4620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac1000e790 tx=0x7fac1000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.533+0000 7fac1e000640 1 -- 172.21.15.119:0/1694536394 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000eea0 con 0x7fac20108ea0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.533+0000 7fac1e000640 1 -- 172.21.15.119:0/1694536394 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac1000ccc0 con 0x7fac20108ea0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.533+0000 7fac1e000640 1 -- 172.21.15.119:0/1694536394 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10018840 con 0x7fac20108ea0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.533+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac201ab3c0 con 0x7fac20108ea0 2024-03-28T18:46:06.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.533+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac201ab770 con 0x7fac20108ea0 2024-03-28T18:46:06.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.543+0000 7fac2ac1f740 1 Processor -- start 2024-03-28T18:46:06.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.543+0000 7fac2ac1f740 1 -- start start 2024-03-28T18:46:06.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.544+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.544+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479c4080 0x555a479cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.544+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479ccc00 0x555a479cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.544+0000 7fac25e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a479272c0 con 0x555a479ccc00 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a478cfc90 con 0x555a479c36d0 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a47923360 con 0x555a479c4080 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37182/0 (socket says 172.21.15.119:37182) 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 -- 172.21.15.119:0/946699427 learned_addr learned my addr 172.21.15.119:0/946699427 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 -- 172.21.15.119:0/946699427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479c4080 msgr2=0x555a479cc630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:06.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479c4080 0x555a479cc630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 -- 172.21.15.119:0/946699427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479ccc00 msgr2=0x555a479cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479ccc00 0x555a479cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.545+0000 7fac25e00640 1 -- 172.21.15.119:0/946699427 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a478f4080 con 0x555a479c36d0 2024-03-28T18:46:06.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.546+0000 7fac25e00640 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac18011570 tx=0x7fac18011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7215e2932d4674c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.546+0000 7fac16600640 1 -- 172.21.15.119:0/946699427 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18004b20 con 0x555a479c36d0 2024-03-28T18:46:06.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.547+0000 7fac16600640 1 -- 172.21.15.119:0/946699427 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac18004cc0 con 0x555a479c36d0 2024-03-28T18:46:06.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.547+0000 7fac16600640 1 -- 172.21.15.119:0/946699427 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18005410 con 0x555a479c36d0 2024-03-28T18:46:06.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 -- 172.21.15.119:0/946699427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 msgr2=0x555a479c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac18011570 tx=0x7fac18011a30 comp rx=0 tx=0).stop 2024-03-28T18:46:06.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 -- 172.21.15.119:0/946699427 shutdown_connections 2024-03-28T18:46:06.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479ccc00 0x555a479cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479c4080 0x555a479cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/946699427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c3ab0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.549+0000 7fac2ac1f740 1 -- 172.21.15.119:0/946699427 >> 172.21.15.119:0/946699427 conn(0x555a479bdda0 msgr2=0x555a479be180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.553+0000 7fac1e000640 1 -- 172.21.15.119:0/1694536394 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fac100189e0 con 0x7fac20108ea0 2024-03-28T18:46:06.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.556+0000 7fac1e000640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 0x7fac040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.556+0000 7fac1e000640 1 -- 172.21.15.119:0/1694536394 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fac100ceda0 con 0x7fac20108ea0 2024-03-28T18:46:06.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.558+0000 7fac24a00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 0x7fac040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.561+0000 7fac24a00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 0x7fac040a4ec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac0c02c100 tx=0x7fac0c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.563+0000 7fac2ac1f740 1 -- 172.21.15.119:0/946699427 shutdown_connections 2024-03-28T18:46:06.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.564+0000 7fac2ac1f740 1 -- 172.21.15.119:0/946699427 wait complete. 2024-03-28T18:46:06.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.564+0000 7fac2ac1f740 1 -- start start 2024-03-28T18:46:06.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.565+0000 7fac2ac1f740 1 Processor -- start 2024-03-28T18:46:06.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479cc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a479cfb70 con 0x555a479c9c50 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a478e30d0 con 0x555a479c36d0 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.566+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a479cfcf0 con 0x555a479ccc00 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.567+0000 7fac2ac1f740 10 client.0 _ll_register_callbacks cb 0x555a478e7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.567+0000 7fac24a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.567+0000 7fac25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.567+0000 7fac24a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59260/0 (socket says 172.21.15.119:59260) 2024-03-28T18:46:06.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.567+0000 7fac24a00640 1 -- 172.21.15.119:0/2633047976 learned_addr learned my addr 172.21.15.119:0/2633047976 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac25e00640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac24a00640 1 -- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 msgr2=0x555a479cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac24a00640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac24a00640 1 -- 172.21.15.119:0/2633047976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 msgr2=0x555a479ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac24a00640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.568+0000 7fac24a00640 1 -- 172.21.15.119:0/2633047976 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a4763b3f0 con 0x555a479ccc00 2024-03-28T18:46:06.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.569+0000 7fac24a00640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac0c035ef0 tx=0x7fac0c02bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f93a7facfdd600 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.569+0000 7fac0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:06.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.569+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c04d020 con 0x555a479ccc00 2024-03-28T18:46:06.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.570+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fac0c034850 con 0x555a479ccc00 2024-03-28T18:46:06.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.571+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac18004cc0 con 0x555a479ccc00 2024-03-28T18:46:06.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.572+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac0c039040 con 0x555a479ccc00 2024-03-28T18:46:06.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.572+0000 7fac2ac1f740 10 client.26011 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:06.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.572+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555a477057d0 con 0x555a479ccc00 2024-03-28T18:46:06.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.573+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac18004cc0 con 0x555a479ccc00 2024-03-28T18:46:06.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.575+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fac0c032ac0 con 0x555a479ccc00 2024-03-28T18:46:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 10 client.26011 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 10 client.26011 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:46:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 20 client.26011 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a479edd80 0x555a479f0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 4 client.26011 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.578+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a47934800 con 0x555a479edd80 2024-03-28T18:46:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.579+0000 7fac25400640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a479edd80 0x555a479f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.580+0000 7fac25400640 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a479edd80 0x555a479f0170 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b922b1fa451d6d01 server_cookie=6bdeffde4b664710 in_seq=0 out_seq=0 2024-03-28T18:46:06.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.581+0000 7fac0aa00640 10 client.26011 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:06.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.582+0000 7fac0aa00640 1 -- 172.21.15.119:0/2633047976 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a47934800 con 0x555a479edd80 2024-03-28T18:46:06.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.582+0000 7fac0aa00640 10 client.26011 handle_command_reply: tid=0 2024-03-28T18:46:06.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.582+0000 7fac2ac1f740 1 client.26011 shutdown 2024-03-28T18:46:06.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.586+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 msgr2=0x555a479c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.586+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fac0c035ef0 tx=0x7fac0c02bb70 comp rx=0 tx=0).stop 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.586+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 shutdown_connections 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.586+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a479edd80 0x555a479f0170 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479ccc00 0x555a479c9710 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2633047976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac0aa00640 0 client.26011 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 >> 172.21.15.119:0/2633047976 conn(0x555a479bdda0 msgr2=0x555a479bfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 shutdown_connections 2024-03-28T18:46:06.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2633047976 wait complete. 2024-03-28T18:46:06.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.587+0000 7fac2ac1f740 20 client.26011 trim_cache size 0 max 16384 2024-03-28T18:46:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.637+0000 7fac2ac1f740 1 Processor -- start 2024-03-28T18:46:06.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.639+0000 7fac2ac1f740 1 -- start start 2024-03-28T18:46:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0000e720 con 0x555a479c9c50 2024-03-28T18:46:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac000100a0 con 0x555a479c36d0 2024-03-28T18:46:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.640+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0000ed80 con 0x555a479f5a80 2024-03-28T18:46:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.641+0000 7fac25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.641+0000 7fac25400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.642+0000 7fac25e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59272/0 (socket says 172.21.15.119:59272) 2024-03-28T18:46:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.642+0000 7fac25e00640 1 -- 172.21.15.119:0/2037535875 learned_addr learned my addr 172.21.15.119:0/2037535875 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.642+0000 7fac24a00640 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.643+0000 7fac25e00640 1 -- 172.21.15.119:0/2037535875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 msgr2=0x555a479c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.644+0000 7fac25e00640 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.644+0000 7fac25e00640 1 -- 172.21.15.119:0/2037535875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 msgr2=0x555a479cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.644+0000 7fac25e00640 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.644+0000 7fac25e00640 1 -- 172.21.15.119:0/2037535875 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a479c29f0 con 0x555a479f5a80 2024-03-28T18:46:06.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.644+0000 7fac24a00640 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:06.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.645+0000 7fac25e00640 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fac18011510 tx=0x7fac18009d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fabbb38207213ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.646+0000 7fac15c00640 1 -- 172.21.15.119:0/2037535875 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18027070 con 0x555a479f5a80 2024-03-28T18:46:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.646+0000 7fac15c00640 1 -- 172.21.15.119:0/2037535875 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac1800f5c0 con 0x555a479f5a80 2024-03-28T18:46:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.646+0000 7fac15c00640 1 -- 172.21.15.119:0/2037535875 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18005410 con 0x555a479f5a80 2024-03-28T18:46:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.647+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2037535875 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 msgr2=0x555a479d8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.647+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fac18011510 tx=0x7fac18009d90 comp rx=0 tx=0).stop 2024-03-28T18:46:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2037535875 shutdown_connections 2024-03-28T18:46:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d8490 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/2037535875 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2037535875 >> 172.21.15.119:0/2037535875 conn(0x555a479bdda0 msgr2=0x555a479c44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2037535875 shutdown_connections 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.648+0000 7fac2ac1f740 1 -- 172.21.15.119:0/2037535875 wait complete. 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.649+0000 7fac2ac1f740 1 -- start start 2024-03-28T18:46:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.649+0000 7fac2ac1f740 1 Processor -- start 2024-03-28T18:46:06.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479d49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0000fa40 con 0x555a479c9c50 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0000f3e0 con 0x555a479c36d0 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0000c8e0 con 0x555a479f5a80 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.650+0000 7fac2ac1f740 10 client.0 _ll_register_callbacks cb 0x7fac0c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:06.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.651+0000 7fac24a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.651+0000 7fac24a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50910/0 (socket says 172.21.15.119:50910) 2024-03-28T18:46:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.651+0000 7fac24a00640 1 -- 172.21.15.119:0/4250372901 learned_addr learned my addr 172.21.15.119:0/4250372901 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.651+0000 7fac25e00640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 -- 172.21.15.119:0/4250372901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 msgr2=0x555a479d0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 -- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 msgr2=0x555a479d49e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479d49e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 -- 172.21.15.119:0/4250372901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a479d1760 con 0x555a479c9c50 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac24a00640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac0c032ee0 tx=0x7fac0c034900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13f9ba009094360 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.652+0000 7fac0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.653+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c060070 con 0x555a479c9c50 2024-03-28T18:46:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.653+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fac0c003e00 con 0x555a479c9c50 2024-03-28T18:46:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.653+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac1800f5c0 con 0x555a479c9c50 2024-03-28T18:46:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.653+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac0c06a050 con 0x555a479c9c50 2024-03-28T18:46:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.654+0000 7fac2ac1f740 10 client.15849 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.654+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555a47a4a2e0 con 0x555a479c9c50 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.654+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac1800f5c0 con 0x555a479c9c50 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.654+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fac0c02c510 con 0x555a479c9c50 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 10 client.15849 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 10 client.15849 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 20 client.15849 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a47a4a9a0 0x555a47a4cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 4 client.15849 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.655+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555a4792eb50 con 0x555a47a4a9a0 2024-03-28T18:46:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.656+0000 7fac25400640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a47a4a9a0 0x555a47a4cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.660+0000 7fac25400640 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a47a4a9a0 0x555a47a4cdc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=992e48cff6a5e4cb server_cookie=6eef45ae81d429fa in_seq=0 out_seq=0 2024-03-28T18:46:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.660+0000 7fac0aa00640 10 client.15849 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac0aa00640 1 -- 172.21.15.119:0/4250372901 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555a4792eb50 con 0x555a47a4a9a0 2024-03-28T18:46:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac0aa00640 10 client.15849 handle_command_reply: tid=0 2024-03-28T18:46:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 client.15849 shutdown 2024-03-28T18:46:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 msgr2=0x555a479d4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac0c032ee0 tx=0x7fac0c034900 comp rx=0 tx=0).stop 2024-03-28T18:46:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 shutdown_connections 2024-03-28T18:46:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a479f5a80 0x555a479d0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x555a47a4a9a0 0x555a47a4cdc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a479c9c50 0x555a479d4f20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 --2- 172.21.15.119:0/4250372901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555a479c36d0 0x555a479d49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.662+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 >> 172.21.15.119:0/4250372901 conn(0x555a479bdda0 msgr2=0x555a479f5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.663+0000 7fac0aa00640 0 client.15849 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.663+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 shutdown_connections 2024-03-28T18:46:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.664+0000 7fac2ac1f740 1 -- 172.21.15.119:0/4250372901 wait complete. 2024-03-28T18:46:06.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.664+0000 7fac2ac1f740 20 client.15849 trim_cache size 0 max 16384 2024-03-28T18:46:06.664 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 msgr2=0x7fac040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 0x7fac040a4ec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac0c02c100 tx=0x7fac0c005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 msgr2=0x7fac201a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac201a4620 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fac1000e790 tx=0x7fac1000ec50 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 shutdown_connections 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac040a2a00 0x7fac040a4ec0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac2019e6e0 0x7fac2019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac2010dde0 0x7fac201a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 --2- 172.21.15.119:0/1694536394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac20108ea0 0x7fac201a4620 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.665+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 >> 172.21.15.119:0/1694536394 conn(0x7fac20077400 msgr2=0x7fac2010ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.666+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 shutdown_connections 2024-03-28T18:46:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.666+0000 7fac27e00640 1 -- 172.21.15.119:0/1694536394 wait complete. 2024-03-28T18:46:06.690 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:06.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:46:06.780 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:06 smithi119 ceph-mon[31813]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 85 MiB/s wr, 73 op/s 2024-03-28T18:46:06.780 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:06 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/876166850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:06 smithi196 ceph-mon[36300]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 85 MiB/s wr, 73 op/s 2024-03-28T18:46:06.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:06 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/876166850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:06 smithi131 ceph-mon[36065]: pgmap v64: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 85 MiB/s wr, 73 op/s 2024-03-28T18:46:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:06 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/876166850' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:06.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 -- 172.21.15.119:0/1723873026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea2410deb0 msgr2=0x7fea241102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 --2- 172.21.15.119:0/1723873026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea2410deb0 0x7fea241102e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fea10009860 tx=0x7fea1002bc10 comp rx=0 tx=0).stop 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 -- 172.21.15.119:0/1723873026 shutdown_connections 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 --2- 172.21.15.119:0/1723873026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea2410deb0 0x7fea241102e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 --2- 172.21.15.119:0/1723873026 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea240743b0 0x7fea2410d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 --2- 172.21.15.119:0/1723873026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea24073a90 0x7fea24073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.963+0000 7fea2ae00640 1 -- 172.21.15.119:0/1723873026 >> 172.21.15.119:0/1723873026 conn(0x7fea24078a60 msgr2=0x7fea24078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.964+0000 7fea2ae00640 1 -- 172.21.15.119:0/1723873026 shutdown_connections 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.964+0000 7fea2ae00640 1 -- 172.21.15.119:0/1723873026 wait complete. 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 Processor -- start 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 -- start start 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea240743b0 0x7fea2419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea2410deb0 0x7fea2419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24113c30 con 0x7fea24073a90 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24113ab0 con 0x7fea2410deb0 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.965+0000 7fea2ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24113db0 con 0x7fea240743b0 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50932/0 (socket says 172.21.15.119:50932) 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 -- 172.21.15.119:0/3555336096 learned_addr learned my addr 172.21.15.119:0/3555336096 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 -- 172.21.15.119:0/3555336096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea240743b0 msgr2=0x7fea2419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea28e00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea2410deb0 0x7fea2419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23400640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea240743b0 0x7fea2419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea240743b0 0x7fea2419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 -- 172.21.15.119:0/3555336096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea2410deb0 msgr2=0x7fea2419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea2410deb0 0x7fea2419ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.966+0000 7fea23e00640 1 -- 172.21.15.119:0/3555336096 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2419f110 con 0x7fea24073a90 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.967+0000 7fea23e00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea1400d770 tx=0x7fea1400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.967+0000 7fea20c00640 1 -- 172.21.15.119:0/3555336096 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14017070 con 0x7fea24073a90 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.968+0000 7fea20c00640 1 -- 172.21.15.119:0/3555336096 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea14005500 con 0x7fea24073a90 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.968+0000 7fea20c00640 1 -- 172.21.15.119:0/3555336096 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14004dd0 con 0x7fea24073a90 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.968+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2419f400 con 0x7fea24073a90 2024-03-28T18:46:06.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.968+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2406b0c0 con 0x7fea24073a90 2024-03-28T18:46:06.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.973+0000 7fea2db75740 1 Processor -- start 2024-03-28T18:46:06.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.974+0000 7fea2db75740 1 -- start start 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba57820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eb9a72c0 con 0x5647eba55410 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eb94fc90 con 0x5647eba43670 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea2db75740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eb9a3360 con 0x5647eba44020 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea28e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea28e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37218/0 (socket says 172.21.15.119:37218) 2024-03-28T18:46:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea28e00640 1 -- 172.21.15.119:0/690311722 learned_addr learned my addr 172.21.15.119:0/690311722 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.975+0000 7fea23400640 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba57820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 -- 172.21.15.119:0/690311722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 msgr2=0x5647eba54e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:06.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 -- 172.21.15.119:0/690311722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 msgr2=0x5647eba57820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba57820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 -- 172.21.15.119:0/690311722 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647eb974080 con 0x5647eba43670 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea28e00640 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fea1002c150 tx=0x7fea10005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=838f875a79c2c227 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea19600640 1 -- 172.21.15.119:0/690311722 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10039050 con 0x5647eba43670 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea20c00640 1 -- 172.21.15.119:0/3555336096 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fea14009c30 con 0x7fea24073a90 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea19600640 1 -- 172.21.15.119:0/690311722 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea10043070 con 0x5647eba43670 2024-03-28T18:46:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.976+0000 7fea19600640 1 -- 172.21.15.119:0/690311722 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea100052a0 con 0x5647eba43670 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 -- 172.21.15.119:0/690311722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 msgr2=0x5647eba43a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fea1002c150 tx=0x7fea10005d60 comp rx=0 tx=0).stop 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 -- 172.21.15.119:0/690311722 shutdown_connections 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba57820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 --2- 172.21.15.119:0/690311722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba43a50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.977+0000 7fea2db75740 1 -- 172.21.15.119:0/690311722 >> 172.21.15.119:0/690311722 conn(0x5647eba3dda0 msgr2=0x5647eba3e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea2db75740 1 -- 172.21.15.119:0/690311722 shutdown_connections 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea2db75740 1 -- 172.21.15.119:0/690311722 wait complete. 2024-03-28T18:46:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea20c00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 0x7fea080a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea20c00640 1 -- 172.21.15.119:0/3555336096 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fea140ce9c0 con 0x7fea24073a90 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea2db75740 1 -- start start 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.978+0000 7fea23400640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 0x7fea080a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.979+0000 7fea2db75740 1 Processor -- start 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.979+0000 7fea2db75740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba52150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba52890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea23400640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 0x7fea080a4dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x5647eba3ef30 tx=0x7fea1c0095c0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea23e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea23e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37222/0 (socket says 172.21.15.119:37222) 2024-03-28T18:46:06.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea23400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba52150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea23e00640 1 -- 172.21.15.119:0/1903903031 learned_addr learned my addr 172.21.15.119:0/1903903031 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eba582e0 con 0x5647eba55410 2024-03-28T18:46:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eb9630d0 con 0x5647eba43670 2024-03-28T18:46:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647eba58460 con 0x5647eba44020 2024-03-28T18:46:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea2db75740 10 client.0 _ll_register_callbacks cb 0x5647eb967e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:06.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.980+0000 7fea28e00640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba52890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 -- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 msgr2=0x5647eba52150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba52150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 -- 172.21.15.119:0/1903903031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 msgr2=0x5647eba52890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba52890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647eb6bb3f0 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea23e00640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fea140cc0b0 tx=0x7fea14004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abfd8398f1a212d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea0e000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14020040 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea10043070 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.981+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fea1409a3b0 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea1409f0b0 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 10 client.26009 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5647eb7857d0 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea10043070 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fea14095cd0 con 0x5647eba43670 2024-03-28T18:46:06.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 10 client.26009 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 10 client.26009 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 20 client.26009 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647eba652b0 0x5647eba676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 4 client.26009 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647eb9b4800 con 0x5647eba652b0 2024-03-28T18:46:06.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.982+0000 7fea23400640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647eba652b0 0x5647eba676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:06.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.983+0000 7fea23400640 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647eba652b0 0x5647eba676a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88003147e9486ddc server_cookie=351cb9c78d5f83f2 in_seq=0 out_seq=0 2024-03-28T18:46:06.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.983+0000 7fea0e000640 10 client.26009 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:06.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea0e000640 1 -- 172.21.15.119:0/1903903031 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5647eb9b4800 con 0x5647eba652b0 2024-03-28T18:46:06.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea0e000640 10 client.26009 handle_command_reply: tid=0 2024-03-28T18:46:06.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 client.26009 shutdown 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 msgr2=0x5647eba54dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fea140cc0b0 tx=0x7fea14004800 comp rx=0 tx=0).stop 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 shutdown_connections 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647eba652b0 0x5647eba676a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba52890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba52150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 --2- 172.21.15.119:0/1903903031 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54dc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 >> 172.21.15.119:0/1903903031 conn(0x5647eba3dda0 msgr2=0x5647eba3faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.984+0000 7fea0e000640 0 client.26009 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.985+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 shutdown_connections 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.985+0000 7fea2db75740 1 -- 172.21.15.119:0/1903903031 wait complete. 2024-03-28T18:46:06.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.985+0000 7fea2db75740 20 client.26009 trim_cache size 0 max 16384 2024-03-28T18:46:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.998+0000 7fea2db75740 1 Processor -- start 2024-03-28T18:46:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.998+0000 7fea2db75740 1 -- start start 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba4f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2406f470 con 0x5647eba55410 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24070df0 con 0x5647eba43670 2024-03-28T18:46:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea2db75740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2406fad0 con 0x5647eba44020 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37226/0 (socket says 172.21.15.119:37226) 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 -- 172.21.15.119:0/1359836909 learned_addr learned my addr 172.21.15.119:0/1359836909 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea28e00640 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba4f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 -- 172.21.15.119:0/1359836909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 msgr2=0x5647eba54dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 -- 172.21.15.119:0/1359836909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 msgr2=0x5647eba4f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba4f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:06.999+0000 7fea23e00640 1 -- 172.21.15.119:0/1359836909 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647eba39510 con 0x5647eba43670 2024-03-28T18:46:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.000+0000 7fea23e00640 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fea1409aef0 tx=0x7fea140cc4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67792f49ec3c3bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.000+0000 7fea18c00640 1 -- 172.21.15.119:0/1359836909 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14020070 con 0x5647eba43670 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.000+0000 7fea18c00640 1 -- 172.21.15.119:0/1359836909 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea1407e070 con 0x5647eba43670 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.000+0000 7fea18c00640 1 -- 172.21.15.119:0/1359836909 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1409a610 con 0x5647eba43670 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 -- 172.21.15.119:0/1359836909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 msgr2=0x5647eba54880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fea1409aef0 tx=0x7fea140cc4e0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 -- 172.21.15.119:0/1359836909 shutdown_connections 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba55410 0x5647eba4f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba44020 0x5647eba54dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 --2- 172.21.15.119:0/1359836909 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba43670 0x5647eba54880 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 -- 172.21.15.119:0/1359836909 >> 172.21.15.119:0/1359836909 conn(0x5647eba3dda0 msgr2=0x5647eba3f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 -- 172.21.15.119:0/1359836909 shutdown_connections 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.001+0000 7fea2db75740 1 -- 172.21.15.119:0/1359836909 wait complete. 2024-03-28T18:46:07.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 -- start start 2024-03-28T18:46:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 Processor -- start 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba43670 0x5647eba5cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 0x5647eba5b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24070790 con 0x5647eba43670 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea24070130 con 0x5647eba44020 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2406d630 con 0x5647eba55410 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.002+0000 7fea2db75740 10 client.0 _ll_register_callbacks cb 0x7fea14097200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37232/0 (socket says 172.21.15.119:37232) 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 -- 172.21.15.119:0/432910926 learned_addr learned my addr 172.21.15.119:0/432910926 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 -- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 msgr2=0x5647eba5b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea28e00640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 0x5647eba5b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 0x5647eba5b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 -- 172.21.15.119:0/432910926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba43670 msgr2=0x5647eba5cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba43670 0x5647eba5cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 -- 172.21.15.119:0/432910926 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647eba5ba40 con 0x5647eba44020 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea28e00640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 0x5647eba5b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea23400640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fea24101040 tx=0x7fea1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83c98c7a3b275bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.003+0000 7fea0e000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c01bdf0 con 0x5647eba44020 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fea1c029700 con 0x5647eba44020 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea1407e070 con 0x5647eba44020 2024-03-28T18:46:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea1c037050 con 0x5647eba44020 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea2db75740 10 client.26015 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5647ebac1560 con 0x5647eba44020 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.004+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea1407e070 con 0x5647eba44020 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fea1c00b040 con 0x5647eba44020 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 10 client.26015 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 10 client.26015 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 20 client.26015 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647ebac1c90 0x5647ebac40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 4 client.26015 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5647eb9aeb50 con 0x5647ebac1c90 2024-03-28T18:46:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.005+0000 7fea23e00640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647ebac1c90 0x5647ebac40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.006+0000 7fea23e00640 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647ebac1c90 0x5647ebac40b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=831ec6322653327e server_cookie=b8a1730ffc8f3d33 in_seq=0 out_seq=0 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.006+0000 7fea0e000640 10 client.26015 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.007+0000 7fea0e000640 1 -- 172.21.15.119:0/432910926 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5647eb9aeb50 con 0x5647ebac1c90 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.007+0000 7fea0e000640 10 client.26015 handle_command_reply: tid=0 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.007+0000 7fea2db75740 1 client.26015 shutdown 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.007+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 msgr2=0x5647eba5d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.007+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fea24101040 tx=0x7fea1c004a40 comp rx=0 tx=0).stop 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 shutdown_connections 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5647ebac1c90 0x5647ebac40b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5647eba55410 0x5647eba5b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5647eba44020 0x5647eba5d2c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 --2- 172.21.15.119:0/432910926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647eba43670 0x5647eba5cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 >> 172.21.15.119:0/432910926 conn(0x5647eba3dda0 msgr2=0x5647eba3f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea0e000640 0 client.26015 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 shutdown_connections 2024-03-28T18:46:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 1 -- 172.21.15.119:0/432910926 wait complete. 2024-03-28T18:46:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.008+0000 7fea2db75740 20 client.26015 trim_cache size 0 max 16384 2024-03-28T18:46:07.010 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.009+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 msgr2=0x7fea080a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.009+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 0x7fea080a4dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x5647eba3ef30 tx=0x7fea1c0095c0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.009+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 msgr2=0x7fea2419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.009+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea1400d770 tx=0x7fea1400dc30 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 shutdown_connections 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fea080a2900 0x7fea080a4dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fea2410deb0 0x7fea2419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea240743b0 0x7fea2419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 --2- 172.21.15.119:0/3555336096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea24073a90 0x7fea2419a150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 >> 172.21.15.119:0/3555336096 conn(0x7fea24078a60 msgr2=0x7fea240fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 shutdown_connections 2024-03-28T18:46:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.010+0000 7fea2ae00640 1 -- 172.21.15.119:0/3555336096 wait complete. 2024-03-28T18:46:07.035 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:07.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:46:07.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 -- 172.21.15.119:0/1192128361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 msgr2=0x7f4750105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 --2- 172.21.15.119:0/1192128361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f4750105d50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4748009ca0 tx=0x7f474802c180 comp rx=0 tx=0).stop 2024-03-28T18:46:07.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 -- 172.21.15.119:0/1192128361 shutdown_connections 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 --2- 172.21.15.119:0/1192128361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f4750105d50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 --2- 172.21.15.119:0/1192128361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 0x7f47501052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 --2- 172.21.15.119:0/1192128361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f47501008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 -- 172.21.15.119:0/1192128361 >> 172.21.15.119:0/1192128361 conn(0x7f47500fbef0 msgr2=0x7f47500fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 -- 172.21.15.119:0/1192128361 shutdown_connections 2024-03-28T18:46:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.323+0000 7f4756a00640 1 -- 172.21.15.119:0/1192128361 wait complete. 2024-03-28T18:46:07.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.324+0000 7f4756a00640 1 Processor -- start 2024-03-28T18:46:07.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.324+0000 7f4756a00640 1 -- start start 2024-03-28T18:46:07.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f475019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 0x7f475019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750117ef0 con 0x7f47501004c0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750117d70 con 0x7f47501058c0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750118070 con 0x7f4750100de0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37238/0 (socket says 172.21.15.119:37238) 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4756000640 1 -- 172.21.15.119:0/1298568524 learned_addr learned my addr 172.21.15.119:0/1298568524 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4754c00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 0x7f475019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.325+0000 7f4755600640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f475019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 -- 172.21.15.119:0/1298568524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 msgr2=0x7f475019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 0x7f475019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 -- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 msgr2=0x7f475019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f475019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 -- 172.21.15.119:0/1298568524 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47501a3400 con 0x7f47501058c0 2024-03-28T18:46:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4755600640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f475019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f4756000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4748005e30 tx=0x7f4748035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f474e000640 1 -- 172.21.15.119:0/1298568524 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748039040 con 0x7f47501058c0 2024-03-28T18:46:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f474e000640 1 -- 172.21.15.119:0/1298568524 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4748044070 con 0x7f47501058c0 2024-03-28T18:46:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.326+0000 7f474e000640 1 -- 172.21.15.119:0/1298568524 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480328d0 con 0x7f47501058c0 2024-03-28T18:46:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.327+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47501a3690 con 0x7f47501058c0 2024-03-28T18:46:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.327+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47501aaf80 con 0x7f47501058c0 2024-03-28T18:46:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.331+0000 7f475ada6740 1 Processor -- start 2024-03-28T18:46:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.331+0000 7f474e000640 1 -- 172.21.15.119:0/1298568524 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4748003b70 con 0x7f47501058c0 2024-03-28T18:46:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.332+0000 7f475ada6740 1 -- start start 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20b98b0 0x55b0b20b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20ba260 0x55b0b20c2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b201d2c0 con 0x55b0b20b98b0 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b1fc5c90 con 0x55b0b20c2de0 2024-03-28T18:46:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.333+0000 7f475ada6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b2019360 con 0x55b0b20ba260 2024-03-28T18:46:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f4754c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f4754c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37242/0 (socket says 172.21.15.119:37242) 2024-03-28T18:46:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f4754c00640 1 -- 172.21.15.119:0/3883902871 learned_addr learned my addr 172.21.15.119:0/3883902871 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f474e000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 0x7f47380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f474e000640 1 -- 172.21.15.119:0/1298568524 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f47480f0c40 con 0x7f47501058c0 2024-03-28T18:46:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f4754c00640 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20b98b0 0x55b0b20b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.334+0000 7f4755600640 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20ba260 0x55b0b20c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 -- 172.21.15.119:0/3883902871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20ba260 msgr2=0x55b0b20c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20ba260 0x55b0b20c2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 -- 172.21.15.119:0/3883902871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20b98b0 msgr2=0x55b0b20b9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20b98b0 0x55b0b20b9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 -- 172.21.15.119:0/3883902871 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0b1fea080 con 0x55b0b20c2de0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4754c00640 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f47440058c0 tx=0x7f4744005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32166f4af570435b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4742600640 1 -- 172.21.15.119:0/3883902871 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744004030 con 0x55b0b20c2de0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4742600640 1 -- 172.21.15.119:0/3883902871 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4744005500 con 0x55b0b20c2de0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.335+0000 7f4742600640 1 -- 172.21.15.119:0/3883902871 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744004b60 con 0x55b0b20c2de0 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 -- 172.21.15.119:0/3883902871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 msgr2=0x55b0b20c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f47440058c0 tx=0x7f4744005850 comp rx=0 tx=0).stop 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 -- 172.21.15.119:0/3883902871 shutdown_connections 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20c2de0 0x55b0b20c5230 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20ba260 0x55b0b20c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 --2- 172.21.15.119:0/3883902871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20b98b0 0x55b0b20b9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 -- 172.21.15.119:0/3883902871 >> 172.21.15.119:0/3883902871 conn(0x55b0b20b3f80 msgr2=0x55b0b20b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.336+0000 7f475ada6740 1 -- 172.21.15.119:0/3883902871 shutdown_connections 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.337+0000 7f475ada6740 1 -- 172.21.15.119:0/3883902871 wait complete. 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.337+0000 7f475ada6740 1 -- start start 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.337+0000 7f475ada6740 1 Processor -- start 2024-03-28T18:46:07.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20bfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20c00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b20c5d50 con 0x55b0b20ba260 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b1fd90d0 con 0x55b0b20b98b0 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0b20c5ed0 con 0x55b0b20c2de0 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f475ada6740 10 client.0 _ll_register_callbacks cb 0x55b0b1fdde80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f4754c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f4754c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37254/0 (socket says 172.21.15.119:37254) 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f4754c00640 1 -- 172.21.15.119:0/1990572009 learned_addr learned my addr 172.21.15.119:0/1990572009 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.338+0000 7f4754c00640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20c00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4755600640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20bfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4754c00640 1 -- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 msgr2=0x55b0b20c00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4754c00640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20c00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4754c00640 1 -- 172.21.15.119:0/1990572009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 msgr2=0x55b0b20bfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4754c00640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20bfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4754c00640 1 -- 172.21.15.119:0/1990572009 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0b1d313f0 con 0x55b0b20b98b0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.339+0000 7f4755600640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20bfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f4754c00640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4744009380 tx=0x7f4744009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5c4442b39b6abb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f4736a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744023070 con 0x55b0b20b98b0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4744004b00 con 0x55b0b20b98b0 2024-03-28T18:46:07.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4744005500 con 0x55b0b20b98b0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4744035050 con 0x55b0b20b98b0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f475ada6740 10 client.26027 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.340+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b0b1dfb7d0 con 0x55b0b20b98b0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4744005500 con 0x55b0b20b98b0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f474401f440 con 0x55b0b20b98b0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f4756000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 0x7f47380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 10 client.26027 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 10 client.26027 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 20 client.26027 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b20d2c20 0x55b0b20d5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 4 client.26027 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:07.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.341+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0b202a800 con 0x55b0b20d2c20 2024-03-28T18:46:07.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.342+0000 7f4755600640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b20d2c20 0x55b0b20d5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.342+0000 7f4736a00640 10 client.26027 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:07.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.343+0000 7f4756000640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 0x7f47380a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f47480c55c0 tx=0x7f47480021f0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.343+0000 7f4755600640 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b20d2c20 0x55b0b20d5010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=151f7db4b86e4ca4 server_cookie=e6ab72cb69511012 in_seq=0 out_seq=0 2024-03-28T18:46:07.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.344+0000 7f4736a00640 1 -- 172.21.15.119:0/1990572009 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b0b202a800 con 0x55b0b20d2c20 2024-03-28T18:46:07.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.344+0000 7f4736a00640 10 client.26027 handle_command_reply: tid=0 2024-03-28T18:46:07.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.345+0000 7f475ada6740 1 client.26027 shutdown 2024-03-28T18:46:07.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.346+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 msgr2=0x55b0b20c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.346+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4744009380 tx=0x7f4744009530 comp rx=0 tx=0).stop 2024-03-28T18:46:07.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 shutdown_connections 2024-03-28T18:46:07.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b20d2c20 0x55b0b20d5010 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20c00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20bfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f475ada6740 1 --2- 172.21.15.119:0/1990572009 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c2720 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.347+0000 7f4736a00640 0 client.26027 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.348+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 >> 172.21.15.119:0/1990572009 conn(0x55b0b20b3f80 msgr2=0x55b0b20b5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.348+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 shutdown_connections 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.348+0000 7f475ada6740 1 -- 172.21.15.119:0/1990572009 wait complete. 2024-03-28T18:46:07.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.348+0000 7f475ada6740 20 client.26027 trim_cache size 0 max 16384 2024-03-28T18:46:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.381+0000 7f475ada6740 1 Processor -- start 2024-03-28T18:46:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.381+0000 7f475ada6740 1 -- start start 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b37a0 con 0x55b0b20ba260 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b5120 con 0x55b0b20b98b0 2024-03-28T18:46:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f475ada6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b3e00 con 0x55b0b20c2de0 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f4755600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f4754c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f4754c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59370/0 (socket says 172.21.15.119:59370) 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.382+0000 7f4754c00640 1 -- 172.21.15.119:0/1887165148 learned_addr learned my addr 172.21.15.119:0/1887165148 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 -- 172.21.15.119:0/1887165148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 msgr2=0x55b0b20c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 -- 172.21.15.119:0/1887165148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 msgr2=0x55b0b20c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 -- 172.21.15.119:0/1887165148 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0b20c0430 con 0x55b0b20c2de0 2024-03-28T18:46:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4754c00640 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4744005bc0 tx=0x7f4744009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb8d0779419ad7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.383+0000 7f4741c00640 1 -- 172.21.15.119:0/1887165148 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744023070 con 0x55b0b20c2de0 2024-03-28T18:46:07.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.384+0000 7f4741c00640 1 -- 172.21.15.119:0/1887165148 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4744028050 con 0x55b0b20c2de0 2024-03-28T18:46:07.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.384+0000 7f4741c00640 1 -- 172.21.15.119:0/1887165148 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474402a070 con 0x55b0b20c2de0 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 -- 172.21.15.119:0/1887165148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 msgr2=0x55b0b20bcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4744005bc0 tx=0x7f4744009790 comp rx=0 tx=0).stop 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 -- 172.21.15.119:0/1887165148 shutdown_connections 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c2de0 0x55b0b20bcd30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 --2- 172.21.15.119:0/1887165148 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 -- 172.21.15.119:0/1887165148 >> 172.21.15.119:0/1887165148 conn(0x55b0b20b3f80 msgr2=0x55b0b20b3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 -- 172.21.15.119:0/1887165148 shutdown_connections 2024-03-28T18:46:07.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.385+0000 7f475ada6740 1 -- 172.21.15.119:0/1887165148 wait complete. 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 -- start start 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 Processor -- start 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c8d80 0x55b0b20c9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b4ac0 con 0x55b0b20ba260 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b4460 con 0x55b0b20b98b0 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47501b1960 con 0x55b0b20c8d80 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.386+0000 7f475ada6740 10 client.0 _ll_register_callbacks cb 0x7f4744009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51050/0 (socket says 172.21.15.119:51050) 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 -- 172.21.15.119:0/287170846 learned_addr learned my addr 172.21.15.119:0/287170846 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4754c00640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c8d80 0x55b0b20c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4754c00640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 -- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c8d80 msgr2=0x55b0b20c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c8d80 0x55b0b20c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 -- 172.21.15.119:0/287170846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 msgr2=0x55b0b20ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 -- 172.21.15.119:0/287170846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0b20c9840 con 0x55b0b20ba260 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4754c00640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4755600640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55b0b20b4ff0 tx=0x7f473c01a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78c72484d1699dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.387+0000 7f4736a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c004220 con 0x55b0b20ba260 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f473c0043c0 con 0x55b0b20ba260 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4744028050 con 0x55b0b20ba260 2024-03-28T18:46:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f473c029050 con 0x55b0b20ba260 2024-03-28T18:46:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 10 client.15876 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b0b212f1f0 con 0x55b0b20ba260 2024-03-28T18:46:07.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4744028050 con 0x55b0b20ba260 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f473c011650 con 0x55b0b20ba260 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 10 client.15876 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 10 client.15876 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 20 client.15876 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b212f950 0x55b0b2131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 4 client.15876 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.388+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b0b2024b50 con 0x55b0b212f950 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.389+0000 7f4754c00640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b212f950 0x55b0b2131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.389+0000 7f4736a00640 10 client.15876 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.389+0000 7f4754c00640 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b212f950 0x55b0b2131d70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4ae2563577b7fc9 server_cookie=da2b55dbbd79d5df in_seq=0 out_seq=0 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f4736a00640 1 -- 172.21.15.119:0/287170846 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b0b2024b50 con 0x55b0b212f950 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f4736a00640 10 client.15876 handle_command_reply: tid=0 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 client.15876 shutdown 2024-03-28T18:46:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 msgr2=0x55b0b20cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55b0b20b4ff0 tx=0x7f473c01a9d0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 shutdown_connections 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b0b212f950 0x55b0b2131d70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0b20c8d80 0x55b0b20c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b0b20ba260 0x55b0b20cade0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 --2- 172.21.15.119:0/287170846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b0b20b98b0 0x55b0b20ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 >> 172.21.15.119:0/287170846 conn(0x55b0b20b3f80 msgr2=0x55b0b20ba690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f4736a00640 0 client.15876 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 shutdown_connections 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.390+0000 7f475ada6740 1 -- 172.21.15.119:0/287170846 wait complete. 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f475ada6740 20 client.15876 trim_cache size 0 max 16384 2024-03-28T18:46:07.393 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 msgr2=0x7f47380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 0x7f47380a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f47480c55c0 tx=0x7f47480021f0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 msgr2=0x7f47501a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4748005e30 tx=0x7f4748035d80 comp rx=0 tx=0).stop 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 shutdown_connections 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47380a2970 0x7f47380a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47501058c0 0x7f47501a2cd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4750100de0 0x7f475019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.391+0000 7f4756a00640 1 --2- 172.21.15.119:0/1298568524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47501004c0 0x7f475019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.392+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 >> 172.21.15.119:0/1298568524 conn(0x7f47500fbef0 msgr2=0x7f4750103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.392+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 shutdown_connections 2024-03-28T18:46:07.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.392+0000 7f4756a00640 1 -- 172.21.15.119:0/1298568524 wait complete. 2024-03-28T18:46:07.415 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:07.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.696+0000 7f9009800640 1 -- 172.21.15.119:0/1092026823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004103840 msgr2=0x7f9004103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.696+0000 7f9009800640 1 --2- 172.21.15.119:0/1092026823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004103840 0x7f9004103cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009930 tx=0x7f8ff002bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 -- 172.21.15.119:0/1092026823 shutdown_connections 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 --2- 172.21.15.119:0/1092026823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f9004110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 --2- 172.21.15.119:0/1092026823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004103840 0x7f9004103cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 --2- 172.21.15.119:0/1092026823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004109850 0x7f9004109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 -- 172.21.15.119:0/1092026823 >> 172.21.15.119:0/1092026823 conn(0x7f90040ff310 msgr2=0x7f9004101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 -- 172.21.15.119:0/1092026823 shutdown_connections 2024-03-28T18:46:07.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.697+0000 7f9009800640 1 -- 172.21.15.119:0/1092026823 wait complete. 2024-03-28T18:46:07.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.698+0000 7f9009800640 1 Processor -- start 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.698+0000 7f9009800640 1 -- start start 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.698+0000 7f9009800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004103840 0x7f900419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.698+0000 7f9009800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.698+0000 7f9009800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004109850 0x7f900419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9009800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004113cb0 con 0x7f9004103840 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9009800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004113b30 con 0x7f9004104200 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9009800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004113e30 con 0x7f9004109850 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004103840 0x7f900419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37298/0 (socket says 172.21.15.119:37298) 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 -- 172.21.15.119:0/2516244303 learned_addr learned my addr 172.21.15.119:0/2516244303 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9003400640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004109850 0x7f900419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 -- 172.21.15.119:0/2516244303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004109850 msgr2=0x7f900419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004109850 0x7f900419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 -- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004103840 msgr2=0x7f900419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004103840 0x7f900419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.699+0000 7f9002000640 1 -- 172.21.15.119:0/2516244303 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900419f130 con 0x7f9004104200 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f9002000640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009900 tx=0x7f8ff00042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f8ffb400640 1 -- 172.21.15.119:0/2516244303 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0043020 con 0x7f9004104200 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900419f3c0 con 0x7f9004104200 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f8ffb400640 1 -- 172.21.15.119:0/2516244303 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff00044d0 con 0x7f9004104200 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f8ffb400640 1 -- 172.21.15.119:0/2516244303 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0003be0 con 0x7f9004104200 2024-03-28T18:46:07.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.700+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90041a6cb0 con 0x7f9004104200 2024-03-28T18:46:07.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.705+0000 7f8ffb400640 1 -- 172.21.15.119:0/2516244303 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8ff0046020 con 0x7f9004104200 2024-03-28T18:46:07.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.706+0000 7f900c427740 1 Processor -- start 2024-03-28T18:46:07.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.706+0000 7f900c427740 1 -- start start 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.706+0000 7f900c427740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.706+0000 7f900c427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6aa7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f900c427740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6aaa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f900c427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef69fa2c0 con 0x561ef6a97020 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f900c427740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef69a2c90 con 0x561ef6a96670 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f900c427740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef69f6360 con 0x561ef6aa8410 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f9003400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f9003400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37300/0 (socket says 172.21.15.119:37300) 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.707+0000 7f9003400640 1 -- 172.21.15.119:0/3091806264 learned_addr learned my addr 172.21.15.119:0/3091806264 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 -- 172.21.15.119:0/3091806264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 msgr2=0x561ef6aaa820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6aaa820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 -- 172.21.15.119:0/3091806264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 msgr2=0x561ef6aa7e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6aa7e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f8ffb400640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 0x7f8fe80a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 -- 172.21.15.119:0/3091806264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef69c7080 con 0x561ef6a96670 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f8ffb400640 1 -- 172.21.15.119:0/2516244303 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8ff00f0d00 con 0x7f9004104200 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.708+0000 7f9003400640 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009fd0 tx=0x7f8ffc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2714e676fe3192e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.709+0000 7f9003400640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 0x7f8fe80a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.709+0000 7f8fefe00640 1 -- 172.21.15.119:0/3091806264 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc01e070 con 0x561ef6a96670 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.710+0000 7f8fefe00640 1 -- 172.21.15.119:0/3091806264 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ffc003dd0 con 0x561ef6a96670 2024-03-28T18:46:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.710+0000 7f8fefe00640 1 -- 172.21.15.119:0/3091806264 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00fd40 con 0x561ef6a96670 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 -- 172.21.15.119:0/3091806264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 msgr2=0x561ef6a96a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009fd0 tx=0x7f8ffc0118d0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f9003400640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 0x7f8fe80a4f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc01b000 tx=0x7f8ffc019c40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 -- 172.21.15.119:0/3091806264 shutdown_connections 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6aaa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6aa7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 --2- 172.21.15.119:0/3091806264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6a96a50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 -- 172.21.15.119:0/3091806264 >> 172.21.15.119:0/3091806264 conn(0x561ef6a90da0 msgr2=0x561ef6a91180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 -- 172.21.15.119:0/3091806264 shutdown_connections 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.711+0000 7f900c427740 1 -- 172.21.15.119:0/3091806264 wait complete. 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.712+0000 7f900c427740 1 -- start start 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.712+0000 7f900c427740 1 Processor -- start 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 0x561ef6aa22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef6a979d0 con 0x561ef6aa8410 2024-03-28T18:46:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef69b60d0 con 0x561ef6a96670 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ef6a97b50 con 0x561ef6a97020 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.713+0000 7f900c427740 10 client.0 _ll_register_callbacks cb 0x561ef69bae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9002000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 0x561ef6aa22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9002a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51074/0 (socket says 172.21.15.119:51074) 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9002000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 0x561ef6aa22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:59422/0 (socket says 172.21.15.119:59422) 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 -- 172.21.15.119:0/1638812038 learned_addr learned my addr 172.21.15.119:0/1638812038 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9002a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37304/0 (socket says 172.21.15.119:37304) 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 -- 172.21.15.119:0/1638812038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 msgr2=0x561ef6aa22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 0x561ef6aa22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 -- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 msgr2=0x561ef6aa1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9003400640 1 -- 172.21.15.119:0/1638812038 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef670e3f0 con 0x561ef6aa8410 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.714+0000 7f9002a00640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.715+0000 7f9003400640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009270 tx=0x7f8ffc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d9345e009b19c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.715+0000 7f8fecc00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.715+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc034020 con 0x561ef6aa8410 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.715+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8ffc00eb00 con 0x561ef6aa8410 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.715+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ffc003dd0 con 0x561ef6aa8410 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.716+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ffc03f050 con 0x561ef6aa8410 2024-03-28T18:46:07.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.716+0000 7f900c427740 10 client.15885 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.716+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561ef67d87d0 con 0x561ef6aa8410 2024-03-28T18:46:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.716+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ffc003dd0 con 0x561ef6aa8410 2024-03-28T18:46:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ffc0044e0 con 0x561ef6aa8410 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 10 client.15885 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 20 client.15885 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6ac0b70 0x561ef6ac2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 4 client.15885 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.717+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ef6a07800 con 0x561ef6ac0b70 2024-03-28T18:46:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.718+0000 7f9002a00640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6ac0b70 0x561ef6ac2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.720+0000 7f9002a00640 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6ac0b70 0x561ef6ac2f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2bdfed584564276 server_cookie=368e45fc12f0ea1a in_seq=0 out_seq=0 2024-03-28T18:46:07.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.720+0000 7f8fecc00640 10 client.15885 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f8fecc00640 1 -- 172.21.15.119:0/1638812038 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561ef6a07800 con 0x561ef6ac0b70 2024-03-28T18:46:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f8fecc00640 10 client.15885 handle_command_reply: tid=0 2024-03-28T18:46:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 client.15885 shutdown 2024-03-28T18:46:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 msgr2=0x561ef6aa7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009270 tx=0x7f8ffc019000 comp rx=0 tx=0).stop 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 shutdown_connections 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6ac0b70 0x561ef6ac2f60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6aa8410 0x561ef6aa7ad0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6a97020 0x561ef6aa22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 --2- 172.21.15.119:0/1638812038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f8fecc00640 0 client.15885 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 >> 172.21.15.119:0/1638812038 conn(0x561ef6a90da0 msgr2=0x561ef6a92af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 shutdown_connections 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.723+0000 7f900c427740 1 -- 172.21.15.119:0/1638812038 wait complete. 2024-03-28T18:46:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.724+0000 7f900c427740 20 client.15885 trim_cache size 0 max 16384 2024-03-28T18:46:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.749+0000 7f900c427740 1 Processor -- start 2024-03-28T18:46:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.749+0000 7f900c427740 1 -- start start 2024-03-28T18:46:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6a8feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6a9c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041ab1e0 con 0x561ef6a97020 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041acb60 con 0x561ef6a96670 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f900c427740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041ab840 con 0x561ef6aa8410 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9002000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6a8feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9002a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9002a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37318/0 (socket says 172.21.15.119:37318) 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9002a00640 1 -- 172.21.15.119:0/3031419120 learned_addr learned my addr 172.21.15.119:0/3031419120 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9002000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6a8feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51086/0 (socket says 172.21.15.119:51086) 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.750+0000 7f9003400640 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6a9c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 -- 172.21.15.119:0/3031419120 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 msgr2=0x561ef6a9c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6a9c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 -- 172.21.15.119:0/3031419120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 msgr2=0x561ef6a8feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6a8feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 -- 172.21.15.119:0/3031419120 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ef6a96dc0 con 0x561ef6a96670 2024-03-28T18:46:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f9002a00640 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x561ef6ac3660 tx=0x7f8ff4004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f938293047538b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f8fef400640 1 -- 172.21.15.119:0/3031419120 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4003b00 con 0x561ef6a96670 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f8fef400640 1 -- 172.21.15.119:0/3031419120 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4005500 con 0x561ef6a96670 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.751+0000 7f8fef400640 1 -- 172.21.15.119:0/3031419120 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4004e00 con 0x561ef6a96670 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 -- 172.21.15.119:0/3031419120 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 msgr2=0x561ef6aa79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x561ef6ac3660 tx=0x7f8ff4004410 comp rx=0 tx=0).stop 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 -- 172.21.15.119:0/3031419120 shutdown_connections 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6a9c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a97020 0x561ef6a8feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 --2- 172.21.15.119:0/3031419120 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a96670 0x561ef6aa79f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 -- 172.21.15.119:0/3031419120 >> 172.21.15.119:0/3031419120 conn(0x561ef6a90da0 msgr2=0x561ef6a9b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 -- 172.21.15.119:0/3031419120 shutdown_connections 2024-03-28T18:46:07.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.752+0000 7f900c427740 1 -- 172.21.15.119:0/3031419120 wait complete. 2024-03-28T18:46:07.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.753+0000 7f900c427740 1 -- start start 2024-03-28T18:46:07.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.753+0000 7f900c427740 1 Processor -- start 2024-03-28T18:46:07.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a96670 0x561ef6b10250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6b10cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041ac500 con 0x561ef6a96670 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041abea0 con 0x561ef6a97020 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041a93a0 con 0x561ef6aa8410 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f900c427740 10 client.0 _ll_register_callbacks cb 0x7f8ffc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f9002000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f9002a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a96670 0x561ef6b10250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f9002000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37332/0 (socket says 172.21.15.119:37332) 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f9002000640 1 -- 172.21.15.119:0/1445593053 learned_addr learned my addr 172.21.15.119:0/1445593053 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.754+0000 7f9003400640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6b10cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 -- 172.21.15.119:0/1445593053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 msgr2=0x561ef6b10cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6b10cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 -- 172.21.15.119:0/1445593053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a96670 msgr2=0x561ef6b10250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a96670 0x561ef6b10250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 -- 172.21.15.119:0/1445593053 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ef6b1b430 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f9002000640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0000c00 tx=0x7f8ff00035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47e25a5a1bab7c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f8fecc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00d6260 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ff4005500 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8ff00c56c0 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ff0004c80 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f900c427740 10 client.26045 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.755+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561ef6b1c7e0 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff4005500 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ff002ca90 con 0x561ef6a97020 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 10 client.26045 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 20 client.26045 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6b1cef0 0x561ef6b1f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 4 client.26045 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561ef6a01b50 con 0x561ef6b1cef0 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.756+0000 7f9002a00640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6b1cef0 0x561ef6b1f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f9002a00640 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6b1cef0 0x561ef6b1f310 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29a97732c2fbfa21 server_cookie=e6b34cff703ebf22 in_seq=0 out_seq=0 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f8fecc00640 10 client.26045 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f8fecc00640 1 -- 172.21.15.119:0/1445593053 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561ef6a01b50 con 0x561ef6b1cef0 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f8fecc00640 10 client.26045 handle_command_reply: tid=0 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 client.26045 shutdown 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 msgr2=0x561ef6b10790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0000c00 tx=0x7f8ff00035f0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 shutdown_connections 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561ef6b1cef0 0x561ef6b1f310 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f8fecc00640 0 client.26045 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561ef6aa8410 0x561ef6b10cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561ef6a97020 0x561ef6b10790 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 --2- 172.21.15.119:0/1445593053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561ef6a96670 0x561ef6b10250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.757+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 >> 172.21.15.119:0/1445593053 conn(0x561ef6a90da0 msgr2=0x561ef6a9c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.758+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 shutdown_connections 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.758+0000 7f900c427740 1 -- 172.21.15.119:0/1445593053 wait complete. 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.758+0000 7f900c427740 20 client.26045 trim_cache size 0 max 16384 2024-03-28T18:46:07.760 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.761+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 msgr2=0x7f8fe80a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.761+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 0x7f8fe80a4f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc01b000 tx=0x7f8ffc019c40 comp rx=0 tx=0).stop 2024-03-28T18:46:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.761+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 msgr2=0x7f900419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.761+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009900 tx=0x7f8ff00042e0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 shutdown_connections 2024-03-28T18:46:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8fe80a2ab0 0x7f8fe80a4f70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9004109850 0x7f900419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9004104200 0x7f900419a6d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 --2- 172.21.15.119:0/2516244303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9004103840 0x7f900419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.763+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 >> 172.21.15.119:0/2516244303 conn(0x7f90040ff310 msgr2=0x7f90040ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.765+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 shutdown_connections 2024-03-28T18:46:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:07.765+0000 7f9009800640 1 -- 172.21.15.119:0/2516244303 wait complete. 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 rewriters = 341504.19 kB/sec 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 rewriters = 341420.25 kB/sec 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 341504.19 kB/sec 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 341504.19 kB/sec 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 341504.19 kB/sec 2024-03-28T18:46:08.394 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:46:08.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:08 smithi196 ceph-mon[36300]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 MiB/s wr, 68 op/s 2024-03-28T18:46:08.804 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:46:08.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:46:08.804 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:46:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:08 smithi119 ceph-mon[31813]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 MiB/s wr, 68 op/s 2024-03-28T18:46:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:08 smithi131 ceph-mon[36065]: pgmap v65: 97 pgs: 97 active+clean; 1.0 GiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 MiB/s wr, 68 op/s 2024-03-28T18:46:09.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.020+0000 7f2de5000640 1 -- 172.21.15.119:0/3970182882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 msgr2=0x7f2de0103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.020+0000 7f2de5000640 1 --2- 172.21.15.119:0/3970182882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de0103cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009930 tx=0x7f2dc802bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:09.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 -- 172.21.15.119:0/3970182882 shutdown_connections 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 --2- 172.21.15.119:0/3970182882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 --2- 172.21.15.119:0/3970182882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de0103cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 --2- 172.21.15.119:0/3970182882 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 -- 172.21.15.119:0/3970182882 >> 172.21.15.119:0/3970182882 conn(0x7f2de00ff350 msgr2=0x7f2de0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 -- 172.21.15.119:0/3970182882 shutdown_connections 2024-03-28T18:46:09.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.021+0000 7f2de5000640 1 -- 172.21.15.119:0/3970182882 wait complete. 2024-03-28T18:46:09.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 Processor -- start 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 -- start start 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0113c20 con 0x7f2de0104200 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0113aa0 con 0x7f2de0109850 2024-03-28T18:46:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2de5000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0113da0 con 0x7f2de0103840 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2ddde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2dde800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de019a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33880/0 (socket says 172.21.15.119:33880) 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2ddde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48530/0 (socket says 172.21.15.119:48530) 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.022+0000 7f2dd5400640 1 -- 172.21.15.119:0/891542956 learned_addr learned my addr 172.21.15.119:0/891542956 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 -- 172.21.15.119:0/891542956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 msgr2=0x7f2de019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 -- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 msgr2=0x7f2de019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 -- 172.21.15.119:0/891542956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de019f160 con 0x7f2de0109850 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dde800640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de019ea30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009fd0 tx=0x7f2dd800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dd7400640 1 -- 172.21.15.119:0/891542956 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800ce20 con 0x7f2de0109850 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de019f450 con 0x7f2de0109850 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dd7400640 1 -- 172.21.15.119:0/891542956 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8005020 con 0x7f2de0109850 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.023+0000 7f2dd7400640 1 -- 172.21.15.119:0/891542956 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8010720 con 0x7f2de0109850 2024-03-28T18:46:09.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.024+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de01a6ce0 con 0x7f2de0109850 2024-03-28T18:46:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.027+0000 7f2de7bf5740 1 Processor -- start 2024-03-28T18:46:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2dd7400640 1 -- 172.21.15.119:0/891542956 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2dd8005540 con 0x7f2de0109850 2024-03-28T18:46:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 -- start start 2024-03-28T18:46:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f9a0 0x55b98d718040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d718580 0x55b98d71a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d673eb0 con 0x55b98d70f9a0 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d61ca60 con 0x55b98d718580 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d66ff50 con 0x55b98d70f080 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.028+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48532/0 (socket says 172.21.15.119:48532) 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 -- 172.21.15.119:0/2625615620 learned_addr learned my addr 172.21.15.119:0/2625615620 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d718580 0x55b98d71a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2ddde00640 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f9a0 0x55b98d718040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 -- 172.21.15.119:0/2625615620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d718580 msgr2=0x55b98d71a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d718580 0x55b98d71a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 -- 172.21.15.119:0/2625615620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f9a0 msgr2=0x55b98d718040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f9a0 0x55b98d718040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 -- 172.21.15.119:0/2625615620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b98d4507e0 con 0x55b98d70f080 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd5400640 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8005c40 tx=0x7f2dc8005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed9113bde930d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dcf400640 1 -- 172.21.15.119:0/2625615620 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc802ccf0 con 0x55b98d70f080 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dcf400640 1 -- 172.21.15.119:0/2625615620 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dc80046c0 con 0x55b98d70f080 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dcf400640 1 -- 172.21.15.119:0/2625615620 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8005300 con 0x55b98d70f080 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.029+0000 7f2dd7400640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 0x7f2dc40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2dd7400640 1 -- 172.21.15.119:0/891542956 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dd80cf5b0 con 0x7f2de0109850 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2ddde00640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 0x7f2dc40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 -- 172.21.15.119:0/2625615620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 msgr2=0x55b98d70f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8005c40 tx=0x7f2dc8005b50 comp rx=0 tx=0).stop 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 -- 172.21.15.119:0/2625615620 shutdown_connections 2024-03-28T18:46:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d718580 0x55b98d71a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f9a0 0x55b98d718040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/2625615620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f080 0x55b98d70f460 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 -- 172.21.15.119:0/2625615620 >> 172.21.15.119:0/2625615620 conn(0x55b98d7097b0 msgr2=0x55b98d709b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 -- 172.21.15.119:0/2625615620 shutdown_connections 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2ddde00640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 0x7f2dc40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55b98d70eb50 tx=0x7f2dd0005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.030+0000 7f2de7bf5740 1 -- 172.21.15.119:0/2625615620 wait complete. 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 -- start start 2024-03-28T18:46:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 Processor -- start 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d717ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d7157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d71b310 con 0x55b98d718580 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d62feb0 con 0x55b98d70f080 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.031+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b98d71b490 con 0x55b98d70f9a0 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2de7bf5740 10 client.0 _ll_register_callbacks cb 0x55b98d624150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48534/0 (socket says 172.21.15.119:48534) 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 -- 172.21.15.119:0/1383557625 learned_addr learned my addr 172.21.15.119:0/1383557625 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2ddde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d717ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dde800640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d7157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 -- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 msgr2=0x55b98d717ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d717ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 -- 172.21.15.119:0/1383557625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 msgr2=0x55b98d7157c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d7157c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 -- 172.21.15.119:0/1383557625 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b98d4b0f70 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dd5400640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2dc802bc70 tx=0x7f2dc8005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb09aeb0653493bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8004030 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2dc80046c0 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.032+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dc80041d0 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dc8058050 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 10 client.26062 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b98d348170 con 0x55b98d70f9a0 2024-03-28T18:46:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dc80046c0 con 0x55b98d70f9a0 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2dc8053030 con 0x55b98d70f9a0 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 10 client.26062 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 10 client.26062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 20 client.26062 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d728490 0x55b98d72a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 4 client.26062 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b98d6813f0 con 0x55b98d728490 2024-03-28T18:46:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.033+0000 7f2ddde00640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d728490 0x55b98d72a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.034+0000 7f2dc3e00640 10 client.26062 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.034+0000 7f2ddde00640 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d728490 0x55b98d72a880 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cedc4422028c7c81 server_cookie=a2a02adcf83e2ee3 in_seq=0 out_seq=0 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2dc3e00640 1 -- 172.21.15.119:0/1383557625 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b98d6813f0 con 0x55b98d728490 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2dc3e00640 10 client.26062 handle_command_reply: tid=0 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 client.26062 shutdown 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 msgr2=0x55b98d715260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2dc802bc70 tx=0x7f2dc8005a20 comp rx=0 tx=0).stop 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 shutdown_connections 2024-03-28T18:46:09.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d728490 0x55b98d72a880 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d7157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d715260 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1383557625 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d717ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2dc3e00640 0 client.26062 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 >> 172.21.15.119:0/1383557625 conn(0x55b98d7097b0 msgr2=0x55b98d70b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 shutdown_connections 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1383557625 wait complete. 2024-03-28T18:46:09.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.035+0000 7f2de7bf5740 20 client.26062 trim_cache size 0 max 16384 2024-03-28T18:46:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 Processor -- start 2024-03-28T18:46:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 -- start start 2024-03-28T18:46:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 0x55b98d70b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 0x55b98d717990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01af4b0 con 0x55b98d70f080 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01b0e30 con 0x55b98d70f9a0 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.052+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01afb10 con 0x55b98d718580 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48548/0 (socket says 172.21.15.119:48548) 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 -- 172.21.15.119:0/1555385272 learned_addr learned my addr 172.21.15.119:0/1555385272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2ddde00640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 0x55b98d70b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dd5400640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 0x55b98d717990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 -- 172.21.15.119:0/1555385272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 msgr2=0x55b98d717990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 0x55b98d717990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 -- 172.21.15.119:0/1555385272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 msgr2=0x55b98d70b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 0x55b98d70b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 -- 172.21.15.119:0/1555385272 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b98d71ace0 con 0x55b98d718580 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dd5400640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 0x55b98d717990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2ddde00640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 0x55b98d70b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dde800640 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8098580 tx=0x7f2dd809bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a0b59e2b64fe70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dcea00640 1 -- 172.21.15.119:0/1555385272 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd80051d0 con 0x55b98d718580 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dcea00640 1 -- 172.21.15.119:0/1555385272 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8098220 con 0x55b98d718580 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.053+0000 7f2dcea00640 1 -- 172.21.15.119:0/1555385272 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd809b000 con 0x55b98d718580 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1555385272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 msgr2=0x55b98d717ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8098580 tx=0x7f2dd809bc70 comp rx=0 tx=0).stop 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1555385272 shutdown_connections 2024-03-28T18:46:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d718580 0x55b98d717ed0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f9a0 0x55b98d717990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/1555385272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d70f080 0x55b98d70b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1555385272 >> 172.21.15.119:0/1555385272 conn(0x55b98d7097b0 msgr2=0x55b98d70ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1555385272 shutdown_connections 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- 172.21.15.119:0/1555385272 wait complete. 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.054+0000 7f2de7bf5740 1 -- start start 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 Processor -- start 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d72c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d78c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01b07d0 con 0x55b98d718580 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01b0170 con 0x55b98d70f080 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01ad670 con 0x55b98d70f9a0 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2de7bf5740 10 client.0 _ll_register_callbacks cb 0x7f2dd00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2ddde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2dd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d72c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2ddde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:36954/0 (socket says 172.21.15.119:36954) 2024-03-28T18:46:09.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2ddde00640 1 -- 172.21.15.119:0/591225552 learned_addr learned my addr 172.21.15.119:0/591225552 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.055+0000 7f2dde800640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d78c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 -- 172.21.15.119:0/591225552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 msgr2=0x55b98d72c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d72c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 -- 172.21.15.119:0/591225552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 msgr2=0x55b98d78c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d78c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 -- 172.21.15.119:0/591225552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b98d78c720 con 0x55b98d70f080 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2dd5400640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d72c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2ddde00640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55b98d71b4e0 tx=0x7f2dd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7853b7429efc060a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2dc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd002f070 con 0x55b98d70f080 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dd0004480 con 0x55b98d70f080 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2dd8098220 con 0x55b98d70f080 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dd003b050 con 0x55b98d70f080 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2de7bf5740 10 client.26060 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.056+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b98d78d860 con 0x55b98d70f080 2024-03-28T18:46:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8098220 con 0x55b98d70f080 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2dd001f430 con 0x55b98d70f080 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 10 client.26060 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 10 client.26060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 20 client.26060 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d78df40 0x55b98d790360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 4 client.26060 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b98d67b740 con 0x55b98d78df40 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2dd5400640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d78df40 0x55b98d790360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2dc3e00640 10 client.26060 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.057+0000 7f2dd5400640 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d78df40 0x55b98d790360 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52122e958719527f server_cookie=b84c01752aba56b3 in_seq=0 out_seq=0 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.058+0000 7f2dc3e00640 1 -- 172.21.15.119:0/591225552 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b98d67b740 con 0x55b98d78df40 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.058+0000 7f2dc3e00640 10 client.26060 handle_command_reply: tid=0 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.058+0000 7f2de7bf5740 1 client.26060 shutdown 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 msgr2=0x55b98d72bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55b98d71b4e0 tx=0x7f2dd00046c0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 shutdown_connections 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b98d78df40 0x55b98d790360 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b98d718580 0x55b98d78c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b98d70f9a0 0x55b98d72c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 --2- 172.21.15.119:0/591225552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b98d70f080 0x55b98d72bff0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 >> 172.21.15.119:0/591225552 conn(0x55b98d7097b0 msgr2=0x55b98d70b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2dc3e00640 0 client.26060 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 shutdown_connections 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 1 -- 172.21.15.119:0/591225552 wait complete. 2024-03-28T18:46:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.059+0000 7f2de7bf5740 20 client.26060 trim_cache size 0 max 16384 2024-03-28T18:46:09.071 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:09.071 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:46:09.071 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "29bbbbb3-3afe-4d28-9ac4-4afacff68acc", 2024-03-28T18:46:09.071 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:46:09.071 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 msgr2=0x7f2dc40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 0x7f2dc40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55b98d70eb50 tx=0x7f2dd0005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 msgr2=0x7f2de019ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de019ea30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009fd0 tx=0x7f2dd800ef10 comp rx=0 tx=0).stop 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 shutdown_connections 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dc40a2970 0x7f2dc40a4e30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de0109850 0x7f2de019ea30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de0104200 0x7f2de019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 --2- 172.21.15.119:0/891542956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de0103840 0x7f2de019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 >> 172.21.15.119:0/891542956 conn(0x7f2de00ff350 msgr2=0x7f2de00ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 shutdown_connections 2024-03-28T18:46:09.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.060+0000 7f2de5000640 1 -- 172.21.15.119:0/891542956 wait complete. 2024-03-28T18:46:09.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:46:09.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:46:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 -- 172.21.15.119:0/257772311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c10cf80 msgr2=0x7f760c114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 --2- 172.21.15.119:0/257772311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c10cf80 0x7f760c114b50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7604009ca0 tx=0x7f760402c100 comp rx=0 tx=0).stop 2024-03-28T18:46:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 -- 172.21.15.119:0/257772311 shutdown_connections 2024-03-28T18:46:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 --2- 172.21.15.119:0/257772311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c10cf80 0x7f760c114b50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 --2- 172.21.15.119:0/257772311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c108860 0x7f760c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 --2- 172.21.15.119:0/257772311 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c107eb0 0x7f760c108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 -- 172.21.15.119:0/257772311 >> 172.21.15.119:0/257772311 conn(0x7f760c075be0 msgr2=0x7f760c078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 -- 172.21.15.119:0/257772311 shutdown_connections 2024-03-28T18:46:09.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.334+0000 7f7611600640 1 -- 172.21.15.119:0/257772311 wait complete. 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 Processor -- start 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 -- start start 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c10cf80 0x7f760c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c1a2df0 0x7f760c1a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c118030 con 0x7f760c1a2df0 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c117eb0 con 0x7f760c10cf80 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f7611600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1181b0 con 0x7f760c107eb0 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f760aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f760aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48574/0 (socket says 172.21.15.119:48574) 2024-03-28T18:46:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.335+0000 7f760aa00640 1 -- 172.21.15.119:0/1181245053 learned_addr learned my addr 172.21.15.119:0/1181245053 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7602000640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c10cf80 0x7f760c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760b400640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c1a2df0 0x7f760c1a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 -- 172.21.15.119:0/1181245053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c10cf80 msgr2=0x7f760c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c10cf80 0x7f760c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 -- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c1a2df0 msgr2=0x7f760c1a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c1a2df0 0x7f760c1a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 -- 172.21.15.119:0/1181245053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c1a38e0 con 0x7f760c107eb0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f760aa00640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00ea60 tx=0x7f75fc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7603e00640 1 -- 172.21.15.119:0/1181245053 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00cde0 con 0x7f760c107eb0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7603e00640 1 -- 172.21.15.119:0/1181245053 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75fc0035f0 con 0x7f760c107eb0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7603e00640 1 -- 172.21.15.119:0/1181245053 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc010670 con 0x7f760c107eb0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c1a3bd0 con 0x7f760c107eb0 2024-03-28T18:46:09.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.336+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c1ab410 con 0x7f760c107eb0 2024-03-28T18:46:09.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.340+0000 7f7614282740 1 Processor -- start 2024-03-28T18:46:09.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.340+0000 7f7614282740 1 -- start start 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c0892303a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c08923b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c089194250 con 0x55c08922ff20 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c08913ca30 con 0x55c089239290 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7614282740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0891902f0 con 0x55c089230c00 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7603e00640 1 -- 172.21.15.119:0/1181245053 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f75fc0037a0 con 0x7f760c107eb0 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f760b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c08923b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f760b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48578/0 (socket says 172.21.15.119:48578) 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f760b400640 1 -- 172.21.15.119:0/3522528027 learned_addr learned my addr 172.21.15.119:0/3522528027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f760aa00640 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c0892303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 -- 172.21.15.119:0/3522528027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 msgr2=0x55c08922f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 -- 172.21.15.119:0/3522528027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 msgr2=0x55c0892303a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c0892303a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f7602000640 1 -- 172.21.15.119:0/3522528027 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c089127690 con 0x55c089239290 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.341+0000 7f760b400640 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f7602000640 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c08923b680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f760c06acf0 tx=0x7f75f4011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a72bef5f4e338720 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f75fb400640 1 -- 172.21.15.119:0/3522528027 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4004310 con 0x55c089239290 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f75fb400640 1 -- 172.21.15.119:0/3522528027 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75f40044b0 con 0x55c089239290 2024-03-28T18:46:09.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f75fb400640 1 -- 172.21.15.119:0/3522528027 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4019ae0 con 0x55c089239290 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f7614282740 1 -- 172.21.15.119:0/3522528027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 msgr2=0x55c08923b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.342+0000 7f7614282740 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c08923b680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f760c06acf0 tx=0x7f75f4011440 comp rx=0 tx=0).stop 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 -- 172.21.15.119:0/3522528027 shutdown_connections 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c08923b680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c0892303a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 --2- 172.21.15.119:0/3522528027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c08922f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 -- 172.21.15.119:0/3522528027 >> 172.21.15.119:0/3522528027 conn(0x55c0891f7200 msgr2=0x55c08922a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 -- 172.21.15.119:0/3522528027 shutdown_connections 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7603e00640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 0x7f75f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 -- 172.21.15.119:0/3522528027 wait complete. 2024-03-28T18:46:09.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7603e00640 1 -- 172.21.15.119:0/1181245053 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f75fc0cf1f0 con 0x7f760c107eb0 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7602000640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 0x7f75f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 -- start start 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.343+0000 7f7614282740 1 Processor -- start 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7602000640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 0x7f75f00a4e60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75f40195c0 tx=0x7f75f40193e0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c089235ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c0892361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c08923c100 con 0x55c08922ff20 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c08914fe80 con 0x55c089239290 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c08923c280 con 0x55c089230c00 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7614282740 10 client.0 _ll_register_callbacks cb 0x55c089162130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33936/0 (socket says 172.21.15.119:33936) 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760aa00640 1 -- 172.21.15.119:0/3110178815 learned_addr learned my addr 172.21.15.119:0/3110178815 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c0892361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f7602000640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c089235ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 -- 172.21.15.119:0/3110178815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 msgr2=0x55c089235ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c089235ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 -- 172.21.15.119:0/3110178815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 msgr2=0x55c089238910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760b400640 1 -- 172.21.15.119:0/3110178815 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c088f727d0 con 0x55c089239290 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.344+0000 7f760aa00640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f760b400640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c0892361e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7604006010 tx=0x7f76040323c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8abe47e97c358b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f75efe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604043070 con 0x55c089239290 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f76040329a0 con 0x55c089239290 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75f40044b0 con 0x55c089239290 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f760404f050 con 0x55c089239290 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f7614282740 10 client.26072 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:09.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c088ea83f0 con 0x55c089239290 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.345+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75f40044b0 con 0x55c089239290 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f76040045c0 con 0x55c089239290 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 10 client.26072 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 10 client.26072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 20 client.26072 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c089249080 0x55c08924b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 4 client.26072 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0891a1790 con 0x55c089249080 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f760aa00640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c089249080 0x55c08924b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f760aa00640 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c089249080 0x55c08924b490 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d4d1805eb956a09 server_cookie=b286d89d1bd5b8b2 in_seq=0 out_seq=0 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.346+0000 7f75efe00640 10 client.26072 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f75efe00640 1 -- 172.21.15.119:0/3110178815 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c0891a1790 con 0x55c089249080 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f75efe00640 10 client.26072 handle_command_reply: tid=0 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 client.26072 shutdown 2024-03-28T18:46:09.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 msgr2=0x55c0892361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c0892361e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7604006010 tx=0x7f76040323c0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 shutdown_connections 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c089249080 0x55c08924b490 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089239290 0x55c0892361e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089230c00 0x55c089235ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f75efe00640 0 client.26072 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 --2- 172.21.15.119:0/3110178815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c08922ff20 0x55c089238910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 >> 172.21.15.119:0/3110178815 conn(0x55c0891f7200 msgr2=0x55c08922c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.347+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 shutdown_connections 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.348+0000 7f7614282740 1 -- 172.21.15.119:0/3110178815 wait complete. 2024-03-28T18:46:09.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.348+0000 7f7614282740 20 client.26072 trim_cache size 0 max 16384 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 Processor -- start 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 -- start start 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 0x55c089238910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 0x55c08923a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b39b0 con 0x55c089230c00 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b5330 con 0x55c08922ff20 2024-03-28T18:46:09.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7614282740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b4010 con 0x55c089239290 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f760aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f7602000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 0x55c089238910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f760b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 0x55c08923a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f760aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37014/0 (socket says 172.21.15.119:37014) 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.365+0000 7f760aa00640 1 -- 172.21.15.119:0/3279427480 learned_addr learned my addr 172.21.15.119:0/3279427480 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 -- 172.21.15.119:0/3279427480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 msgr2=0x55c08923a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 0x55c08923a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 -- 172.21.15.119:0/3279427480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 msgr2=0x55c089238910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 0x55c089238910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 -- 172.21.15.119:0/3279427480 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c088e680f0 con 0x55c08922ff20 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760b400640 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 0x55c08923a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f7602000640 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 0x55c089238910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f760aa00640 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0981c0 tx=0x7f75fc0cf510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf4f7291949af7b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f75faa00640 1 -- 172.21.15.119:0/3279427480 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc09b330 con 0x55c08922ff20 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f75faa00640 1 -- 172.21.15.119:0/3279427480 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75fc09bef0 con 0x55c08922ff20 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.366+0000 7f75faa00640 1 -- 172.21.15.119:0/3279427480 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc09b150 con 0x55c08922ff20 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- 172.21.15.119:0/3279427480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 msgr2=0x55c0892383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0981c0 tx=0x7f75fc0cf510 comp rx=0 tx=0).stop 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- 172.21.15.119:0/3279427480 shutdown_connections 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c089239290 0x55c08923a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089230c00 0x55c089238910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 --2- 172.21.15.119:0/3279427480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c08922ff20 0x55c0892383d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- 172.21.15.119:0/3279427480 >> 172.21.15.119:0/3279427480 conn(0x55c0891f7200 msgr2=0x55c08922a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- 172.21.15.119:0/3279427480 shutdown_connections 2024-03-28T18:46:09.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- 172.21.15.119:0/3279427480 wait complete. 2024-03-28T18:46:09.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 -- start start 2024-03-28T18:46:09.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.367+0000 7f7614282740 1 Processor -- start 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089239290 0x55c08923ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b4cd0 con 0x55c089239290 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b4670 con 0x55c089230c00 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1b1b70 con 0x55c08922ff20 2024-03-28T18:46:09.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7614282740 10 client.0 _ll_register_callbacks cb 0x7f75f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f760aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37030/0 (socket says 172.21.15.119:37030) 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f760aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:48602/0 (socket says 172.21.15.119:48602) 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 -- 172.21.15.119:0/1327627212 learned_addr learned my addr 172.21.15.119:0/1327627212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f760b400640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089239290 0x55c08923ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 -- 172.21.15.119:0/1327627212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 msgr2=0x55c08923e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 -- 172.21.15.119:0/1327627212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089239290 msgr2=0x55c08923ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.368+0000 7f7602000640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089239290 0x55c08923ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f7602000640 1 -- 172.21.15.119:0/1327627212 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c089233d30 con 0x55c089230c00 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f760aa00640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f7602000640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f75f4019690 tx=0x7f75f4011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdc45aedf54d765a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f75efe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:09.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f401f040 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75fc09bef0 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f75f40038c0 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75f4003ac0 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f7614282740 10 client.26078 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.369+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c0892a59e0 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75fc09bef0 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75f4032070 con 0x55c089230c00 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 10 client.26078 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 10 client.26078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 20 client.26078 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c0892a5f40 0x55c0892a8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 4 client.26078 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c08919bae0 con 0x55c0892a5f40 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f760aa00640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c0892a5f40 0x55c0892a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:09.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f760aa00640 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c0892a5f40 0x55c0892a8360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73d2b316cc8e48b3 server_cookie=afd61ccae4a2f53c in_seq=0 out_seq=0 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.370+0000 7f75efe00640 10 client.26078 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f75efe00640 1 -- 172.21.15.119:0/1327627212 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c08919bae0 con 0x55c0892a5f40 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f75efe00640 10 client.26078 handle_command_reply: tid=0 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 client.26078 shutdown 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 msgr2=0x55c08923ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f75f4019690 tx=0x7f75f4011440 comp rx=0 tx=0).stop 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 shutdown_connections 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c0892a5f40 0x55c0892a8360 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c089239290 0x55c08923ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f75efe00640 0 client.26078 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c089230c00 0x55c08923ea20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 --2- 172.21.15.119:0/1327627212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c08922ff20 0x55c08923e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 >> 172.21.15.119:0/1327627212 conn(0x55c0891f7200 msgr2=0x55c08922bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 shutdown_connections 2024-03-28T18:46:09.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 1 -- 172.21.15.119:0/1327627212 wait complete. 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.371+0000 7f7614282740 20 client.26078 trim_cache size 0 max 16384 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "29bbbbb3-3afe-4d28-9ac4-4afacff68acc": { 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "29bbbbb3-3afe-4d28-9ac4-4afacff68acc", 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:46:09.376 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 msgr2=0x7f75f00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 0x7f75f00a4e60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75f40195c0 tx=0x7f75f40193e0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 msgr2=0x7f760c19e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00ea60 tx=0x7f75fc00ef20 comp rx=0 tx=0).stop 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 shutdown_connections 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75f00a29a0 0x7f75f00a4e60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760c1a2df0 0x7f760c1a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f760c10cf80 0x7f760c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 --2- 172.21.15.119:0/1181245053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f760c107eb0 0x7f760c19e530 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:09.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 >> 172.21.15.119:0/1181245053 conn(0x7f760c075be0 msgr2=0x7f760c077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:09.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 shutdown_connections 2024-03-28T18:46:09.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:09.372+0000 7f7611600640 1 -- 172.21.15.119:0/1181245053 wait complete. 2024-03-28T18:46:09.387 INFO:tasks.cephfs.filesystem:scrub status for tag:29bbbbb3-3afe-4d28-9ac4-4afacff68acc - {'path': '/', 'tag': '29bbbbb3-3afe-4d28-9ac4-4afacff68acc', 'options': 'recursive,force'} 2024-03-28T18:46:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:09 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:46:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:09 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:09 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:46:09.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:09 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:09.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:09 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:46:09.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:09 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:09.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:09 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:46:09.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:09 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:09 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:46:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:09 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:09 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:46:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:09 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:10 smithi119 ceph-mon[31813]: pgmap v66: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T18:46:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:10 smithi196 ceph-mon[36300]: pgmap v66: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T18:46:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:10 smithi131 ceph-mon[36065]: pgmap v66: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 171 MiB/s wr, 94 op/s 2024-03-28T18:46:11.294 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:11.441 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 readers = 4849240.00 kB/sec 2024-03-28T18:46:11.442 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 readers = 4828190.98 kB/sec 2024-03-28T18:46:11.442 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 4849240.00 kB/sec 2024-03-28T18:46:11.442 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 4849240.00 kB/sec 2024-03-28T18:46:11.442 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 4849240.00 kB/sec 2024-03-28T18:46:11.442 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:46:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:12 smithi119 ceph-mon[31813]: pgmap v67: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:12.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:12 smithi196 ceph-mon[36300]: pgmap v67: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:12.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:12 smithi131 ceph-mon[36065]: pgmap v67: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:13.774 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:13.774 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 re-readers = 3117933.00 kB/sec 2024-03-28T18:46:13.774 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 re-readers = 3109701.07 kB/sec 2024-03-28T18:46:13.774 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 3117933.00 kB/sec 2024-03-28T18:46:13.775 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 3117933.00 kB/sec 2024-03-28T18:46:13.775 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 3117933.00 kB/sec 2024-03-28T18:46:13.775 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 1048576.00 kB 2024-03-28T18:46:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:13 smithi119 ceph-mon[31813]: pgmap v68: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:13 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:46:13.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:13 smithi196 ceph-mon[36300]: pgmap v68: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:13.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:13 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:46:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:13 smithi131 ceph-mon[36065]: pgmap v68: 97 pgs: 97 active+clean; 2.0 GiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 171 MiB/s wr, 67 op/s 2024-03-28T18:46:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:13 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:46:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:14 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:14.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:14 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:14 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:46:15.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:15 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:15.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:15 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:15.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:15 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:46:15.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:15 smithi119 ceph-mon[31813]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 71 op/s 2024-03-28T18:46:15.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:15 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:15 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:15 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:46:15.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:15 smithi196 ceph-mon[36300]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 71 op/s 2024-03-28T18:46:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:15 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:46:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:15 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:46:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:15 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:46:15.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:15 smithi131 ceph-mon[36065]: pgmap v69: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 171 MiB/s wr, 71 op/s 2024-03-28T18:46:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:18 smithi119 ceph-mon[31813]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:46:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:18 smithi196 ceph-mon[36300]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:46:18.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:18 smithi131 ceph-mon[36065]: pgmap v70: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-03-28T18:46:20.530 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.531 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.531 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.531 INFO:tasks.workunit.client.0.smithi119.stdout:iozone test complete. 2024-03-28T18:46:20.533 INFO:tasks.workunit.client.0.smithi119.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T18:46:20.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:20 smithi119 ceph-mon[31813]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-03-28T18:46:20.624 INFO:tasks.workunit.client.0.smithi119.stdout: Iozone: Performance Test of File I/O 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Version $Revision: 3.494 $ 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Compiled for 64 bit mode. 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Build: linux-AMD64 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-28T18:46:20.625 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Run began: Thu Mar 28 18:46:20 2024 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Include close in write timing 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Include fsync in write timing 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: File size set to 10485760 kB 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Record Size 1024 kB 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Output is in kBytes/sec 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Time Resolution = 0.000001 seconds. 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache size set to 1024 kBytes. 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Processor cache line size set to 32 bytes. 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: File stride size set to 17 * record size. 2024-03-28T18:46:20.626 INFO:tasks.workunit.client.0.smithi119.stdout: Throughput test with 1 process 2024-03-28T18:46:20.627 INFO:tasks.workunit.client.0.smithi119.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-03-28T18:46:20.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:20 smithi196 ceph-mon[36300]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-03-28T18:46:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:20 smithi131 ceph-mon[36065]: pgmap v71: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-03-28T18:46:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:22 smithi119 ceph-mon[31813]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:22.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:22 smithi196 ceph-mon[36300]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:22.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:22 smithi131 ceph-mon[36065]: pgmap v72: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:24.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:24 smithi119 ceph-mon[31813]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:24.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:24 smithi196 ceph-mon[36300]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:24.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:24 smithi131 ceph-mon[36065]: pgmap v73: 97 pgs: 97 active+clean; 2.0 GiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-03-28T18:46:26.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:26 smithi196 ceph-mon[36300]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 38 op/s 2024-03-28T18:46:26.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:26 smithi131 ceph-mon[36065]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 38 op/s 2024-03-28T18:46:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:26 smithi119 ceph-mon[31813]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 38 op/s 2024-03-28T18:46:28.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:28 smithi196 ceph-mon[36300]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 34 op/s 2024-03-28T18:46:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:28 smithi131 ceph-mon[36065]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 34 op/s 2024-03-28T18:46:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:28 smithi119 ceph-mon[31813]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 34 op/s 2024-03-28T18:46:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:30 smithi196 ceph-mon[36300]: pgmap v76: 97 pgs: 97 active+clean; 2.5 GiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 125 MiB/s wr, 69 op/s 2024-03-28T18:46:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:30 smithi131 ceph-mon[36065]: pgmap v76: 97 pgs: 97 active+clean; 2.5 GiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 125 MiB/s wr, 69 op/s 2024-03-28T18:46:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:30 smithi119 ceph-mon[31813]: pgmap v76: 97 pgs: 97 active+clean; 2.5 GiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 125 MiB/s wr, 69 op/s 2024-03-28T18:46:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:32 smithi196 ceph-mon[36300]: pgmap v77: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:32 smithi131 ceph-mon[36065]: pgmap v77: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:32 smithi119 ceph-mon[31813]: pgmap v77: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:34 smithi196 ceph-mon[36300]: pgmap v78: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:34 smithi131 ceph-mon[36065]: pgmap v78: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:34 smithi119 ceph-mon[31813]: pgmap v78: 97 pgs: 97 active+clean; 2.9 GiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 161 MiB/s wr, 73 op/s 2024-03-28T18:46:36.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:36 smithi196 ceph-mon[36300]: pgmap v79: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 312 MiB/s wr, 116 op/s 2024-03-28T18:46:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:36 smithi131 ceph-mon[36065]: pgmap v79: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 312 MiB/s wr, 116 op/s 2024-03-28T18:46:36.828 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:36 smithi119 ceph-mon[31813]: pgmap v79: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 312 MiB/s wr, 116 op/s 2024-03-28T18:46:38.630 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:38 smithi119 ceph-mon[31813]: pgmap v80: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:38.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:38 smithi196 ceph-mon[36300]: pgmap v80: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:38 smithi131 ceph-mon[36065]: pgmap v80: 97 pgs: 97 active+clean; 4.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:39.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:46:39.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:46:39.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.682+0000 7ff525e00640 1 -- 172.21.15.119:0/2984718013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff520108910 msgr2=0x7ff520108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.682+0000 7ff525e00640 1 --2- 172.21.15.119:0/2984718013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff520108910 0x7ff520108cf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff510009ca0 tx=0x7ff51002c120 comp rx=0 tx=0).stop 2024-03-28T18:46:39.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.686+0000 7ff525e00640 1 -- 172.21.15.119:0/2984718013 shutdown_connections 2024-03-28T18:46:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.686+0000 7ff525e00640 1 --2- 172.21.15.119:0/2984718013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff52010cfe0 0x7ff5200799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.686+0000 7ff525e00640 1 --2- 172.21.15.119:0/2984718013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5201092c0 0x7ff52010c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.686+0000 7ff525e00640 1 --2- 172.21.15.119:0/2984718013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff520108910 0x7ff520108cf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.686+0000 7ff525e00640 1 -- 172.21.15.119:0/2984718013 >> 172.21.15.119:0/2984718013 conn(0x7ff52006c780 msgr2=0x7ff52006cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.692+0000 7ff525e00640 1 -- 172.21.15.119:0/2984718013 shutdown_connections 2024-03-28T18:46:39.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.692+0000 7ff525e00640 1 -- 172.21.15.119:0/2984718013 wait complete. 2024-03-28T18:46:39.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.693+0000 7ff525e00640 1 Processor -- start 2024-03-28T18:46:39.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.694+0000 7ff525e00640 1 -- start start 2024-03-28T18:46:39.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.694+0000 7ff525e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff525e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff52010cfe0 0x7ff52007d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff525e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 0x7ff52007dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39008/0 (socket says 172.21.15.119:39008) 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff51ea00640 1 -- 172.21.15.119:0/3307349315 learned_addr learned my addr 172.21.15.119:0/3307349315 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52007bbf0 con 0x7ff52010cfe0 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52007ba70 con 0x7ff52007d740 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.695+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52007bd70 con 0x7ff520108910 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.696+0000 7ff51f400640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 0x7ff52007dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.696+0000 7ff51f400640 1 -- 172.21.15.119:0/3307349315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 msgr2=0x7ff520083140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.696+0000 7ff51f400640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.697+0000 7ff51f400640 1 -- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff52010cfe0 msgr2=0x7ff52007d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.697+0000 7ff51f400640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff52010cfe0 0x7ff52007d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.697+0000 7ff51f400640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5201ccc30 con 0x7ff52007d740 2024-03-28T18:46:39.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.697+0000 7ff51ea00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:39.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.697+0000 7ff51f400640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 0x7ff52007dbf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b410 tx=0x7ff51800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.698+0000 7ff524a00640 1 -- 172.21.15.119:0/3307349315 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518015020 con 0x7ff52007d740 2024-03-28T18:46:39.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.698+0000 7ff524a00640 1 -- 172.21.15.119:0/3307349315 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff518011d10 con 0x7ff52007d740 2024-03-28T18:46:39.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.698+0000 7ff524a00640 1 -- 172.21.15.119:0/3307349315 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51800f2e0 con 0x7ff52007d740 2024-03-28T18:46:39.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.698+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5201ccf20 con 0x7ff52007d740 2024-03-28T18:46:39.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.698+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5201cd330 con 0x7ff52007d740 2024-03-28T18:46:39.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.702+0000 7ff524a00640 1 -- 172.21.15.119:0/3307349315 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ff518018020 con 0x7ff52007d740 2024-03-28T18:46:39.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.704+0000 7ff528c30740 1 Processor -- start 2024-03-28T18:46:39.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.704+0000 7ff528c30740 1 -- start start 2024-03-28T18:46:39.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff524a00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 0x7ff5040a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff524a00640 1 -- 172.21.15.119:0/3307349315 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff5180ce3f0 con 0x7ff52007d740 2024-03-28T18:46:39.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b6049950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6055680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b5fae250 con 0x5643b6053290 2024-03-28T18:46:39.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b5f56a30 con 0x5643b604ac00 2024-03-28T18:46:39.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.705+0000 7ff528c30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b5faa2f0 con 0x5643b6049f20 2024-03-28T18:46:39.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.706+0000 7ff51e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.706+0000 7ff51e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39020/0 (socket says 172.21.15.119:39020) 2024-03-28T18:46:39.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.706+0000 7ff51e000640 1 -- 172.21.15.119:0/1831985764 learned_addr learned my addr 172.21.15.119:0/1831985764 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:39.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.706+0000 7ff51ea00640 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b6049950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.706+0000 7ff51f400640 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6055680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 0x7ff5040a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 -- 172.21.15.119:0/1831985764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 msgr2=0x5643b6049950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b6049950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 -- 172.21.15.119:0/1831985764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 msgr2=0x5643b6055680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6055680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.707+0000 7ff51e000640 1 -- 172.21.15.119:0/1831985764 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643b5f41690 con 0x5643b6049f20 2024-03-28T18:46:39.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.708+0000 7ff51e000640 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff508013630 tx=0x7ff50800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f83c614caac29a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.708+0000 7ff50fe00640 1 -- 172.21.15.119:0/1831985764 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800cbd0 con 0x5643b6049f20 2024-03-28T18:46:39.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.708+0000 7ff50fe00640 1 -- 172.21.15.119:0/1831985764 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff50801ece0 con 0x5643b6049f20 2024-03-28T18:46:39.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.708+0000 7ff50fe00640 1 -- 172.21.15.119:0/1831985764 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800cbd0 con 0x5643b6049f20 2024-03-28T18:46:39.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.709+0000 7ff528c30740 1 -- 172.21.15.119:0/1831985764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 msgr2=0x5643b604a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.709+0000 7ff528c30740 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff508013630 tx=0x7ff50800cf90 comp rx=0 tx=0).stop 2024-03-28T18:46:39.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.710+0000 7ff51e000640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 0x7ff5040a4ff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff50801e000 tx=0x7ff508017000 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 -- 172.21.15.119:0/1831985764 shutdown_connections 2024-03-28T18:46:39.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6055680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b604a3a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 --2- 172.21.15.119:0/1831985764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b6049950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 -- 172.21.15.119:0/1831985764 >> 172.21.15.119:0/1831985764 conn(0x5643b6011200 msgr2=0x5643b6044970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 -- 172.21.15.119:0/1831985764 shutdown_connections 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 -- 172.21.15.119:0/1831985764 wait complete. 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.711+0000 7ff528c30740 1 -- start start 2024-03-28T18:46:39.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.712+0000 7ff528c30740 1 Processor -- start 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.712+0000 7ff528c30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.712+0000 7ff528c30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b604fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.712+0000 7ff528c30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6050230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.712+0000 7ff528c30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b6056100 con 0x5643b6053290 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff528c30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b5f69e80 con 0x5643b604ac00 2024-03-28T18:46:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff528c30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643b6056280 con 0x5643b6049f20 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff528c30740 10 client.0 _ll_register_callbacks cb 0x5643b5f7c130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39030/0 (socket says 172.21.15.119:39030) 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff51ea00640 1 -- 172.21.15.119:0/1997808712 learned_addr learned my addr 172.21.15.119:0/1997808712 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.713+0000 7ff51e000640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b604fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51f400640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6050230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51ea00640 1 -- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 msgr2=0x5643b604fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51ea00640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b604fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51ea00640 1 -- 172.21.15.119:0/1997808712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 msgr2=0x5643b6050230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51ea00640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6050230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51ea00640 1 -- 172.21.15.119:0/1997808712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643b5d8c7d0 con 0x5643b6049f20 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.714+0000 7ff51f400640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6050230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff51ea00640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff510032670 tx=0x7ff5100326a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e678bba6e6f071d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff50cc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510044070 con 0x5643b6049f20 2024-03-28T18:46:39.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff50801ece0 con 0x5643b6049f20 2024-03-28T18:46:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff510003b70 con 0x5643b6049f20 2024-03-28T18:46:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff510039040 con 0x5643b6049f20 2024-03-28T18:46:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff528c30740 10 client.26095 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.715+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5643b5cc23f0 con 0x5643b6049f20 2024-03-28T18:46:39.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.716+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff510003d70 con 0x5643b6049f20 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff510004560 con 0x5643b6049f20 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 10 client.26095 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 10 client.26095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 20 client.26095 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b6062f80 0x5643b6065370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 4 client.26095 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:39.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff50801ece0 con 0x5643b6062f80 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.717+0000 7ff51e000640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b6062f80 0x5643b6065370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.718+0000 7ff51e000640 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b6062f80 0x5643b6065370 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c2f2039231dd038 server_cookie=e4af669b5c38b683 in_seq=0 out_seq=0 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.718+0000 7ff50cc00640 10 client.26095 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.718+0000 7ff50cc00640 1 -- 172.21.15.119:0/1997808712 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7ff50801ece0 con 0x5643b6062f80 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.718+0000 7ff50cc00640 10 client.26095 handle_command_reply: tid=0 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.718+0000 7ff528c30740 1 client.26095 shutdown 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 msgr2=0x5643b6052960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff510032670 tx=0x7ff5100326a0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 shutdown_connections 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b6062f80 0x5643b6065370 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6053290 0x5643b6050230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b604ac00 0x5643b604fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 --2- 172.21.15.119:0/1997808712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052960 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 >> 172.21.15.119:0/1997808712 conn(0x5643b6011200 msgr2=0x5643b60466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff50cc00640 0 client.26095 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 shutdown_connections 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.719+0000 7ff528c30740 1 -- 172.21.15.119:0/1997808712 wait complete. 2024-03-28T18:46:39.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.720+0000 7ff528c30740 20 client.26095 trim_cache size 0 max 16384 2024-03-28T18:46:39.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.762+0000 7ff528c30740 1 Processor -- start 2024-03-28T18:46:39.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 -- start start 2024-03-28T18:46:39.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b604ac00 0x5643b6052960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b6043910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d5930 con 0x5643b604ac00 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d72b0 con 0x5643b6053290 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.763+0000 7ff528c30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d5f90 con 0x5643b6049f20 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39038/0 (socket says 172.21.15.119:39038) 2024-03-28T18:46:39.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51ea00640 1 -- 172.21.15.119:0/4158236669 learned_addr learned my addr 172.21.15.119:0/4158236669 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:39.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b6043910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 -- 172.21.15.119:0/4158236669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 msgr2=0x5643b6052420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 -- 172.21.15.119:0/4158236669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b604ac00 msgr2=0x5643b6052960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b604ac00 0x5643b6052960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.764+0000 7ff51f400640 1 -- 172.21.15.119:0/4158236669 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643b5c820f0 con 0x5643b6053290 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.765+0000 7ff51f400640 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b6043910 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5180cbb90 tx=0x7ff5180ce320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e965178db249f8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.765+0000 7ff50f400640 1 -- 172.21.15.119:0/4158236669 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180b2070 con 0x5643b6053290 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.765+0000 7ff50f400640 1 -- 172.21.15.119:0/4158236669 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff51809ebf0 con 0x5643b6053290 2024-03-28T18:46:39.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.765+0000 7ff50f400640 1 -- 172.21.15.119:0/4158236669 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180cbd00 con 0x5643b6053290 2024-03-28T18:46:39.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.766+0000 7ff528c30740 1 -- 172.21.15.119:0/4158236669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 msgr2=0x5643b6043910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.766+0000 7ff528c30740 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b6043910 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5180cbb90 tx=0x7ff5180ce320 comp rx=0 tx=0).stop 2024-03-28T18:46:39.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.770+0000 7ff528c30740 1 -- 172.21.15.119:0/4158236669 shutdown_connections 2024-03-28T18:46:39.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.770+0000 7ff528c30740 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b6043910 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.770+0000 7ff528c30740 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b604ac00 0x5643b6052960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.770+0000 7ff528c30740 1 --2- 172.21.15.119:0/4158236669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b6049f20 0x5643b6052420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.770+0000 7ff528c30740 1 -- 172.21.15.119:0/4158236669 >> 172.21.15.119:0/4158236669 conn(0x5643b6011200 msgr2=0x5643b6043200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.772+0000 7ff528c30740 1 -- 172.21.15.119:0/4158236669 shutdown_connections 2024-03-28T18:46:39.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.772+0000 7ff528c30740 1 -- 172.21.15.119:0/4158236669 wait complete. 2024-03-28T18:46:39.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.773+0000 7ff528c30740 1 -- start start 2024-03-28T18:46:39.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.773+0000 7ff528c30740 1 Processor -- start 2024-03-28T18:46:39.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6049f20 0x5643b605af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b60593b0 0x5643b6059830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d6c50 con 0x5643b6049f20 2024-03-28T18:46:39.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d65f0 con 0x5643b6053290 2024-03-28T18:46:39.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201d3af0 con 0x5643b60593b0 2024-03-28T18:46:39.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.774+0000 7ff528c30740 10 client.0 _ll_register_callbacks cb 0x7ff510009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:39.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39592/0 (socket says 172.21.15.119:39592) 2024-03-28T18:46:39.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 -- 172.21.15.119:0/4194597308 learned_addr learned my addr 172.21.15.119:0/4194597308 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b60593b0 0x5643b6059830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 -- 172.21.15.119:0/4194597308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b60593b0 msgr2=0x5643b6059830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b60593b0 0x5643b6059830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 -- 172.21.15.119:0/4194597308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6049f20 msgr2=0x5643b605af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6049f20 0x5643b605af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 -- 172.21.15.119:0/4194597308 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643b605a030 con 0x5643b6053290 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.775+0000 7ff51e000640 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff50800c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac53cf8cb371a211 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:39.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.776+0000 7ff50cc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:39.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.776+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508037060 con 0x5643b6053290 2024-03-28T18:46:39.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.776+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff50800cab0 con 0x5643b6053290 2024-03-28T18:46:39.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.776+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff51809ebf0 con 0x5643b6053290 2024-03-28T18:46:39.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.776+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff50800ccb0 con 0x5643b6053290 2024-03-28T18:46:39.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.780+0000 7ff528c30740 10 client.26096 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:39.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.780+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5643b60bf840 con 0x5643b6053290 2024-03-28T18:46:39.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.781+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff51809ebf0 con 0x5643b6053290 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff50800f040 con 0x5643b6053290 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 10 client.26096 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 10 client.26096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 20 client.26096 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b60bfdb0 0x5643b60c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:39.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 4 client.26096 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:46:39.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.783+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5643b5fbb790 con 0x5643b60bfdb0 2024-03-28T18:46:39.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.784+0000 7ff51ea00640 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b60bfdb0 0x5643b60c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:39.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.785+0000 7ff51ea00640 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b60bfdb0 0x5643b60c21d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a5446609c754e8 server_cookie=43fd5a854c8f85c8 in_seq=0 out_seq=0 2024-03-28T18:46:39.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.785+0000 7ff50cc00640 10 client.26096 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:39.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.786+0000 7ff50cc00640 1 -- 172.21.15.119:0/4194597308 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5643b5fbb790 con 0x5643b60bfdb0 2024-03-28T18:46:39.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff50cc00640 10 client.26096 handle_command_reply: tid=0 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 client.26096 shutdown 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 msgr2=0x5643b605b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff508000c00 tx=0x7ff50800c690 comp rx=0 tx=0).stop 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 shutdown_connections 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5643b60bfdb0 0x5643b60c21d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5643b60593b0 0x5643b6059830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5643b6053290 0x5643b605b4b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 --2- 172.21.15.119:0/4194597308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5643b6049f20 0x5643b605af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff50cc00640 0 client.26096 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:39.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 >> 172.21.15.119:0/4194597308 conn(0x5643b6011200 msgr2=0x5643b6043a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.787+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 shutdown_connections 2024-03-28T18:46:39.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.788+0000 7ff528c30740 1 -- 172.21.15.119:0/4194597308 wait complete. 2024-03-28T18:46:39.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.801+0000 7ff528c30740 20 client.26096 trim_cache size 0 max 16384 2024-03-28T18:46:39.802 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:39.803 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:46:39.803 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:46:39.803 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.804+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 msgr2=0x7ff5040a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.804+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 0x7ff5040a4ff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff50801e000 tx=0x7ff508017000 comp rx=0 tx=0).stop 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.804+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 msgr2=0x7ff52007dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.804+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 0x7ff52007dbf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b410 tx=0x7ff51800b8d0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 shutdown_connections 2024-03-28T18:46:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff5040a2b30 0x7ff5040a4ff0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff52007d740 0x7ff52007dbf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff52010cfe0 0x7ff52007d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 --2- 172.21.15.119:0/3307349315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff520108910 0x7ff520083140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 >> 172.21.15.119:0/3307349315 conn(0x7ff52006c780 msgr2=0x7ff5200715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.806+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 shutdown_connections 2024-03-28T18:46:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:39.807+0000 7ff525e00640 1 -- 172.21.15.119:0/3307349315 wait complete. 2024-03-28T18:46:39.842 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:46:39.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:46:40.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 -- 172.21.15.119:0/1094939805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc11ae80 msgr2=0x7f84dc11d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:40.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 --2- 172.21.15.119:0/1094939805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc11ae80 0x7f84dc11d270 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009be0 tx=0x7f84d402bef0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 -- 172.21.15.119:0/1094939805 shutdown_connections 2024-03-28T18:46:40.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 --2- 172.21.15.119:0/1094939805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc11ae80 0x7f84dc11d270 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 --2- 172.21.15.119:0/1094939805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc10df00 0x7f84dc10e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 --2- 172.21.15.119:0/1094939805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10fed0 0x7f84dc10d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 -- 172.21.15.119:0/1094939805 >> 172.21.15.119:0/1094939805 conn(0x7f84dc06c790 msgr2=0x7f84dc06cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:40.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 -- 172.21.15.119:0/1094939805 shutdown_connections 2024-03-28T18:46:40.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.200+0000 7f84e2200640 1 -- 172.21.15.119:0/1094939805 wait complete. 2024-03-28T18:46:40.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.202+0000 7f84e2200640 1 Processor -- start 2024-03-28T18:46:40.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.202+0000 7f84e2200640 1 -- start start 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10df00 0x7f84dc1ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc10fed0 0x7f84dc1ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc1207c0 con 0x7f84dc10df00 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc120640 con 0x7f84dc11ae80 2024-03-28T18:46:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.203+0000 7f84e2200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc120940 con 0x7f84dc10fed0 2024-03-28T18:46:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.204+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.204+0000 7f84dbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39610/0 (socket says 172.21.15.119:39610) 2024-03-28T18:46:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.204+0000 7f84dbe00640 1 -- 172.21.15.119:0/4025191578 learned_addr learned my addr 172.21.15.119:0/4025191578 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.204+0000 7f84daa00640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc10fed0 0x7f84dc1ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84db400640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10df00 0x7f84dc1ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84dbe00640 1 -- 172.21.15.119:0/4025191578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc10fed0 msgr2=0x7f84dc1ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84dbe00640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc10fed0 0x7f84dc1ace70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84dbe00640 1 -- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10df00 msgr2=0x7f84dc1ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84dbe00640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10df00 0x7f84dc1ac930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.205+0000 7f84dbe00640 1 -- 172.21.15.119:0/4025191578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc1a7260 con 0x7f84dc11ae80 2024-03-28T18:46:40.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.206+0000 7f84dbe00640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f84d4033420 tx=0x7f84d4003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:40.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.206+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4043020 con 0x7f84dc11ae80 2024-03-28T18:46:40.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.206+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84d40344d0 con 0x7f84dc11ae80 2024-03-28T18:46:40.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.206+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40336e0 con 0x7f84dc11ae80 2024-03-28T18:46:40.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.206+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc1a74f0 con 0x7f84dc11ae80 2024-03-28T18:46:40.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.207+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc110660 con 0x7f84dc11ae80 2024-03-28T18:46:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.209+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc068490 con 0x7f84dc11ae80 2024-03-28T18:46:40.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.214+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f84d4046020 con 0x7f84dc11ae80 2024-03-28T18:46:40.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.218+0000 7f84d3e00640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 0x7f84c00a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:40.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.218+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84d40f1ac0 con 0x7f84dc11ae80 2024-03-28T18:46:40.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.219+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f84d404d020 con 0x7f84dc11ae80 2024-03-28T18:46:40.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.219+0000 7f84db400640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 0x7f84c00a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:40.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.220+0000 7f84db400640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 0x7f84c00a5030 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009ae0 tx=0x7f84cc005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:40.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.660+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84dc110830 con 0x7f84dc11ae80 2024-03-28T18:46:40.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.662+0000 7f84d3e00640 1 -- 172.21.15.119:0/4025191578 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f84d40be8f0 con 0x7f84dc11ae80 2024-03-28T18:46:40.662 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:46:40.663 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:46:40.663 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:46:40.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.666+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 msgr2=0x7f84c00a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:40.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.666+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 0x7f84c00a5030 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009ae0 tx=0x7f84cc005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:40.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.666+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 msgr2=0x7f84dc1a6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:40.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.666+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f84d4033420 tx=0x7f84d4003f80 comp rx=0 tx=0).stop 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 shutdown_connections 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84c00a2b70 0x7f84c00a5030 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84dc11ae80 0x7f84dc1a6ae0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84dc10fed0 0x7f84dc1ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 --2- 172.21.15.119:0/4025191578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84dc10df00 0x7f84dc1ac930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 >> 172.21.15.119:0/4025191578 conn(0x7f84dc06c790 msgr2=0x7f84dc072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 shutdown_connections 2024-03-28T18:46:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:40.667+0000 7f84e2200640 1 -- 172.21.15.119:0/4025191578 wait complete. 2024-03-28T18:46:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:40 smithi196 ceph-mon[36300]: pgmap v81: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 440 MiB/s wr, 127 op/s 2024-03-28T18:46:40.692 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:40 smithi119 ceph-mon[31813]: pgmap v81: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 440 MiB/s wr, 127 op/s 2024-03-28T18:46:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:40 smithi131 ceph-mon[36065]: pgmap v81: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 440 MiB/s wr, 127 op/s 2024-03-28T18:46:40.696 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:40.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 -- 172.21.15.119:0/1591654661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 msgr2=0x7f25781148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 --2- 172.21.15.119:0/1591654661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781148d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2568009930 tx=0x7f256802bc30 comp rx=0 tx=0).stop 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 -- 172.21.15.119:0/1591654661 shutdown_connections 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 --2- 172.21.15.119:0/1591654661 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781148d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 --2- 172.21.15.119:0/1591654661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25780764f0 0x7f2578111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 --2- 172.21.15.119:0/1591654661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2578075bd0 0x7f2578075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 -- 172.21.15.119:0/1591654661 >> 172.21.15.119:0/1591654661 conn(0x7f25780ff350 msgr2=0x7f2578101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 -- 172.21.15.119:0/1591654661 shutdown_connections 2024-03-28T18:46:41.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.035+0000 7f2580600640 1 -- 172.21.15.119:0/1591654661 wait complete. 2024-03-28T18:46:41.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.038+0000 7f2580600640 1 Processor -- start 2024-03-28T18:46:41.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.038+0000 7f2580600640 1 -- start start 2024-03-28T18:46:41.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.038+0000 7f2580600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.039+0000 7f2580600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25780764f0 0x7f257819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.039+0000 7f2580600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.039+0000 7f2580600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578118040 con 0x7f2578075bd0 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.039+0000 7f2580600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2578117ec0 con 0x7f25781124e0 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.039+0000 7f2580600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781181c0 con 0x7f25780764f0 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.040+0000 7f257dc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.040+0000 7f257dc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39746/0 (socket says 172.21.15.119:39746) 2024-03-28T18:46:41.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.040+0000 7f257dc00640 1 -- 172.21.15.119:0/3610597374 learned_addr learned my addr 172.21.15.119:0/3610597374 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257e600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257d200640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25780764f0 0x7f257819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257dc00640 1 -- 172.21.15.119:0/3610597374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25780764f0 msgr2=0x7f257819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257dc00640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25780764f0 0x7f257819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257dc00640 1 -- 172.21.15.119:0/3610597374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 msgr2=0x7f25781a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257dc00640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.041+0000 7f257dc00640 1 -- 172.21.15.119:0/3610597374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25781a3560 con 0x7f2578075bd0 2024-03-28T18:46:41.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.042+0000 7f257dc00640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f257000d950 tx=0x7f257000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.042+0000 7f2576000640 1 -- 172.21.15.119:0/3610597374 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570017070 con 0x7f2578075bd0 2024-03-28T18:46:41.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.042+0000 7f2576000640 1 -- 172.21.15.119:0/3610597374 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2570009c30 con 0x7f2578075bd0 2024-03-28T18:46:41.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.042+0000 7f2576000640 1 -- 172.21.15.119:0/3610597374 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257000f6b0 con 0x7f2578075bd0 2024-03-28T18:46:41.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.043+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25781a3850 con 0x7f2578075bd0 2024-03-28T18:46:41.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.043+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25781ab0e0 con 0x7f2578075bd0 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.048+0000 7f258336d740 1 Processor -- start 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 -- start start 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144ee0 0x556cee15a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee0a82c0 con 0x556cee15ac40 2024-03-28T18:46:41.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee050c90 con 0x556cee144ee0 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.049+0000 7f258336d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee0a4360 con 0x556cee144530 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.050+0000 7f257d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.050+0000 7f257e600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.050+0000 7f257e600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39094/0 (socket says 172.21.15.119:39094) 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.050+0000 7f257e600640 1 -- 172.21.15.119:0/954630317 learned_addr learned my addr 172.21.15.119:0/954630317 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.051+0000 7f257e600640 1 -- 172.21.15.119:0/954630317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144ee0 msgr2=0x556cee15a620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.051+0000 7f257e600640 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144ee0 0x556cee15a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.051+0000 7f257e600640 1 -- 172.21.15.119:0/954630317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 msgr2=0x556cee15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.051+0000 7f257e600640 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.051+0000 7f257e600640 1 -- 172.21.15.119:0/954630317 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cee075080 con 0x556cee144530 2024-03-28T18:46:41.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.052+0000 7f257e600640 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2568036040 tx=0x7f2568009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=345a2d94c7652512 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.052+0000 7f2566a00640 1 -- 172.21.15.119:0/954630317 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568065020 con 0x556cee144530 2024-03-28T18:46:41.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.053+0000 7f2566a00640 1 -- 172.21.15.119:0/954630317 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256802c880 con 0x556cee144530 2024-03-28T18:46:41.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.053+0000 7f2566a00640 1 -- 172.21.15.119:0/954630317 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25680054d0 con 0x556cee144530 2024-03-28T18:46:41.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.054+0000 7f258336d740 1 -- 172.21.15.119:0/954630317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 msgr2=0x556cee144910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.054+0000 7f258336d740 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2568036040 tx=0x7f2568009560 comp rx=0 tx=0).stop 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 -- 172.21.15.119:0/954630317 shutdown_connections 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144ee0 0x556cee15a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 --2- 172.21.15.119:0/954630317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee144910 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 -- 172.21.15.119:0/954630317 >> 172.21.15.119:0/954630317 conn(0x556cee13eda0 msgr2=0x556cee13f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f2576000640 1 -- 172.21.15.119:0/3610597374 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f25700109d0 con 0x7f2578075bd0 2024-03-28T18:46:41.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.055+0000 7f258336d740 1 -- 172.21.15.119:0/954630317 shutdown_connections 2024-03-28T18:46:41.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.056+0000 7f258336d740 1 -- 172.21.15.119:0/954630317 wait complete. 2024-03-28T18:46:41.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.057+0000 7f258336d740 1 -- start start 2024-03-28T18:46:41.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.059+0000 7f2576000640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 0x7f25600a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.060+0000 7f2576000640 1 -- 172.21.15.119:0/3610597374 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f25700ceb40 con 0x7f2578075bd0 2024-03-28T18:46:41.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.061+0000 7f257e600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 0x7f25600a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.062+0000 7f258336d740 1 Processor -- start 2024-03-28T18:46:41.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.063+0000 7f257e600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 0x7f25600a4f40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2568009850 tx=0x7f2568009680 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.064+0000 7f258336d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.064+0000 7f258336d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee15ac40 0x556cee14c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f258336d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee146f80 0x556cee147400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f258336d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee1457b0 con 0x556cee146f80 2024-03-28T18:46:41.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f258336d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee0640d0 con 0x556cee144530 2024-03-28T18:46:41.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f258336d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cee145930 con 0x556cee15ac40 2024-03-28T18:46:41.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f258336d740 10 client.0 _ll_register_callbacks cb 0x556cee068e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f257dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f257e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee146f80 0x556cee147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f257dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39642/0 (socket says 172.21.15.119:39642) 2024-03-28T18:46:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.065+0000 7f257dc00640 1 -- 172.21.15.119:0/3646166293 learned_addr learned my addr 172.21.15.119:0/3646166293 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257d200640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee15ac40 0x556cee14c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257dc00640 1 -- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee15ac40 msgr2=0x556cee14c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257dc00640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee15ac40 0x556cee14c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257dc00640 1 -- 172.21.15.119:0/3646166293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee146f80 msgr2=0x556cee147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257dc00640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee146f80 0x556cee147400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.066+0000 7f257dc00640 1 -- 172.21.15.119:0/3646166293 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ceddbc3f0 con 0x556cee144530 2024-03-28T18:46:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f257dc00640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2570097540 tx=0x7f2570009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5e6193cb36fb6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f255b400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570020040 con 0x556cee144530 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f25700b9be0 con 0x556cee144530 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f256802c880 con 0x556cee144530 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25700cc650 con 0x556cee144530 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f258336d740 10 client.26108 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556cede867d0 con 0x556cee144530 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.067+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f256802c880 con 0x556cee144530 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.068+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f257009a6c0 con 0x556cee144530 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.068+0000 7f258336d740 10 client.26108 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.068+0000 7f258336d740 10 client.26108 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.069+0000 7f258336d740 20 client.26108 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.069+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee157010 0x556cee159470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.069+0000 7f258336d740 4 client.26108 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.069+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556cee0b5800 con 0x556cee157010 2024-03-28T18:46:41.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.069+0000 7f257d200640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee157010 0x556cee159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.070+0000 7f257d200640 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee157010 0x556cee159470 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d4c6dfef466776 server_cookie=4b13ac4c44dc0593 in_seq=0 out_seq=0 2024-03-28T18:46:41.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.070+0000 7f255b400640 10 client.26108 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.070+0000 7f255b400640 1 -- 172.21.15.119:0/3646166293 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556cee0b5800 con 0x556cee157010 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.070+0000 7f255b400640 10 client.26108 handle_command_reply: tid=0 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.070+0000 7f258336d740 1 client.26108 shutdown 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 msgr2=0x556cee14bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2570097540 tx=0x7f2570009ea0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 shutdown_connections 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee157010 0x556cee159470 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee146f80 0x556cee147400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee15ac40 0x556cee14c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 --2- 172.21.15.119:0/3646166293 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee14bc20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f255b400640 0 client.26108 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 >> 172.21.15.119:0/3646166293 conn(0x556cee13eda0 msgr2=0x556cee140af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 shutdown_connections 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 1 -- 172.21.15.119:0/3646166293 wait complete. 2024-03-28T18:46:41.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.071+0000 7f258336d740 20 client.26108 trim_cache size 0 max 16384 2024-03-28T18:46:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.089+0000 7f258336d740 1 Processor -- start 2024-03-28T18:46:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.089+0000 7f258336d740 1 -- start start 2024-03-28T18:46:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.089+0000 7f258336d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.089+0000 7f258336d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee146f80 0x556cee14bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f258336d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f258336d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781ab310 con 0x556cee15ac40 2024-03-28T18:46:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f258336d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781acb70 con 0x556cee146f80 2024-03-28T18:46:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f258336d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781ab850 con 0x556cee144530 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39114/0 (socket says 172.21.15.119:39114) 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 -- 172.21.15.119:0/2716255629 learned_addr learned my addr 172.21.15.119:0/2716255629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257d200640 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee146f80 0x556cee14bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257e600640 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 -- 172.21.15.119:0/2716255629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee146f80 msgr2=0x556cee14bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee146f80 0x556cee14bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 -- 172.21.15.119:0/2716255629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 msgr2=0x556cee15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.090+0000 7f257dc00640 1 -- 172.21.15.119:0/2716255629 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cee161000 con 0x556cee144530 2024-03-28T18:46:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.091+0000 7f257dc00640 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f257009b020 tx=0x7f2570010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f06ab17a97acc5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.091+0000 7f2566000640 1 -- 172.21.15.119:0/2716255629 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25700a7030 con 0x556cee144530 2024-03-28T18:46:41.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.091+0000 7f2566000640 1 -- 172.21.15.119:0/2716255629 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25700a3880 con 0x556cee144530 2024-03-28T18:46:41.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.091+0000 7f2566000640 1 -- 172.21.15.119:0/2716255629 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257009a530 con 0x556cee144530 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- 172.21.15.119:0/2716255629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 msgr2=0x556cee14b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f257009b020 tx=0x7f2570010d70 comp rx=0 tx=0).stop 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- 172.21.15.119:0/2716255629 shutdown_connections 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee146f80 0x556cee14bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 --2- 172.21.15.119:0/2716255629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee144530 0x556cee14b7d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- 172.21.15.119:0/2716255629 >> 172.21.15.119:0/2716255629 conn(0x556cee13eda0 msgr2=0x556cee13deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- 172.21.15.119:0/2716255629 shutdown_connections 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- 172.21.15.119:0/2716255629 wait complete. 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 -- start start 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.092+0000 7f258336d740 1 Processor -- start 2024-03-28T18:46:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee166470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee146f80 0x556cee163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781ac510 con 0x556cee15ac40 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25781abeb0 con 0x556cee144530 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257810b940 con 0x556cee146f80 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.093+0000 7f258336d740 10 client.0 _ll_register_callbacks cb 0x7f257009ab10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.094+0000 7f257e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.094+0000 7f257e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39778/0 (socket says 172.21.15.119:39778) 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.094+0000 7f257e600640 1 -- 172.21.15.119:0/2081517955 learned_addr learned my addr 172.21.15.119:0/2081517955 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.094+0000 7f257dc00640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 -- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee146f80 msgr2=0x556cee163850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee146f80 0x556cee163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 -- 172.21.15.119:0/2081517955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 msgr2=0x556cee166470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee166470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 -- 172.21.15.119:0/2081517955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cee1642d0 con 0x556cee15ac40 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f257e600640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f256802ceb0 tx=0x7f2568009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1593e46beaa3a301 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.095+0000 7f255b400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:41.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256806e070 con 0x556cee15ac40 2024-03-28T18:46:41.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2568003c80 con 0x556cee15ac40 2024-03-28T18:46:41.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f25700a3880 con 0x556cee15ac40 2024-03-28T18:46:41.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2568034440 con 0x556cee15ac40 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f258336d740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.096+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556cee1ba300 con 0x556cee15ac40 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25700a3880 con 0x556cee15ac40 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f256806a5b0 con 0x556cee15ac40 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 10 client.15951 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 20 client.15951 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee1ba5f0 0x556cee1bc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 4 client.15951 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.097+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556cee0afb50 con 0x556cee1ba5f0 2024-03-28T18:46:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f257dc00640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee1ba5f0 0x556cee1bc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f257dc00640 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee1ba5f0 0x556cee1bc9e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4c25e2bf65fcde90 server_cookie=7fedf2d73e0e4443 in_seq=0 out_seq=0 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f255b400640 10 client.15951 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f255b400640 1 -- 172.21.15.119:0/2081517955 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556cee0afb50 con 0x556cee1ba5f0 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f255b400640 10 client.15951 handle_command_reply: tid=0 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f258336d740 1 client.15951 shutdown 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 msgr2=0x556cee163d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.098+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f256802ceb0 tx=0x7f2568009560 comp rx=0 tx=0).stop 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 shutdown_connections 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x556cee1ba5f0 0x556cee1bc9e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556cee15ac40 0x556cee163d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556cee146f80 0x556cee163850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 --2- 172.21.15.119:0/2081517955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556cee144530 0x556cee166470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 >> 172.21.15.119:0/2081517955 conn(0x556cee13eda0 msgr2=0x556cee13ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f255b400640 0 client.15951 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:46:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 shutdown_connections 2024-03-28T18:46:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 1 -- 172.21.15.119:0/2081517955 wait complete. 2024-03-28T18:46:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.099+0000 7f258336d740 20 client.15951 trim_cache size 0 max 16384 2024-03-28T18:46:41.101 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 msgr2=0x7f25600a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 0x7f25600a4f40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2568009850 tx=0x7f2568009680 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 msgr2=0x7f257819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f257000d950 tx=0x7f257000de10 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 shutdown_connections 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f25600a2a80 0x7f25600a4f40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25781124e0 0x7f25781a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25780764f0 0x7f257819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 --2- 172.21.15.119:0/3610597374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2578075bd0 0x7f257819e5c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 >> 172.21.15.119:0/3610597374 conn(0x7f25780ff350 msgr2=0x7f2578101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 shutdown_connections 2024-03-28T18:46:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.100+0000 7f2580600640 1 -- 172.21.15.119:0/3610597374 wait complete. 2024-03-28T18:46:41.126 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:41.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 -- 172.21.15.119:0/3705923689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101530 msgr2=0x7fa1981019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 --2- 172.21.15.119:0/3705923689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101530 0x7fa1981019b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa184009930 tx=0x7fa18402bc20 comp rx=0 tx=0).stop 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 -- 172.21.15.119:0/3705923689 shutdown_connections 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 --2- 172.21.15.119:0/3705923689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa198101ef0 0x7fa19810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 --2- 172.21.15.119:0/3705923689 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101530 0x7fa1981019b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 --2- 172.21.15.119:0/3705923689 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1981075f0 0x7fa1981079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.425+0000 7fa197200640 1 -- 172.21.15.119:0/3705923689 >> 172.21.15.119:0/3705923689 conn(0x7fa1980fd040 msgr2=0x7fa1980ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.426+0000 7fa197200640 1 -- 172.21.15.119:0/3705923689 shutdown_connections 2024-03-28T18:46:41.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.426+0000 7fa197200640 1 -- 172.21.15.119:0/3705923689 wait complete. 2024-03-28T18:46:41.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.427+0000 7fa197200640 1 Processor -- start 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.427+0000 7fa197200640 1 -- start start 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 0x7fa198072e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 0x7fa19806c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198111900 con 0x7fa1981075f0 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198111780 con 0x7fa198101ef0 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa197200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa198111a80 con 0x7fa198101530 2024-03-28T18:46:41.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.428+0000 7fa196800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 0x7fa19806c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.429+0000 7fa195400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.429+0000 7fa195400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39668/0 (socket says 172.21.15.119:39668) 2024-03-28T18:46:41.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.429+0000 7fa195400640 1 -- 172.21.15.119:0/3700052655 learned_addr learned my addr 172.21.15.119:0/3700052655 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.429+0000 7fa195e00640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 0x7fa198072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.429+0000 7fa195400640 1 -- 172.21.15.119:0/3700052655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 msgr2=0x7fa198072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195400640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 0x7fa198072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195400640 1 -- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 msgr2=0x7fa19806c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195400640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 0x7fa19806c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195400640 1 -- 172.21.15.119:0/3700052655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19806c930 con 0x7fa198101ef0 2024-03-28T18:46:41.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195e00640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 0x7fa198072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:41.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa196800640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 0x7fa19806c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:41.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.431+0000 7fa195400640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184004340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.432+0000 7fa18ea00640 1 -- 172.21.15.119:0/3700052655 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184043020 con 0x7fa198101ef0 2024-03-28T18:46:41.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.434+0000 7fa18ea00640 1 -- 172.21.15.119:0/3700052655 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1840054e0 con 0x7fa198101ef0 2024-03-28T18:46:41.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.434+0000 7fa18ea00640 1 -- 172.21.15.119:0/3700052655 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184003ab0 con 0x7fa198101ef0 2024-03-28T18:46:41.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.434+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa19806cbc0 con 0x7fa198101ef0 2024-03-28T18:46:41.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.434+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198070b10 con 0x7fa198101ef0 2024-03-28T18:46:41.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.439+0000 7fa19fdff740 1 Processor -- start 2024-03-28T18:46:41.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.440+0000 7fa19fdff740 1 -- start start 2024-03-28T18:46:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.440+0000 7fa18ea00640 1 -- 172.21.15.119:0/3700052655 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa184046020 con 0x7fa198101ef0 2024-03-28T18:46:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.440+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.441+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486ab0 0x55d16c472a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.441+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c473050 0x55d16c4734b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa19fdff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c3dc2c0 con 0x55d16c486100 2024-03-28T18:46:41.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa19fdff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c384c90 con 0x55d16c486ab0 2024-03-28T18:46:41.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa19fdff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c3d8360 con 0x55d16c473050 2024-03-28T18:46:41.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa196800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa196800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39820/0 (socket says 172.21.15.119:39820) 2024-03-28T18:46:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa196800640 1 -- 172.21.15.119:0/2539718596 learned_addr learned my addr 172.21.15.119:0/2539718596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.442+0000 7fa18ea00640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 0x7fa17c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa18ea00640 1 -- 172.21.15.119:0/3700052655 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa1840f0de0 con 0x7fa198101ef0 2024-03-28T18:46:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa196800640 1 -- 172.21.15.119:0/2539718596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c473050 msgr2=0x55d16c4734b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa196800640 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c473050 0x55d16c4734b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa196800640 1 -- 172.21.15.119:0/2539718596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486ab0 msgr2=0x55d16c472a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa196800640 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486ab0 0x55d16c472a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.443+0000 7fa196800640 1 -- 172.21.15.119:0/2539718596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d16c3a9080 con 0x55d16c486100 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.444+0000 7fa196800640 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa190012570 tx=0x7fa190012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8816b8fef778841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.444+0000 7fa196800640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 0x7fa17c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.445+0000 7fa183400640 1 -- 172.21.15.119:0/2539718596 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19001cad0 con 0x55d16c486100 2024-03-28T18:46:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.445+0000 7fa183400640 1 -- 172.21.15.119:0/2539718596 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa190021070 con 0x55d16c486100 2024-03-28T18:46:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.445+0000 7fa183400640 1 -- 172.21.15.119:0/2539718596 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190024380 con 0x55d16c486100 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 -- 172.21.15.119:0/2539718596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 msgr2=0x55d16c4864e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa196800640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 0x7fa17c0a4f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa190020000 tx=0x7fa19001e000 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa190012570 tx=0x7fa190012a30 comp rx=0 tx=0).stop 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 -- 172.21.15.119:0/2539718596 shutdown_connections 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c473050 0x55d16c4734b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486ab0 0x55d16c472a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 --2- 172.21.15.119:0/2539718596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c4864e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 -- 172.21.15.119:0/2539718596 >> 172.21.15.119:0/2539718596 conn(0x55d16c4754f0 msgr2=0x55d16c4758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 -- 172.21.15.119:0/2539718596 shutdown_connections 2024-03-28T18:46:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.446+0000 7fa19fdff740 1 -- 172.21.15.119:0/2539718596 wait complete. 2024-03-28T18:46:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.447+0000 7fa19fdff740 1 -- start start 2024-03-28T18:46:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.447+0000 7fa19fdff740 1 Processor -- start 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c48caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c48d540 con 0x55d16c473050 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c3980d0 con 0x55d16c486100 2024-03-28T18:46:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16c48d6c0 con 0x55d16c486ab0 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa19fdff740 10 client.0 _ll_register_callbacks cb 0x55d16c39ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa195400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa195400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39694/0 (socket says 172.21.15.119:39694) 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.448+0000 7fa195400640 1 -- 172.21.15.119:0/3230001778 learned_addr learned my addr 172.21.15.119:0/3230001778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 -- 172.21.15.119:0/3230001778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 msgr2=0x55d16c48a570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 -- 172.21.15.119:0/3230001778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 msgr2=0x55d16c48caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c48caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 -- 172.21.15.119:0/3230001778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d16c0f03f0 con 0x55d16c486100 2024-03-28T18:46:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa195400640 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa18402ce80 tx=0x7fa1840bcdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0bc08a08ec8067e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa177e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1840bd490 con 0x55d16c486100 2024-03-28T18:46:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa1840ee6f0 con 0x55d16c486100 2024-03-28T18:46:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.449+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa190021070 con 0x55d16c486100 2024-03-28T18:46:41.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.450+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1840ee8f0 con 0x55d16c486100 2024-03-28T18:46:41.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.451+0000 7fa19fdff740 10 client.26123 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.451+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d16c1ba7d0 con 0x55d16c486100 2024-03-28T18:46:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.451+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa190021070 con 0x55d16c486100 2024-03-28T18:46:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa184003c50 con 0x55d16c486100 2024-03-28T18:46:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 10 client.26123 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 10 client.26123 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:46:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 20 client.26123 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c49a410 0x55d16c49c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 4 client.26123 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.452+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d16c3e9800 con 0x55d16c49a410 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.453+0000 7fa195e00640 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c49a410 0x55d16c49c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.453+0000 7fa195e00640 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c49a410 0x55d16c49c820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be761eceffeba564 server_cookie=10fea331371d923f in_seq=0 out_seq=0 2024-03-28T18:46:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.453+0000 7fa177e00640 10 client.26123 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.454+0000 7fa177e00640 1 -- 172.21.15.119:0/3230001778 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d16c3e9800 con 0x55d16c49a410 2024-03-28T18:46:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.454+0000 7fa177e00640 10 client.26123 handle_command_reply: tid=0 2024-03-28T18:46:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.454+0000 7fa19fdff740 1 client.26123 shutdown 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.454+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 msgr2=0x55d16c489e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.454+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa18402ce80 tx=0x7fa1840bcdf0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 shutdown_connections 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c489e30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c49a410 0x55d16c49c820 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3230001778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c48caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 >> 172.21.15.119:0/3230001778 conn(0x55d16c4754f0 msgr2=0x55d16c472160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa177e00640 0 client.26123 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 shutdown_connections 2024-03-28T18:46:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.455+0000 7fa19fdff740 1 -- 172.21.15.119:0/3230001778 wait complete. 2024-03-28T18:46:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.456+0000 7fa19fdff740 20 client.26123 trim_cache size 0 max 16384 2024-03-28T18:46:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.480+0000 7fa19fdff740 1 Processor -- start 2024-03-28T18:46:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.480+0000 7fa19fdff740 1 -- start start 2024-03-28T18:46:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.480+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.480+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c484290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa19fdff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b3750 con 0x55d16c486100 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa19fdff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b50d0 con 0x55d16c473050 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa19fdff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b3db0 con 0x55d16c486ab0 2024-03-28T18:46:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39698/0 (socket says 172.21.15.119:39698) 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 -- 172.21.15.119:0/1813682330 learned_addr learned my addr 172.21.15.119:0/1813682330 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195400640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c484290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa196800640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 -- 172.21.15.119:0/1813682330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 msgr2=0x55d16c48c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 -- 172.21.15.119:0/1813682330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 msgr2=0x55d16c484290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c484290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195e00640 1 -- 172.21.15.119:0/1813682330 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d16c4738d0 con 0x55d16c473050 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.481+0000 7fa195400640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c484290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.482+0000 7fa195e00640 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55d16c474320 tx=0x7fa188009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56532b3f832d5e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.482+0000 7fa182a00640 1 -- 172.21.15.119:0/1813682330 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188012450 con 0x55d16c473050 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa182a00640 1 -- 172.21.15.119:0/1813682330 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa188021070 con 0x55d16c473050 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa182a00640 1 -- 172.21.15.119:0/1813682330 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188011b10 con 0x55d16c473050 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 -- 172.21.15.119:0/1813682330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 msgr2=0x55d16c488ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55d16c474320 tx=0x7fa188009750 comp rx=0 tx=0).stop 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 -- 172.21.15.119:0/1813682330 shutdown_connections 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c48c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c486100 0x55d16c484290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 --2- 172.21.15.119:0/1813682330 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c473050 0x55d16c488ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.483+0000 7fa19fdff740 1 -- 172.21.15.119:0/1813682330 >> 172.21.15.119:0/1813682330 conn(0x55d16c4754f0 msgr2=0x55d16c483b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.484+0000 7fa19fdff740 1 -- 172.21.15.119:0/1813682330 shutdown_connections 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.484+0000 7fa19fdff740 1 -- 172.21.15.119:0/1813682330 wait complete. 2024-03-28T18:46:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.484+0000 7fa19fdff740 1 -- start start 2024-03-28T18:46:41.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.484+0000 7fa19fdff740 1 Processor -- start 2024-03-28T18:46:41.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c490a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c4f4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b4a70 con 0x55d16c473050 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b4410 con 0x55d16c486100 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1981b1910 con 0x55d16c486ab0 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa19fdff740 10 client.0 _ll_register_callbacks cb 0x7fa1840b90b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa195e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa195e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39862/0 (socket says 172.21.15.119:39862) 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa195e00640 1 -- 172.21.15.119:0/3737940256 learned_addr learned my addr 172.21.15.119:0/3737940256 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.485+0000 7fa196800640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c4f4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195400640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c490a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195e00640 1 -- 172.21.15.119:0/3737940256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 msgr2=0x55d16c4f4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195e00640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c4f4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195e00640 1 -- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 msgr2=0x55d16c490a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195e00640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c490a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195e00640 1 -- 172.21.15.119:0/3737940256 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d16c4f5200 con 0x55d16c473050 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.486+0000 7fa195400640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c490a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.487+0000 7fa195e00640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa18801be60 tx=0x7fa1880099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843f6310aa066777 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.487+0000 7fa177e00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.487+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188011aa0 con 0x55d16c473050 2024-03-28T18:46:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.487+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa188021070 con 0x55d16c473050 2024-03-28T18:46:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.488+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa188011c40 con 0x55d16c473050 2024-03-28T18:46:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.488+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa188011e60 con 0x55d16c473050 2024-03-28T18:46:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.488+0000 7fa19fdff740 10 client.15966 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.488+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d16c4f6500 con 0x55d16c473050 2024-03-28T18:46:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa188021070 con 0x55d16c473050 2024-03-28T18:46:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa188037660 con 0x55d16c473050 2024-03-28T18:46:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 10 client.15966 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 10 client.15966 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:46:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 20 client.15966 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c4f6b10 0x55d16c4f8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 4 client.15966 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.489+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d16c3e3b50 con 0x55d16c4f6b10 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.490+0000 7fa195400640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c4f6b10 0x55d16c4f8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.490+0000 7fa195400640 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c4f6b10 0x55d16c4f8f30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=199dbc371ee92a79 server_cookie=937ea2286f97fc0b in_seq=0 out_seq=0 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.490+0000 7fa177e00640 10 client.15966 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.490+0000 7fa177e00640 1 -- 172.21.15.119:0/3737940256 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d16c3e3b50 con 0x55d16c4f6b10 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.490+0000 7fa177e00640 10 client.15966 handle_command_reply: tid=0 2024-03-28T18:46:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 client.15966 shutdown 2024-03-28T18:46:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 msgr2=0x55d16c490500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa18801be60 tx=0x7fa1880099b0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 shutdown_connections 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d16c486ab0 0x55d16c4f4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d16c486100 0x55d16c490a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55d16c4f6b10 0x55d16c4f8f30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 --2- 172.21.15.119:0/3737940256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d16c473050 0x55d16c490500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 >> 172.21.15.119:0/3737940256 conn(0x55d16c4754f0 msgr2=0x55d16c484410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.491+0000 7fa177e00640 0 client.15966 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 shutdown_connections 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa19fdff740 1 -- 172.21.15.119:0/3737940256 wait complete. 2024-03-28T18:46:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa19fdff740 20 client.15966 trim_cache size 0 max 16384 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 msgr2=0x7fa17c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 0x7fa17c0a4f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa190020000 tx=0x7fa19001e000 comp rx=0 tx=0).stop 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 msgr2=0x7fa198073350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.492+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa184000c00 tx=0x7fa184004340 comp rx=0 tx=0).stop 2024-03-28T18:46:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 shutdown_connections 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa17c0a2ab0 0x7fa17c0a4f70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1981075f0 0x7fa19806c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa198101ef0 0x7fa198073350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 --2- 172.21.15.119:0/3700052655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa198101530 0x7fa198072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 >> 172.21.15.119:0/3700052655 conn(0x7fa1980fd040 msgr2=0x7fa1980fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 shutdown_connections 2024-03-28T18:46:41.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.493+0000 7fa197200640 1 -- 172.21.15.119:0/3700052655 wait complete. 2024-03-28T18:46:41.509 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:41.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:46:41.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:41 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4025191578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:41.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:41 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4025191578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:41.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.747+0000 7f48a2400640 1 -- 172.21.15.119:0/3850823543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c068de0 msgr2=0x7f489c069240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.747+0000 7f48a2400640 1 --2- 172.21.15.119:0/3850823543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c068de0 0x7f489c069240 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4884009930 tx=0x7f488402bc40 comp rx=0 tx=0).stop 2024-03-28T18:46:41.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.749+0000 7f48a2400640 1 -- 172.21.15.119:0/3850823543 shutdown_connections 2024-03-28T18:46:41.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.749+0000 7f48a2400640 1 --2- 172.21.15.119:0/3850823543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.749+0000 7f48a2400640 1 --2- 172.21.15.119:0/3850823543 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c068de0 0x7f489c069240 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.749+0000 7f48a2400640 1 --2- 172.21.15.119:0/3850823543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c1065f0 0x7f489c0688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.749+0000 7f48a2400640 1 -- 172.21.15.119:0/3850823543 >> 172.21.15.119:0/3850823543 conn(0x7f489c0776a0 msgr2=0x7f489c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.753+0000 7f48a2400640 1 -- 172.21.15.119:0/3850823543 shutdown_connections 2024-03-28T18:46:41.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.753+0000 7f48a2400640 1 -- 172.21.15.119:0/3850823543 wait complete. 2024-03-28T18:46:41.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.756+0000 7f48a2400640 1 Processor -- start 2024-03-28T18:46:41.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.756+0000 7f48a2400640 1 -- start start 2024-03-28T18:46:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.757+0000 7f48a2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 0x7f489c114850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.757+0000 7f48a2400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c114d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.757+0000 7f48a2400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 0x7f489c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f489aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c114d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f48a2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1180c0 con 0x7f489c068de0 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f489be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 0x7f489c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f489aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c114d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39176/0 (socket says 172.21.15.119:39176) 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f489aa00640 1 -- 172.21.15.119:0/3464054010 learned_addr learned my addr 172.21.15.119:0/3464054010 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f48a2400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c117f40 con 0x7f489c1065f0 2024-03-28T18:46:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f489b400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 0x7f489c114850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.758+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c118240 con 0x7f489c104160 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489be00640 1 -- 172.21.15.119:0/3464054010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 msgr2=0x7f489c114d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489be00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c114d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489be00640 1 -- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 msgr2=0x7f489c114850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489be00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 0x7f489c114850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489be00640 1 -- 172.21.15.119:0/3464054010 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c10e350 con 0x7f489c1065f0 2024-03-28T18:46:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.759+0000 7f489b400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 0x7f489c114850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:41.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.761+0000 7f489be00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 0x7f489c10da90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f489400b410 tx=0x7f489400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.763+0000 7f4893e00640 1 -- 172.21.15.119:0/3464054010 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4894015020 con 0x7f489c1065f0 2024-03-28T18:46:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.764+0000 7f4893e00640 1 -- 172.21.15.119:0/3464054010 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4894011d10 con 0x7f489c1065f0 2024-03-28T18:46:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.764+0000 7f4893e00640 1 -- 172.21.15.119:0/3464054010 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400f2e0 con 0x7f489c1065f0 2024-03-28T18:46:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.764+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c10e640 con 0x7f489c1065f0 2024-03-28T18:46:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.764+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c112610 con 0x7f489c1065f0 2024-03-28T18:46:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.772+0000 7f48a51c6740 1 Processor -- start 2024-03-28T18:46:41.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.778+0000 7f4893e00640 1 -- 172.21.15.119:0/3464054010 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4894018020 con 0x7f489c1065f0 2024-03-28T18:46:41.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.781+0000 7f48a51c6740 1 -- start start 2024-03-28T18:46:41.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.781+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb66d0 0x562c80eb6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.781+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.781+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.782+0000 7f48a51c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80e1a2c0 con 0x562c80eb7080 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.782+0000 7f48a51c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80dc2c90 con 0x562c80ebfc00 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.782+0000 7f48a51c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80e16360 con 0x562c80eb66d0 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.782+0000 7f4893e00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 0x7f48800a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.782+0000 7f4893e00640 1 -- 172.21.15.119:0/3464054010 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f48940ce550 con 0x7f489c1065f0 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.783+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.783+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39728/0 (socket says 172.21.15.119:39728) 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.783+0000 7f489aa00640 1 -- 172.21.15.119:0/3280716693 learned_addr learned my addr 172.21.15.119:0/3280716693 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.783+0000 7f489aa00640 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb66d0 0x562c80eb6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489aa00640 1 -- 172.21.15.119:0/3280716693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb66d0 msgr2=0x562c80eb6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489aa00640 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb66d0 0x562c80eb6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489aa00640 1 -- 172.21.15.119:0/3280716693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 msgr2=0x562c80ebf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489aa00640 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489aa00640 1 -- 172.21.15.119:0/3280716693 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c80de7080 con 0x562c80ebfc00 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.784+0000 7f489be00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 0x7f48800a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.785+0000 7f489aa00640 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4884009900 tx=0x7f4884005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a591b047696e28b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.785+0000 7f4890c00640 1 -- 172.21.15.119:0/3280716693 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488402ca90 con 0x562c80ebfc00 2024-03-28T18:46:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.785+0000 7f4890c00640 1 -- 172.21.15.119:0/3280716693 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4884004ba0 con 0x562c80ebfc00 2024-03-28T18:46:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.785+0000 7f4890c00640 1 -- 172.21.15.119:0/3280716693 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884003c10 con 0x562c80ebfc00 2024-03-28T18:46:41.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.788+0000 7f48a51c6740 1 -- 172.21.15.119:0/3280716693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 msgr2=0x562c80ec2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.788+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4884009900 tx=0x7f4884005a00 comp rx=0 tx=0).stop 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f489be00640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 0x7f48800a5030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48940a2e70 tx=0x7f48940046c0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f48a51c6740 1 -- 172.21.15.119:0/3280716693 shutdown_connections 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80ebfc00 0x562c80ec2050 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3280716693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb66d0 0x562c80eb6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.789+0000 7f48a51c6740 1 -- 172.21.15.119:0/3280716693 >> 172.21.15.119:0/3280716693 conn(0x562c80eb0da0 msgr2=0x562c80eb1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.790+0000 7f48a51c6740 1 -- 172.21.15.119:0/3280716693 shutdown_connections 2024-03-28T18:46:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.790+0000 7f48a51c6740 1 -- 172.21.15.119:0/3280716693 wait complete. 2024-03-28T18:46:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.790+0000 7f48a51c6740 1 -- start start 2024-03-28T18:46:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.791+0000 7f48a51c6740 1 Processor -- start 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80ebd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80ec2b10 con 0x562c80eb7080 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80dd60d0 con 0x562c80eb66d0 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c80ec2c90 con 0x562c80ebfc00 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f48a51c6740 10 client.0 _ll_register_callbacks cb 0x562c80ddae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.792+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39742/0 (socket says 172.21.15.119:39742) 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 -- 172.21.15.119:0/3913719426 learned_addr learned my addr 172.21.15.119:0/3913719426 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80ebd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 -- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 msgr2=0x562c80ebd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80ebd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 -- 172.21.15.119:0/3913719426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 msgr2=0x562c80ebc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.793+0000 7f489aa00640 1 -- 172.21.15.119:0/3913719426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c80b2e3f0 con 0x562c80eb66d0 2024-03-28T18:46:41.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.794+0000 7f489aa00640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4884001eb0 tx=0x7f488402cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d706a7e0934417 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.794+0000 7f4889600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:41.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.794+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840038b0 con 0x562c80eb66d0 2024-03-28T18:46:41.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.794+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4884003a50 con 0x562c80eb66d0 2024-03-28T18:46:41.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.795+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4884004ba0 con 0x562c80eb66d0 2024-03-28T18:46:41.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.796+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4884003c50 con 0x562c80eb66d0 2024-03-28T18:46:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.796+0000 7f48a51c6740 10 client.26141 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.796+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562c80bf87d0 con 0x562c80eb66d0 2024-03-28T18:46:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.796+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4884004ba0 con 0x562c80eb66d0 2024-03-28T18:46:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4884034440 con 0x562c80eb66d0 2024-03-28T18:46:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 10 client.26141 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 10 client.26141 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:46:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 20 client.26141 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80ecf720 0x562c80ed1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 4 client.26141 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.797+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c80e27800 con 0x562c80ecf720 2024-03-28T18:46:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.798+0000 7f489b400640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80ecf720 0x562c80ed1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.800+0000 7f489b400640 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80ecf720 0x562c80ed1b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf3cd1e76700f4c5 server_cookie=c8fa0262078034f8 in_seq=0 out_seq=0 2024-03-28T18:46:41.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.800+0000 7f4889600640 10 client.26141 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.802+0000 7f4889600640 1 -- 172.21.15.119:0/3913719426 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562c80e27800 con 0x562c80ecf720 2024-03-28T18:46:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.802+0000 7f4889600640 10 client.26141 handle_command_reply: tid=0 2024-03-28T18:46:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 client.26141 shutdown 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 msgr2=0x562c80ebf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4884001eb0 tx=0x7f488402cf90 comp rx=0 tx=0).stop 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 shutdown_connections 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80ecf720 0x562c80ed1b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f4889600640 0 client.26141 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80ebd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb7080 0x562c80ebc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 --2- 172.21.15.119:0/3913719426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ebf5b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 >> 172.21.15.119:0/3913719426 conn(0x562c80eb0da0 msgr2=0x562c80eb2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.803+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 shutdown_connections 2024-03-28T18:46:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.804+0000 7f48a51c6740 1 -- 172.21.15.119:0/3913719426 wait complete. 2024-03-28T18:46:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.804+0000 7f48a51c6740 20 client.26141 trim_cache size 0 max 16384 2024-03-28T18:46:41.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:41 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4025191578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:46:41.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.836+0000 7f48a51c6740 1 Processor -- start 2024-03-28T18:46:41.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.837+0000 7f48a51c6740 1 -- start start 2024-03-28T18:46:41.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb66d0 0x562c80ecbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80eb0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1bc1c0 con 0x562c80eb66d0 2024-03-28T18:46:41.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1bdb40 con 0x562c80eb7080 2024-03-28T18:46:41.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.838+0000 7f48a51c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1bc820 con 0x562c80ebfc00 2024-03-28T18:46:41.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.840+0000 7f489b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.840+0000 7f489b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39758/0 (socket says 172.21.15.119:39758) 2024-03-28T18:46:41.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.840+0000 7f489b400640 1 -- 172.21.15.119:0/1710313190 learned_addr learned my addr 172.21.15.119:0/1710313190 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.840+0000 7f489aa00640 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80eb0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489aa00640 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb66d0 0x562c80ecbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489b400640 1 -- 172.21.15.119:0/1710313190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 msgr2=0x562c80eb0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489b400640 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80eb0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489b400640 1 -- 172.21.15.119:0/1710313190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb66d0 msgr2=0x562c80ecbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489b400640 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb66d0 0x562c80ecbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.842+0000 7f489b400640 1 -- 172.21.15.119:0/1710313190 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c80ec2c30 con 0x562c80eb7080 2024-03-28T18:46:41.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.844+0000 7f489b400640 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f488c009d90 tx=0x7f488c009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eb06956a4e75fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.845+0000 7f488be00640 1 -- 172.21.15.119:0/1710313190 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c01f070 con 0x562c80eb7080 2024-03-28T18:46:41.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.846+0000 7f488be00640 1 -- 172.21.15.119:0/1710313190 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f488c01b590 con 0x562c80eb7080 2024-03-28T18:46:41.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.846+0000 7f488be00640 1 -- 172.21.15.119:0/1710313190 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c0137a0 con 0x562c80eb7080 2024-03-28T18:46:41.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.849+0000 7f48a51c6740 1 -- 172.21.15.119:0/1710313190 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 msgr2=0x562c80eafd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.849+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f488c009d90 tx=0x7f488c009720 comp rx=0 tx=0).stop 2024-03-28T18:46:41.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.849+0000 7f48a51c6740 1 -- 172.21.15.119:0/1710313190 shutdown_connections 2024-03-28T18:46:41.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.849+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80ebfc00 0x562c80eb0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.850+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb7080 0x562c80eafd30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.850+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1710313190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80eb66d0 0x562c80ecbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.850+0000 7f48a51c6740 1 -- 172.21.15.119:0/1710313190 >> 172.21.15.119:0/1710313190 conn(0x562c80eb0da0 msgr2=0x562c80eaf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.850+0000 7f48a51c6740 1 -- 172.21.15.119:0/1710313190 shutdown_connections 2024-03-28T18:46:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.850+0000 7f48a51c6740 1 -- 172.21.15.119:0/1710313190 wait complete. 2024-03-28T18:46:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.851+0000 7f48a51c6740 1 -- start start 2024-03-28T18:46:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.851+0000 7f48a51c6740 1 Processor -- start 2024-03-28T18:46:41.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.854+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ed49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.855+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.855+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ed49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.856+0000 7f48a51c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 0x562c80ed2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.856+0000 7f48a51c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1bd4e0 con 0x562c80ebfc00 2024-03-28T18:46:41.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.856+0000 7f48a51c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1bce80 con 0x562c80eb66d0 2024-03-28T18:46:41.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.856+0000 7f48a51c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1ba380 con 0x562c80eb7080 2024-03-28T18:46:41.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.856+0000 7f48a51c6740 10 client.0 _ll_register_callbacks cb 0x7f4884004250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:41.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.860+0000 7f489aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ed49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39764/0 (socket says 172.21.15.119:39764) 2024-03-28T18:46:41.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.860+0000 7f489b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.860+0000 7f489b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39206/0 (socket says 172.21.15.119:39206) 2024-03-28T18:46:41.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.860+0000 7f489b400640 1 -- 172.21.15.119:0/1905146907 learned_addr learned my addr 172.21.15.119:0/1905146907 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:41.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.865+0000 7f489aa00640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 0x562c80ed2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.870+0000 7f489b400640 1 -- 172.21.15.119:0/1905146907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 msgr2=0x562c80ed49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.870+0000 7f489b400640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ed49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.870+0000 7f489b400640 1 -- 172.21.15.119:0/1905146907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 msgr2=0x562c80ed2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:41.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.870+0000 7f489b400640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 0x562c80ed2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.873+0000 7f489b400640 1 -- 172.21.15.119:0/1905146907 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c80ed2b70 con 0x562c80eb7080 2024-03-28T18:46:41.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.873+0000 7f489aa00640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 0x562c80ed2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:41.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.874+0000 7f489b400640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f488c003e60 tx=0x7f488c003e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc59c0a95f8c427b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:41.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.874+0000 7f4889600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:41.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.874+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c01f040 con 0x562c80eb7080 2024-03-28T18:46:41.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.874+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f488c004cc0 con 0x562c80eb7080 2024-03-28T18:46:41.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.875+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f488c01b590 con 0x562c80eb7080 2024-03-28T18:46:41.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.875+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f488c01b740 con 0x562c80eb7080 2024-03-28T18:46:41.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.875+0000 7f48a51c6740 10 client.26146 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:41.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.875+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562c80f2ad50 con 0x562c80eb7080 2024-03-28T18:46:41.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f488c01b590 con 0x562c80eb7080 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f488c00f040 con 0x562c80eb7080 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 10 client.26146 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 10 client.26146 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 20 client.26146 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80f2b3a0 0x562c80f2d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 4 client.26146 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.876+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562c80e21b50 con 0x562c80f2b3a0 2024-03-28T18:46:41.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.877+0000 7f489aa00640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80f2b3a0 0x562c80f2d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.878+0000 7f489aa00640 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80f2b3a0 0x562c80f2d7c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6a1a5970279f1c4 server_cookie=850623ed182afd80 in_seq=0 out_seq=0 2024-03-28T18:46:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.878+0000 7f4889600640 10 client.26146 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.878+0000 7f4889600640 1 -- 172.21.15.119:0/1905146907 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562c80e21b50 con 0x562c80f2b3a0 2024-03-28T18:46:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.878+0000 7f4889600640 10 client.26146 handle_command_reply: tid=0 2024-03-28T18:46:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.878+0000 7f48a51c6740 1 client.26146 shutdown 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 msgr2=0x562c80ed1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f488c003e60 tx=0x7f488c003e90 comp rx=0 tx=0).stop 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 shutdown_connections 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x562c80f2b3a0 0x562c80f2d7c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f4889600640 0 client.26146 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562c80ebfc00 0x562c80ed2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562c80eb7080 0x562c80ed1d60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 --2- 172.21.15.119:0/1905146907 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562c80eb66d0 0x562c80ed49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.879+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 >> 172.21.15.119:0/1905146907 conn(0x562c80eb0da0 msgr2=0x562c80eb2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.882+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 shutdown_connections 2024-03-28T18:46:41.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.883+0000 7f48a51c6740 1 -- 172.21.15.119:0/1905146907 wait complete. 2024-03-28T18:46:41.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.884+0000 7f48a51c6740 20 client.26146 trim_cache size 0 max 16384 2024-03-28T18:46:41.886 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 msgr2=0x7f48800a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 0x7f48800a5030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48940a2e70 tx=0x7f48940046c0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 msgr2=0x7f489c10da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 0x7f489c10da90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f489400b410 tx=0x7f489400b8d0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 shutdown_connections 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f48800a2b70 0x7f48800a5030 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f489c1065f0 0x7f489c10da90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f489c104160 0x7f489c114d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 --2- 172.21.15.119:0/3464054010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f489c068de0 0x7f489c114850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.887+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 >> 172.21.15.119:0/3464054010 conn(0x7f489c0776a0 msgr2=0x7f489c101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.888+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 shutdown_connections 2024-03-28T18:46:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:41.888+0000 7f48a2400640 1 -- 172.21.15.119:0/3464054010 wait complete. 2024-03-28T18:46:41.928 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:41.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:46:42.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 -- 172.21.15.119:0/709687180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 msgr2=0x7f910c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 --2- 172.21.15.119:0/709687180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c108b20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f90f80093a0 tx=0x7f90f802ce60 comp rx=0 tx=0).stop 2024-03-28T18:46:42.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 -- 172.21.15.119:0/709687180 shutdown_connections 2024-03-28T18:46:42.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 --2- 172.21.15.119:0/709687180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c109060 0x7f910c114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 --2- 172.21.15.119:0/709687180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c108b20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 --2- 172.21.15.119:0/709687180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c10d0a0 0x7f910c10d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.290+0000 7f9114200640 1 -- 172.21.15.119:0/709687180 >> 172.21.15.119:0/709687180 conn(0x7f910c06c780 msgr2=0x7f910c06cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.292+0000 7f9114200640 1 -- 172.21.15.119:0/709687180 shutdown_connections 2024-03-28T18:46:42.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.293+0000 7f9114200640 1 -- 172.21.15.119:0/709687180 wait complete. 2024-03-28T18:46:42.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.296+0000 7f9114200640 1 Processor -- start 2024-03-28T18:46:42.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.300+0000 7f9114200640 1 -- start start 2024-03-28T18:46:42.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.301+0000 7f9114200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c1a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.301+0000 7f9114200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 0x7f910c1a8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.302+0000 7f9114200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.302+0000 7f9114200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c118010 con 0x7f910c109060 2024-03-28T18:46:42.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.302+0000 7f9114200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c117e90 con 0x7f910c10d0a0 2024-03-28T18:46:42.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9111800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c1a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9112200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9112200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39772/0 (socket says 172.21.15.119:39772) 2024-03-28T18:46:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9112200640 1 -- 172.21.15.119:0/1028033398 learned_addr learned my addr 172.21.15.119:0/1028033398 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c118190 con 0x7f910c1086a0 2024-03-28T18:46:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.303+0000 7f9112200640 1 -- 172.21.15.119:0/1028033398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 msgr2=0x7f910c1a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9110e00640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 0x7f910c1a8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9112200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c1a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9112200640 1 -- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 msgr2=0x7f910c1a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9112200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 0x7f910c1a8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9112200640 1 -- 172.21.15.119:0/1028033398 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c1a2dc0 con 0x7f910c10d0a0 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9111800640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c1a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.304+0000 7f9110e00640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 0x7f910c1a8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:42.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.307+0000 7f9112200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f910400eb10 tx=0x7f910400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.308+0000 7f910a000640 1 -- 172.21.15.119:0/1028033398 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104004b20 con 0x7f910c10d0a0 2024-03-28T18:46:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.308+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c1a30b0 con 0x7f910c10d0a0 2024-03-28T18:46:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.308+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c1af300 con 0x7f910c10d0a0 2024-03-28T18:46:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.308+0000 7f910a000640 1 -- 172.21.15.119:0/1028033398 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9104004cc0 con 0x7f910c10d0a0 2024-03-28T18:46:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.309+0000 7f910a000640 1 -- 172.21.15.119:0/1028033398 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104004b20 con 0x7f910c10d0a0 2024-03-28T18:46:42.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.321+0000 7f9116fd6740 1 Processor -- start 2024-03-28T18:46:42.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.322+0000 7f910a000640 1 -- 172.21.15.119:0/1028033398 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9104010550 con 0x7f910c10d0a0 2024-03-28T18:46:42.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 -- start start 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0d080 0x5619efc15630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc18050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efb702c0 con 0x5619efc0d080 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efb18c90 con 0x5619efc0c6d0 2024-03-28T18:46:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.323+0000 7f9116fd6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efb6c360 con 0x5619efc15c00 2024-03-28T18:46:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.324+0000 7f9110e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.324+0000 7f9110e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39776/0 (socket says 172.21.15.119:39776) 2024-03-28T18:46:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.324+0000 7f9110e00640 1 -- 172.21.15.119:0/283751033 learned_addr learned my addr 172.21.15.119:0/283751033 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.324+0000 7f9110e00640 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc18050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.325+0000 7f9110e00640 1 -- 172.21.15.119:0/283751033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 msgr2=0x5619efc18050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.325+0000 7f9110e00640 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc18050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.325+0000 7f9110e00640 1 -- 172.21.15.119:0/283751033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0d080 msgr2=0x5619efc15630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.325+0000 7f9110e00640 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0d080 0x5619efc15630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.325+0000 7f9110e00640 1 -- 172.21.15.119:0/283751033 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619efb3d080 con 0x5619efc0c6d0 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.326+0000 7f9110e00640 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005c20 tx=0x7f90f8009a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45908e0394762577 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.326+0000 7f90fea00640 1 -- 172.21.15.119:0/283751033 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f803a040 con 0x5619efc0c6d0 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.326+0000 7f90fea00640 1 -- 172.21.15.119:0/283751033 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f802daf0 con 0x5619efc0c6d0 2024-03-28T18:46:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.326+0000 7f90fea00640 1 -- 172.21.15.119:0/283751033 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8004920 con 0x5619efc0c6d0 2024-03-28T18:46:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.327+0000 7f910a000640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 0x7f90f40a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.327+0000 7f9116fd6740 1 -- 172.21.15.119:0/283751033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 msgr2=0x5619efc0cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.327+0000 7f9116fd6740 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005c20 tx=0x7f90f8009a70 comp rx=0 tx=0).stop 2024-03-28T18:46:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.328+0000 7f910a000640 1 -- 172.21.15.119:0/1028033398 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f910400cce0 con 0x7f910c10d0a0 2024-03-28T18:46:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.328+0000 7f9111800640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 0x7f90f40a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 -- 172.21.15.119:0/283751033 shutdown_connections 2024-03-28T18:46:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc18050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0d080 0x5619efc15630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 --2- 172.21.15.119:0/283751033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0c6d0 0x5619efc0cab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 -- 172.21.15.119:0/283751033 >> 172.21.15.119:0/283751033 conn(0x5619efc06da0 msgr2=0x5619efc07180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 -- 172.21.15.119:0/283751033 shutdown_connections 2024-03-28T18:46:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9111800640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 0x7f90f40a5230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5619efc07e10 tx=0x7f9100005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.329+0000 7f9116fd6740 1 -- 172.21.15.119:0/283751033 wait complete. 2024-03-28T18:46:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 -- start start 2024-03-28T18:46:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 Processor -- start 2024-03-28T18:46:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc128a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc15c00 0x5619efc12e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efc18ab0 con 0x5619efc15c00 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efb2c0d0 con 0x5619efc0d080 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.330+0000 7f9116fd6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619efc18c30 con 0x5619efc0c6d0 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9116fd6740 10 client.0 _ll_register_callbacks cb 0x5619efb30e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39250/0 (socket says 172.21.15.119:39250) 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 -- 172.21.15.119:0/222951014 learned_addr learned my addr 172.21.15.119:0/222951014 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9110e00640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9112200640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc15c00 0x5619efc12e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 -- 172.21.15.119:0/222951014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 msgr2=0x5619efc128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 -- 172.21.15.119:0/222951014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc15c00 msgr2=0x5619efc12e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc15c00 0x5619efc12e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.331+0000 7f9111800640 1 -- 172.21.15.119:0/222951014 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619ef8843f0 con 0x5619efc0c6d0 2024-03-28T18:46:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f9111800640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9100010d60 tx=0x7f91000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac25e1b525759b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f90f3400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100023070 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f910001f650 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90f802daf0 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f910002f050 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f9116fd6740 10 client.26155 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.332+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5619ef94e7d0 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.333+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f802daf0 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f910000f040 con 0x5619efc0c6d0 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 10 client.26155 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 10 client.26155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 20 client.26155 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc25a00 0x5619efc27e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 4 client.26155 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619efb7d800 con 0x5619efc25a00 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.334+0000 7f9110e00640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc25a00 0x5619efc27e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.335+0000 7f9110e00640 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc25a00 0x5619efc27e10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef3bfc0359506b0b server_cookie=f51f88ac7e1194ef in_seq=0 out_seq=0 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.335+0000 7f90f3400640 10 client.26155 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.335+0000 7f90f3400640 1 -- 172.21.15.119:0/222951014 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5619efb7d800 con 0x5619efc25a00 2024-03-28T18:46:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.335+0000 7f90f3400640 10 client.26155 handle_command_reply: tid=0 2024-03-28T18:46:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 client.26155 shutdown 2024-03-28T18:46:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 msgr2=0x5619efc15510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9100010d60 tx=0x7f91000046c0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 shutdown_connections 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc25a00 0x5619efc27e10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc15c00 0x5619efc12e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 --2- 172.21.15.119:0/222951014 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc0c6d0 0x5619efc15510 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 >> 172.21.15.119:0/222951014 conn(0x5619efc06da0 msgr2=0x5619efc08af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.336+0000 7f90f3400640 0 client.26155 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.337+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 shutdown_connections 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.337+0000 7f9116fd6740 1 -- 172.21.15.119:0/222951014 wait complete. 2024-03-28T18:46:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.337+0000 7f9116fd6740 20 client.26155 trim_cache size 0 max 16384 2024-03-28T18:46:42.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.350+0000 7f9116fd6740 1 Processor -- start 2024-03-28T18:46:42.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.350+0000 7f9116fd6740 1 -- start start 2024-03-28T18:46:42.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc14fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec00e720 con 0x5619efc0c6d0 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0100a0 con 0x5619efc0d080 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9116fd6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec00ed80 con 0x5619efc15c00 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9110e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9112200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9110e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39792/0 (socket says 172.21.15.119:39792) 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9111800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc14fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9110e00640 1 -- 172.21.15.119:0/2600873586 learned_addr learned my addr 172.21.15.119:0/2600873586 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.351+0000 7f9112200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39258/0 (socket says 172.21.15.119:39258) 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 -- 172.21.15.119:0/2600873586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 msgr2=0x5619efc0fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 -- 172.21.15.119:0/2600873586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 msgr2=0x5619efc14fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc14fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 -- 172.21.15.119:0/2600873586 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619efc0c2b0 con 0x5619efc0d080 2024-03-28T18:46:42.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9112200640 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:42.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9111800640 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc14fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:42.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f9110e00640 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90f802d3a0 tx=0x7f90f802ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55d5c34032b5d0ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f90fe000640 1 -- 172.21.15.119:0/2600873586 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f80053f0 con 0x5619efc0d080 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f90fe000640 1 -- 172.21.15.119:0/2600873586 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f8057120 con 0x5619efc0d080 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.352+0000 7f90fe000640 1 -- 172.21.15.119:0/2600873586 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f80579e0 con 0x5619efc0d080 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 -- 172.21.15.119:0/2600873586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 msgr2=0x5619efc15510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f90f802d3a0 tx=0x7f90f802ce60 comp rx=0 tx=0).stop 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 -- 172.21.15.119:0/2600873586 shutdown_connections 2024-03-28T18:46:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc0fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc15510 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 --2- 172.21.15.119:0/2600873586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc14fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 -- 172.21.15.119:0/2600873586 >> 172.21.15.119:0/2600873586 conn(0x5619efc06da0 msgr2=0x5619efc05eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 -- 172.21.15.119:0/2600873586 shutdown_connections 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.353+0000 7f9116fd6740 1 -- 172.21.15.119:0/2600873586 wait complete. 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 -- start start 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 Processor -- start 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc1db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc1c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec00fa40 con 0x5619efc0c6d0 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec00f3e0 con 0x5619efc0d080 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.354+0000 7f9116fd6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec00c8e0 con 0x5619efc15c00 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9116fd6740 10 client.0 _ll_register_callbacks cb 0x7f9100016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9112200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc1c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9112200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc1c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39266/0 (socket says 172.21.15.119:39266) 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9112200640 1 -- 172.21.15.119:0/916698100 learned_addr learned my addr 172.21.15.119:0/916698100 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9111800640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc1db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 -- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 msgr2=0x5619efc1c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc1c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 -- 172.21.15.119:0/916698100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 msgr2=0x5619efc1db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc1db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.355+0000 7f9110e00640 1 -- 172.21.15.119:0/916698100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619efc1cbb0 con 0x5619efc0d080 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f9110e00640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc1bf30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005ec0 tx=0x7f90f802d4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a18861b57da33f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f90f3400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:42.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8049e60 con 0x5619efc0d080 2024-03-28T18:46:42.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f90f8057780 con 0x5619efc0d080 2024-03-28T18:46:42.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90f8057120 con 0x5619efc0d080 2024-03-28T18:46:42.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90f80572d0 con 0x5619efc0d080 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f9116fd6740 10 client.26165 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.356+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5619efc82270 con 0x5619efc0d080 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f8057120 con 0x5619efc0d080 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f90f8040070 con 0x5619efc0d080 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 10 client.26165 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 10 client.26165 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 20 client.26165 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc824a0 0x5619efc84890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 4 client.26165 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5619efb77b50 con 0x5619efc824a0 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.357+0000 7f9111800640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc824a0 0x5619efc84890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9111800640 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc824a0 0x5619efc84890 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27dff36c204d4d6b server_cookie=e807db7917981c1e in_seq=0 out_seq=0 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f90f3400640 10 client.26165 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f90f3400640 1 -- 172.21.15.119:0/916698100 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5619efb77b50 con 0x5619efc824a0 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f90f3400640 10 client.26165 handle_command_reply: tid=0 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 client.26165 shutdown 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 msgr2=0x5619efc1bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc1bf30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005ec0 tx=0x7f90f802d4e0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 shutdown_connections 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5619efc824a0 0x5619efc84890 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619efc15c00 0x5619efc1c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619efc0d080 0x5619efc1bf30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 --2- 172.21.15.119:0/916698100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619efc0c6d0 0x5619efc1db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 >> 172.21.15.119:0/916698100 conn(0x5619efc06da0 msgr2=0x5619efc080a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f90f3400640 0 client.26165 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 shutdown_connections 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.358+0000 7f9116fd6740 1 -- 172.21.15.119:0/916698100 wait complete. 2024-03-28T18:46:42.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9116fd6740 20 client.26165 trim_cache size 0 max 16384 2024-03-28T18:46:42.362 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:42.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 msgr2=0x7f90f40a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 0x7f90f40a5230 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5619efc07e10 tx=0x7f9100005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:42.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 msgr2=0x7f910c1a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f910400eb10 tx=0x7f910400c2d0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 shutdown_connections 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90f40a2d70 0x7f90f40a5230 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f910c10d0a0 0x7f910c1a2640 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c109060 0x7f910c1a8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.359+0000 7f9114200640 1 --2- 172.21.15.119:0/1028033398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f910c1086a0 0x7f910c1a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.360+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 >> 172.21.15.119:0/1028033398 conn(0x7f910c06c780 msgr2=0x7f910c109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.360+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 shutdown_connections 2024-03-28T18:46:42.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.360+0000 7f9114200640 1 -- 172.21.15.119:0/1028033398 wait complete. 2024-03-28T18:46:42.378 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:46:42.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:46:42.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.688+0000 7f95ac800640 1 -- 172.21.15.119:0/4283010580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4100240 msgr2=0x7f95a410c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.688+0000 7f95ac800640 1 --2- 172.21.15.119:0/4283010580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4100240 0x7f95a410c750 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009ca0 tx=0x7f95a002c100 comp rx=0 tx=0).stop 2024-03-28T18:46:42.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.689+0000 7f95ac800640 1 -- 172.21.15.119:0/4283010580 shutdown_connections 2024-03-28T18:46:42.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.689+0000 7f95ac800640 1 --2- 172.21.15.119:0/4283010580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4100240 0x7f95a410c750 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.689+0000 7f95ac800640 1 --2- 172.21.15.119:0/4283010580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95a40ff880 0x7f95a40ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.689+0000 7f95ac800640 1 --2- 172.21.15.119:0/4283010580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a4105890 0x7f95a4105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.689+0000 7f95ac800640 1 -- 172.21.15.119:0/4283010580 >> 172.21.15.119:0/4283010580 conn(0x7f95a40766f0 msgr2=0x7f95a40fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.690+0000 7f95ac800640 1 -- 172.21.15.119:0/4283010580 shutdown_connections 2024-03-28T18:46:42.692 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:42 smithi196 ceph-mon[36300]: pgmap v82: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 339 MiB/s wr, 96 op/s 2024-03-28T18:46:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.690+0000 7f95ac800640 1 -- 172.21.15.119:0/4283010580 wait complete. 2024-03-28T18:46:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.694+0000 7f95ac800640 1 Processor -- start 2024-03-28T18:46:42.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.695+0000 7f95ac800640 1 -- start start 2024-03-28T18:46:42.697 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:42 smithi131 ceph-mon[36065]: pgmap v82: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 339 MiB/s wr, 96 op/s 2024-03-28T18:46:42.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.698+0000 7f95ac800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 0x7f95a419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95ac800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95a4100240 0x7f95a419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95ac800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95ac800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410f870 con 0x7f95a40ff880 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95ac800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410f6f0 con 0x7f95a4105890 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95ac800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a410f9f0 con 0x7f95a4100240 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95aa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95aa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39820/0 (socket says 172.21.15.119:39820) 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.699+0000 7f95aa800640 1 -- 172.21.15.119:0/2000501932 learned_addr learned my addr 172.21.15.119:0/2000501932 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.700+0000 7f95aa800640 1 -- 172.21.15.119:0/2000501932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95a4100240 msgr2=0x7f95a419cf00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:46:42.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.700+0000 7f95aa800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95a4100240 0x7f95a419cf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.700+0000 7f95aa800640 1 -- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 msgr2=0x7f95a419c9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T18:46:42.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.700+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 0x7f95a419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.701+0000 7f95aa800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 0x7f95a419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.701+0000 7f95aa800640 1 -- 172.21.15.119:0/2000501932 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4196530 con 0x7f95a4105890 2024-03-28T18:46:42.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.702+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 0x7f95a419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:42.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.702+0000 7f95aa800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f95a00348f0 tx=0x7f95a0035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.704+0000 7f959ea00640 1 -- 172.21.15.119:0/2000501932 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0039040 con 0x7f95a4105890 2024-03-28T18:46:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.704+0000 7f959ea00640 1 -- 172.21.15.119:0/2000501932 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95a0044070 con 0x7f95a4105890 2024-03-28T18:46:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.704+0000 7f959ea00640 1 -- 172.21.15.119:0/2000501932 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0004500 con 0x7f95a4105890 2024-03-28T18:46:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.706+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41967c0 con 0x7f95a4105890 2024-03-28T18:46:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.706+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a419a6c0 con 0x7f95a4105890 2024-03-28T18:46:42.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.715+0000 7f959ea00640 1 -- 172.21.15.119:0/2000501932 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f95a00046a0 con 0x7f95a4105890 2024-03-28T18:46:42.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.726+0000 7f959ea00640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 0x7f958c0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.727+0000 7f959ea00640 1 -- 172.21.15.119:0/2000501932 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95a00f0e40 con 0x7f95a4105890 2024-03-28T18:46:42.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.727+0000 7f95af47b740 1 Processor -- start 2024-03-28T18:46:42.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.734+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 0x7f958c0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.736+0000 7f95af47b740 1 -- start start 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed243ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed1a72c0 con 0x5594ed244080 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed14fc90 con 0x5594ed2436d0 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95af47b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed1a3360 con 0x5594ed24cc00 2024-03-28T18:46:42.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.737+0000 7f95a9400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.738+0000 7f95a9400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39998/0 (socket says 172.21.15.119:39998) 2024-03-28T18:46:42.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.738+0000 7f95a9400640 1 -- 172.21.15.119:0/398560026 learned_addr learned my addr 172.21.15.119:0/398560026 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.738+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed243ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.738+0000 7f95aa800640 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.740+0000 7f95a9400640 1 -- 172.21.15.119:0/398560026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 msgr2=0x5594ed24f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.740+0000 7f95a9400640 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.740+0000 7f95a9400640 1 -- 172.21.15.119:0/398560026 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 msgr2=0x5594ed243ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.740+0000 7f95a9400640 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed243ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.740+0000 7f95a9400640 1 -- 172.21.15.119:0/398560026 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594ed174080 con 0x5594ed244080 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.741+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 0x7f958c0a5030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f959400baf0 tx=0x7f9594005d20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.741+0000 7f95a9400640 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f959800a9c0 tx=0x7f959800ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba08a9e635b018f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.743+0000 7f9593000640 1 -- 172.21.15.119:0/398560026 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800cde0 con 0x5594ed244080 2024-03-28T18:46:42.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.744+0000 7f9593000640 1 -- 172.21.15.119:0/398560026 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95980035f0 con 0x5594ed244080 2024-03-28T18:46:42.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.744+0000 7f9593000640 1 -- 172.21.15.119:0/398560026 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800f6b0 con 0x5594ed244080 2024-03-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.745+0000 7f95af47b740 1 -- 172.21.15.119:0/398560026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 msgr2=0x5594ed24c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.745+0000 7f95af47b740 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f959800a9c0 tx=0x7f959800ae80 comp rx=0 tx=0).stop 2024-03-28T18:46:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.749+0000 7f95af47b740 1 -- 172.21.15.119:0/398560026 shutdown_connections 2024-03-28T18:46:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.749+0000 7f95af47b740 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.749+0000 7f95af47b740 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c630 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.749+0000 7f95af47b740 1 --2- 172.21.15.119:0/398560026 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed243ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.749+0000 7f95af47b740 1 -- 172.21.15.119:0/398560026 >> 172.21.15.119:0/398560026 conn(0x5594ed23dda0 msgr2=0x5594ed23e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.751+0000 7f95af47b740 1 -- 172.21.15.119:0/398560026 shutdown_connections 2024-03-28T18:46:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.751+0000 7f95af47b740 1 -- 172.21.15.119:0/398560026 wait complete. 2024-03-28T18:46:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.752+0000 7f95af47b740 1 -- start start 2024-03-28T18:46:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.752+0000 7f95af47b740 1 Processor -- start 2024-03-28T18:46:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed249930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed24fb70 con 0x5594ed244080 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed1630d0 con 0x5594ed2436d0 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594ed24fcf0 con 0x5594ed24cc00 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.753+0000 7f95af47b740 10 client.0 _ll_register_callbacks cb 0x5594ed167e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.754+0000 7f95aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.754+0000 7f95aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39286/0 (socket says 172.21.15.119:39286) 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.754+0000 7f95aa800640 1 -- 172.21.15.119:0/1995888193 learned_addr learned my addr 172.21.15.119:0/1995888193 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.754+0000 7f95a9e00640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f95aa800640 1 -- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 msgr2=0x5594ed24c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f95aa800640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f95aa800640 1 -- 172.21.15.119:0/1995888193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 msgr2=0x5594ed249930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f95aa800640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed249930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f95aa800640 1 -- 172.21.15.119:0/1995888193 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594ecebb3f0 con 0x5594ed24cc00 2024-03-28T18:46:42.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.755+0000 7f9587400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:42.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.756+0000 7f95aa800640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95a00b9f50 tx=0x7f95a00bdc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8400325841fe386 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.756+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00ee960 con 0x5594ed24cc00 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.756+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95a00333a0 con 0x5594ed24cc00 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.756+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95980035f0 con 0x5594ed24cc00 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.757+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95a00ee960 con 0x5594ed24cc00 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.757+0000 7f95af47b740 10 client.26167 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.757+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5594ecf857d0 con 0x5594ed24cc00 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.757+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95980035f0 con 0x5594ed24cc00 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.757+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f95a003c030 con 0x5594ed24cc00 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 10 client.26167 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 20 client.26167 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed25ca90 0x5594ed25ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 4 client.26167 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.758+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594ed1b4800 con 0x5594ed25ca90 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.759+0000 7f95a9e00640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed25ca90 0x5594ed25ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.760+0000 7f95a9e00640 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed25ca90 0x5594ed25ef00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e2695ffb5edb63 server_cookie=64750458eb0e3a5e in_seq=0 out_seq=0 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.761+0000 7f9587400640 10 client.26167 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.761+0000 7f9587400640 1 -- 172.21.15.119:0/1995888193 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5594ed1b4800 con 0x5594ed25ca90 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.761+0000 7f9587400640 10 client.26167 handle_command_reply: tid=0 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.761+0000 7f95af47b740 1 client.26167 shutdown 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 msgr2=0x5594ed24a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95a00b9f50 tx=0x7f95a00bdc80 comp rx=0 tx=0).stop 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 shutdown_connections 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed25ca90 0x5594ed25ef00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed24a070 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed249930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 --2- 172.21.15.119:0/1995888193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f9587400640 0 client.26167 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:42.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 >> 172.21.15.119:0/1995888193 conn(0x5594ed23dda0 msgr2=0x5594ed23faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.762+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 shutdown_connections 2024-03-28T18:46:42.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.763+0000 7f95af47b740 1 -- 172.21.15.119:0/1995888193 wait complete. 2024-03-28T18:46:42.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.763+0000 7f95af47b740 20 client.26167 trim_cache size 0 max 16384 2024-03-28T18:46:42.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.792+0000 7f95af47b740 1 Processor -- start 2024-03-28T18:46:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 -- start start 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed246b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41bc040 con 0x5594ed244080 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41bd9c0 con 0x5594ed2436d0 2024-03-28T18:46:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.794+0000 7f95af47b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41bc6a0 con 0x5594ed24cc00 2024-03-28T18:46:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.796+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.796+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39842/0 (socket says 172.21.15.119:39842) 2024-03-28T18:46:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.796+0000 7f95a9e00640 1 -- 172.21.15.119:0/1820616043 learned_addr learned my addr 172.21.15.119:0/1820616043 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.796+0000 7f95a9400640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.796+0000 7f95aa800640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed246b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95a9400640 1 -- 172.21.15.119:0/1820616043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 msgr2=0x5594ed246b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95a9400640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed246b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95a9400640 1 -- 172.21.15.119:0/1820616043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 msgr2=0x5594ed24c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95a9400640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95a9400640 1 -- 172.21.15.119:0/1820616043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594ed25f580 con 0x5594ed244080 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.798+0000 7f95aa800640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed246b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.799+0000 7f95a9400640 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5594ed244480 tx=0x7f959800a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26c900a211a8bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.800+0000 7f9592600640 1 -- 172.21.15.119:0/1820616043 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800a5c0 con 0x5594ed244080 2024-03-28T18:46:42.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.800+0000 7f9592600640 1 -- 172.21.15.119:0/1820616043 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9598005520 con 0x5594ed244080 2024-03-28T18:46:42.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.800+0000 7f9592600640 1 -- 172.21.15.119:0/1820616043 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800a5c0 con 0x5594ed244080 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 -- 172.21.15.119:0/1820616043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 msgr2=0x5594ed24c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c5a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x5594ed244480 tx=0x7f959800a4d0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 -- 172.21.15.119:0/1820616043 shutdown_connections 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed246b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed244080 0x5594ed24c5a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 --2- 172.21.15.119:0/1820616043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed2436d0 0x5594ed24c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.801+0000 7f95af47b740 1 -- 172.21.15.119:0/1820616043 >> 172.21.15.119:0/1820616043 conn(0x5594ed23dda0 msgr2=0x5594ed23ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.802+0000 7f95af47b740 1 -- 172.21.15.119:0/1820616043 shutdown_connections 2024-03-28T18:46:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.802+0000 7f95af47b740 1 -- 172.21.15.119:0/1820616043 wait complete. 2024-03-28T18:46:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.802+0000 7f95af47b740 1 -- start start 2024-03-28T18:46:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.802+0000 7f95af47b740 1 Processor -- start 2024-03-28T18:46:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 0x5594ed252fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed2b73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41bd360 con 0x5594ed2436d0 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41bcd00 con 0x5594ed244080 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41ba200 con 0x5594ed24cc00 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.803+0000 7f95af47b740 10 client.0 _ll_register_callbacks cb 0x7f95a00b91e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95a9400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 0x5594ed252fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed2b73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95a9400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 0x5594ed252fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39854/0 (socket says 172.21.15.119:39854) 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed2b73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39308/0 (socket says 172.21.15.119:39308) 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95a9400640 1 -- 172.21.15.119:0/2595036951 learned_addr learned my addr 172.21.15.119:0/2595036951 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.804+0000 7f95a9e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40032/0 (socket says 172.21.15.119:40032) 2024-03-28T18:46:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 -- 172.21.15.119:0/2595036951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 msgr2=0x5594ed2b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed2b73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 -- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 msgr2=0x5594ed252fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 0x5594ed252fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 -- 172.21.15.119:0/2595036951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594ed2b7920 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.805+0000 7f95a9e00640 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f959400b860 tx=0x7f9594005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d74bd104082929 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.806+0000 7f9587400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.806+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9594025720 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.806+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9598005520 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.806+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f95940258c0 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.807+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9594004e10 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.807+0000 7f95af47b740 10 client.16002 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.807+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5594ed2b8c50 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.807+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9598005520 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.807+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f959401d410 con 0x5594ed2436d0 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 10 client.16002 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 20 client.16002 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed2b92d0 0x5594ed2bb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 4 client.16002 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5594ed1aeb50 con 0x5594ed2b92d0 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.808+0000 7f95a9400640 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed2b92d0 0x5594ed2bb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.809+0000 7f95a9400640 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed2b92d0 0x5594ed2bb6c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deef7a0b7cb09efd server_cookie=60797254889df521 in_seq=0 out_seq=0 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f9587400640 10 client.16002 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f9587400640 1 -- 172.21.15.119:0/2595036951 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5594ed1aeb50 con 0x5594ed2b92d0 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f9587400640 10 client.16002 handle_command_reply: tid=0 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 client.16002 shutdown 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 msgr2=0x5594ed252a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f959400b860 tx=0x7f9594005f90 comp rx=0 tx=0).stop 2024-03-28T18:46:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 shutdown_connections 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5594ed2b92d0 0x5594ed2bb6c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5594ed24cc00 0x5594ed2b73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5594ed244080 0x5594ed252fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 --2- 172.21.15.119:0/2595036951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594ed2436d0 0x5594ed252a60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.810+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 >> 172.21.15.119:0/2595036951 conn(0x5594ed23dda0 msgr2=0x5594ed2444b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.811+0000 7f9587400640 0 client.16002 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.811+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 shutdown_connections 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.811+0000 7f95af47b740 1 -- 172.21.15.119:0/2595036951 wait complete. 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.811+0000 7f95af47b740 20 client.16002 trim_cache size 0 max 16384 2024-03-28T18:46:42.815 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 msgr2=0x7f958c0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 0x7f958c0a5030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f959400baf0 tx=0x7f9594005d20 comp rx=0 tx=0).stop 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 msgr2=0x7f95a4195c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f95a00348f0 tx=0x7f95a0035d80 comp rx=0 tx=0).stop 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 shutdown_connections 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f958c0a2b70 0x7f958c0a5030 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95a4105890 0x7f95a4195c70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95a4100240 0x7f95a419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 --2- 172.21.15.119:0/2000501932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95a40ff880 0x7f95a419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.812+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 >> 172.21.15.119:0/2000501932 conn(0x7f95a40766f0 msgr2=0x7f95a40fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.813+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 shutdown_connections 2024-03-28T18:46:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:42.813+0000 7f95ac800640 1 -- 172.21.15.119:0/2000501932 wait complete. 2024-03-28T18:46:42.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:42 smithi119 ceph-mon[31813]: pgmap v82: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 339 MiB/s wr, 96 op/s 2024-03-28T18:46:43.835 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:46:43.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:46:43.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:46:44.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 -- 172.21.15.119:0/3572208930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64100880 msgr2=0x7f1a64100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 --2- 172.21.15.119:0/3572208930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64100880 0x7f1a64100d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009900 tx=0x7f1a5002bcd0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 -- 172.21.15.119:0/3572208930 shutdown_connections 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 --2- 172.21.15.119:0/3572208930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64101240 0x7f1a6410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 --2- 172.21.15.119:0/3572208930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64100880 0x7f1a64100d00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 --2- 172.21.15.119:0/3572208930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a64106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 -- 172.21.15.119:0/3572208930 >> 172.21.15.119:0/3572208930 conn(0x7f1a64077580 msgr2=0x7f1a64077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 -- 172.21.15.119:0/3572208930 shutdown_connections 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.085+0000 7f1a6b800640 1 -- 172.21.15.119:0/3572208930 wait complete. 2024-03-28T18:46:44.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 Processor -- start 2024-03-28T18:46:44.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 -- start start 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 0x7f1a6419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a6419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64113cb0 con 0x7f1a64101240 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64113b30 con 0x7f1a64100880 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.087+0000 7f1a6b800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a64113e30 con 0x7f1a64106750 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a69800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a6419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 0x7f1a6419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a69800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a6419ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39328/0 (socket says 172.21.15.119:39328) 2024-03-28T18:46:44.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39880/0 (socket says 172.21.15.119:39880) 2024-03-28T18:46:44.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a69800640 1 -- 172.21.15.119:0/2370118487 learned_addr learned my addr 172.21.15.119:0/2370118487 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 -- 172.21.15.119:0/2370118487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 msgr2=0x7f1a6419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a6419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 -- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 msgr2=0x7f1a6419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 0x7f1a6419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a68e00640 1 -- 172.21.15.119:0/2370118487 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6419f140 con 0x7f1a64100880 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.088+0000 7f1a63e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 0x7f1a6419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a68e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800d720 tx=0x7f1a5800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a61600640 1 -- 172.21.15.119:0/2370118487 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58017070 con 0x7f1a64100880 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a61600640 1 -- 172.21.15.119:0/2370118487 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a5800de30 con 0x7f1a64100880 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6419f430 con 0x7f1a64100880 2024-03-28T18:46:44.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a61600640 1 -- 172.21.15.119:0/2370118487 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a580047a0 con 0x7f1a64100880 2024-03-28T18:46:44.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.089+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a641a6cc0 con 0x7f1a64100880 2024-03-28T18:46:44.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.092+0000 7f1a6e638740 1 Processor -- start 2024-03-28T18:46:44.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.092+0000 7f1a61600640 1 -- 172.21.15.119:0/2370118487 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1a580099d0 con 0x7f1a64100880 2024-03-28T18:46:44.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.092+0000 7f1a6e638740 1 -- start start 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c29040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c2b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662b84eb0 con 0x55a662c209a0 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662b2da60 con 0x55a662c29580 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a6e638740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662b80f50 con 0x55a662c20080 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c2b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c2b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39896/0 (socket says 172.21.15.119:39896) 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39338/0 (socket says 172.21.15.119:39338) 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a63e00640 1 -- 172.21.15.119:0/3028330818 learned_addr learned my addr 172.21.15.119:0/3028330818 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a68e00640 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c29040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 -- 172.21.15.119:0/3028330818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 msgr2=0x55a662c2b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c2b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 -- 172.21.15.119:0/3028330818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 msgr2=0x55a662c29040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c29040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.093+0000 7f1a69800640 1 -- 172.21.15.119:0/3028330818 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6629617e0 con 0x55a662c20080 2024-03-28T18:46:44.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.094+0000 7f1a69800640 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009f50 tx=0x7f1a5c011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c13cae9968e619ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a56000640 1 -- 172.21.15.119:0/3028330818 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c004d20 con 0x55a662c20080 2024-03-28T18:46:44.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a56000640 1 -- 172.21.15.119:0/3028330818 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a5c0129f0 con 0x55a662c20080 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a56000640 1 -- 172.21.15.119:0/3028330818 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c005650 con 0x55a662c20080 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a61600640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 0x7f1a4c0a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a68e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 0x7f1a4c0a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.095+0000 7f1a61600640 1 -- 172.21.15.119:0/2370118487 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1a580ceaf0 con 0x7f1a64100880 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.096+0000 7f1a6e638740 1 -- 172.21.15.119:0/3028330818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 msgr2=0x55a662c20460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.096+0000 7f1a6e638740 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009f50 tx=0x7f1a5c011870 comp rx=0 tx=0).stop 2024-03-28T18:46:44.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.096+0000 7f1a68e00640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 0x7f1a4c0a4e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55a662c1fb50 tx=0x7f1a580bb040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.099+0000 7f1a6e638740 1 -- 172.21.15.119:0/3028330818 shutdown_connections 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.099+0000 7f1a6e638740 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c2b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.099+0000 7f1a6e638740 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c29040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.099+0000 7f1a6e638740 1 --2- 172.21.15.119:0/3028330818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c20460 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.099+0000 7f1a6e638740 1 -- 172.21.15.119:0/3028330818 >> 172.21.15.119:0/3028330818 conn(0x55a662c1a7b0 msgr2=0x55a662c1ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.100+0000 7f1a6e638740 1 -- 172.21.15.119:0/3028330818 shutdown_connections 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.100+0000 7f1a6e638740 1 -- 172.21.15.119:0/3028330818 wait complete. 2024-03-28T18:46:44.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.100+0000 7f1a6e638740 1 -- start start 2024-03-28T18:46:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.101+0000 7f1a6e638740 1 Processor -- start 2024-03-28T18:46:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.101+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c28f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.101+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.101+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 0x55a662c26860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.101+0000 7f1a6e638740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662c2c370 con 0x55a662c29580 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a6e638740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662b40eb0 con 0x55a662c209a0 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a6e638740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a662c2c4f0 con 0x55a662c20080 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a6e638740 10 client.0 _ll_register_callbacks cb 0x55a662b35150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39906/0 (socket says 172.21.15.119:39906) 2024-03-28T18:46:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a63e00640 1 -- 172.21.15.119:0/4159306224 learned_addr learned my addr 172.21.15.119:0/4159306224 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a69800640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c28f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.102+0000 7f1a69800640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 0x55a662c26860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a63e00640 1 -- 172.21.15.119:0/4159306224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 msgr2=0x55a662c28f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c28f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a63e00640 1 -- 172.21.15.119:0/4159306224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 msgr2=0x55a662c26860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 0x55a662c26860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a63e00640 1 -- 172.21.15.119:0/4159306224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6629c1f70 con 0x55a662c209a0 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.103+0000 7f1a69800640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 0x55a662c26860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.104+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a50036040 tx=0x7f1a50004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=140184ca8f66e8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.104+0000 7f1a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.104+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50044070 con 0x55a662c209a0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.104+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1a5002c950 con 0x55a662c209a0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.104+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a5c0129f0 con 0x55a662c209a0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a5002cb50 con 0x55a662c209a0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a6e638740 10 client.26192 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a662859170 con 0x55a662c209a0 2024-03-28T18:46:44.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a5c0129f0 con 0x55a662c209a0 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1a50003950 con 0x55a662c209a0 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a6e638740 10 client.26192 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a6e638740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.105+0000 7f1a6e638740 20 client.26192 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662c44190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a6e638740 4 client.26192 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a662b923f0 con 0x55a662c41da0 2024-03-28T18:46:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a69800640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662c44190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a69800640 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662c44190 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce2e9b3372e79b5e server_cookie=c4dae562eaab1062 in_seq=0 out_seq=0 2024-03-28T18:46:44.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.106+0000 7f1a4aa00640 10 client.26192 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.107+0000 7f1a4aa00640 1 -- 172.21.15.119:0/4159306224 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a662b923f0 con 0x55a662c41da0 2024-03-28T18:46:44.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.107+0000 7f1a4aa00640 10 client.26192 handle_command_reply: tid=0 2024-03-28T18:46:44.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 client.26192 shutdown 2024-03-28T18:46:44.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 msgr2=0x55a662c262c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a50036040 tx=0x7f1a50004270 comp rx=0 tx=0).stop 2024-03-28T18:46:44.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 shutdown_connections 2024-03-28T18:46:44.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662c44190 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c29580 0x55a662c26860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a4aa00640 0 client.26192 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c209a0 0x55a662c262c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4159306224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c28f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 >> 172.21.15.119:0/4159306224 conn(0x55a662c1a7b0 msgr2=0x55a662c1c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.108+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 shutdown_connections 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.109+0000 7f1a6e638740 1 -- 172.21.15.119:0/4159306224 wait complete. 2024-03-28T18:46:44.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.109+0000 7f1a6e638740 20 client.26192 trim_cache size 0 max 16384 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.143+0000 7f1a6e638740 1 Processor -- start 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 -- start start 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c215d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641ab180 con 0x55a662c209a0 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641acb00 con 0x55a662c29580 2024-03-28T18:46:44.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.144+0000 7f1a6e638740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641ab7e0 con 0x55a662c20080 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40074/0 (socket says 172.21.15.119:40074) 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 -- 172.21.15.119:0/4268006271 learned_addr learned my addr 172.21.15.119:0/4268006271 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a69800640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c215d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a69800640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 -- 172.21.15.119:0/4268006271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 msgr2=0x55a662c289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c289f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 -- 172.21.15.119:0/4268006271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 msgr2=0x55a662c215d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c215d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.145+0000 7f1a63e00640 1 -- 172.21.15.119:0/4268006271 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a662c1c190 con 0x55a662c209a0 2024-03-28T18:46:44.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.146+0000 7f1a69800640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c215d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:44.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.146+0000 7f1a63e00640 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1a50005ab0 tx=0x7f1a50001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd8a49a92dbf995 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.146+0000 7f1a55600640 1 -- 172.21.15.119:0/4268006271 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50044070 con 0x55a662c209a0 2024-03-28T18:46:44.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.146+0000 7f1a55600640 1 -- 172.21.15.119:0/4268006271 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a50003ad0 con 0x55a662c209a0 2024-03-28T18:46:44.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.146+0000 7f1a55600640 1 -- 172.21.15.119:0/4268006271 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50005140 con 0x55a662c209a0 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 -- 172.21.15.119:0/4268006271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 msgr2=0x55a662c28f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1a50005ab0 tx=0x7f1a50001e70 comp rx=0 tx=0).stop 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 -- 172.21.15.119:0/4268006271 shutdown_connections 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c215d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c209a0 0x55a662c28f30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 --2- 172.21.15.119:0/4268006271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c20080 0x55a662c289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 -- 172.21.15.119:0/4268006271 >> 172.21.15.119:0/4268006271 conn(0x55a662c1a7b0 msgr2=0x55a662c20ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 -- 172.21.15.119:0/4268006271 shutdown_connections 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.147+0000 7f1a6e638740 1 -- 172.21.15.119:0/4268006271 wait complete. 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.148+0000 7f1a6e638740 1 -- start start 2024-03-28T18:46:44.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.148+0000 7f1a6e638740 1 Processor -- start 2024-03-28T18:46:44.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.149+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.149+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c209a0 0x55a662c37510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a69800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a6e638740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c37a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a6e638740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641ac4a0 con 0x55a662c20080 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a6e638740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641abe40 con 0x55a662c29580 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a6e638740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641a9340 con 0x55a662c209a0 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a6e638740 10 client.0 _ll_register_callbacks cb 0x7f1a50009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a69800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40088/0 (socket says 172.21.15.119:40088) 2024-03-28T18:46:44.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a69800640 1 -- 172.21.15.119:0/563846169 learned_addr learned my addr 172.21.15.119:0/563846169 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.150+0000 7f1a69800640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c37a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 -- 172.21.15.119:0/563846169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c209a0 msgr2=0x55a662c37510 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c209a0 0x55a662c37510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 -- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 msgr2=0x55a662c37a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c37a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 -- 172.21.15.119:0/563846169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a662c26bd0 con 0x55a662c20080 2024-03-28T18:46:44.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a69800640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009f50 tx=0x7f1a5c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30f9dab60356d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c012900 con 0x55a662c20080 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.151+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1a5c005320 con 0x55a662c20080 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.152+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a50003ad0 con 0x55a662c20080 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.152+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a5c005540 con 0x55a662c20080 2024-03-28T18:46:44.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.152+0000 7f1a6e638740 10 client.16020 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.152+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a662c2bce0 con 0x55a662c20080 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.152+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a50003ad0 con 0x55a662c20080 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1a5c035060 con 0x55a662c20080 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 10 client.16020 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 20 client.16020 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662ca0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 4 client.16020 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.153+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a662b8c740 con 0x55a662c41da0 2024-03-28T18:46:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.154+0000 7f1a63e00640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662ca0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.154+0000 7f1a63e00640 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662ca0b50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=882cef9adb9ba962 server_cookie=11692591569ea412 in_seq=0 out_seq=0 2024-03-28T18:46:44.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.154+0000 7f1a4aa00640 10 client.16020 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.156+0000 7f1a4aa00640 1 -- 172.21.15.119:0/563846169 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a662b8c740 con 0x55a662c41da0 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.156+0000 7f1a4aa00640 10 client.16020 handle_command_reply: tid=0 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.156+0000 7f1a6e638740 1 client.16020 shutdown 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.156+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 msgr2=0x55a662c36fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.156+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009f50 tx=0x7f1a5c005bd0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 shutdown_connections 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a662c41da0 0x55a662ca0b50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a662c29580 0x55a662c37a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a662c209a0 0x55a662c37510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 --2- 172.21.15.119:0/563846169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a662c20080 0x55a662c36fd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 >> 172.21.15.119:0/563846169 conn(0x55a662c1a7b0 msgr2=0x55a662c21750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a4aa00640 0 client.16020 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 shutdown_connections 2024-03-28T18:46:44.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.157+0000 7f1a6e638740 1 -- 172.21.15.119:0/563846169 wait complete. 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.158+0000 7f1a6e638740 20 client.16020 trim_cache size 0 max 16384 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "f7943c71-d2cd-40e7-8dff-e8327817daa6", 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:46:44.158 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.158+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 msgr2=0x7f1a4c0a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.158+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 0x7f1a4c0a4e20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55a662c1fb50 tx=0x7f1a580bb040 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.158+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 msgr2=0x7f1a6419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.158+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800d720 tx=0x7f1a5800dbe0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 shutdown_connections 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1a4c0a2960 0x7f1a4c0a4e20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a64106750 0x7f1a6419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a64101240 0x7f1a6419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 --2- 172.21.15.119:0/2370118487 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1a64100880 0x7f1a6419a150 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 >> 172.21.15.119:0/2370118487 conn(0x7f1a64077580 msgr2=0x7f1a640ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 shutdown_connections 2024-03-28T18:46:44.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.159+0000 7f1a6b800640 1 -- 172.21.15.119:0/2370118487 wait complete. 2024-03-28T18:46:44.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:46:44.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:46:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.466+0000 7f3dd0800640 1 -- 172.21.15.119:0/2451958612 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8109850 msgr2=0x7f3dc8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.466+0000 7f3dd0800640 1 --2- 172.21.15.119:0/2451958612 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8109850 0x7f3dc8109c30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009930 tx=0x7f3db802bc40 comp rx=0 tx=0).stop 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.467+0000 7f3dd0800640 1 -- 172.21.15.119:0/2451958612 shutdown_connections 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.467+0000 7f3dd0800640 1 --2- 172.21.15.119:0/2451958612 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8104200 0x7f3dc8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.467+0000 7f3dd0800640 1 --2- 172.21.15.119:0/2451958612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8103840 0x7f3dc8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.467+0000 7f3dd0800640 1 --2- 172.21.15.119:0/2451958612 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8109850 0x7f3dc8109c30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.467+0000 7f3dd0800640 1 -- 172.21.15.119:0/2451958612 >> 172.21.15.119:0/2451958612 conn(0x7f3dc80ff350 msgr2=0x7f3dc8101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.468+0000 7f3dd0800640 1 -- 172.21.15.119:0/2451958612 shutdown_connections 2024-03-28T18:46:44.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.468+0000 7f3dd0800640 1 -- 172.21.15.119:0/2451958612 wait complete. 2024-03-28T18:46:44.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.470+0000 7f3dd0800640 1 Processor -- start 2024-03-28T18:46:44.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.472+0000 7f3dd0800640 1 -- start start 2024-03-28T18:46:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.472+0000 7f3dd0800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.472+0000 7f3dd0800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.473+0000 7f3dd0800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8109850 0x7f3dc810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.473+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.473+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40104/0 (socket says 172.21.15.119:40104) 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.473+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.473+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39372/0 (socket says 172.21.15.119:39372) 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.474+0000 7f3dcd400640 1 -- 172.21.15.119:0/1015960010 learned_addr learned my addr 172.21.15.119:0/1015960010 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.474+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8076da0 con 0x7f3dc8104200 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.474+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8076c20 con 0x7f3dc8109850 2024-03-28T18:46:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.474+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc8076f20 con 0x7f3dc8103840 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.474+0000 7f3dce800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8109850 0x7f3dc810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcde00640 1 -- 172.21.15.119:0/1015960010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8109850 msgr2=0x7f3dc810fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8109850 0x7f3dc810fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcde00640 1 -- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 msgr2=0x7f3dc810f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcde00640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc81b2fc0 con 0x7f3dc8103840 2024-03-28T18:46:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.475+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.476+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3db802c180 tx=0x7f3db802eb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.476+0000 7f3dc2a00640 1 -- 172.21.15.119:0/1015960010 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8041020 con 0x7f3dc8103840 2024-03-28T18:46:44.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.477+0000 7f3dc2a00640 1 -- 172.21.15.119:0/1015960010 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db802c5e0 con 0x7f3dc8103840 2024-03-28T18:46:44.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.477+0000 7f3dc2a00640 1 -- 172.21.15.119:0/1015960010 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8004ed0 con 0x7f3dc8103840 2024-03-28T18:46:44.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.477+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc81b31f0 con 0x7f3dc8103840 2024-03-28T18:46:44.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.477+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc81b3660 con 0x7f3dc8103840 2024-03-28T18:46:44.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.484+0000 7f3dc2a00640 1 -- 172.21.15.119:0/1015960010 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3db8044020 con 0x7f3dc8103840 2024-03-28T18:46:44.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.486+0000 7f3dc2a00640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 0x7f3db00a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.486+0000 7f3dc2a00640 1 -- 172.21.15.119:0/1015960010 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3db80f1530 con 0x7f3dc8103840 2024-03-28T18:46:44.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.488+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 0x7f3db00a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.490+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 0x7f3db00a5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005e50 tx=0x7f3dbc005de0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.502+0000 7f3dd35e8740 1 Processor -- start 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.502+0000 7f3dd35e8740 1 -- start start 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x5583175103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558317474250 con 0x558317519290 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831741ca30 con 0x558317510c00 2024-03-28T18:46:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.503+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583174702f0 con 0x55831750ff20 2024-03-28T18:46:44.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39968/0 (socket says 172.21.15.119:39968) 2024-03-28T18:46:44.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x5583175103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dce800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dce800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40118/0 (socket says 172.21.15.119:40118) 2024-03-28T18:46:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.504+0000 7f3dce800640 1 -- 172.21.15.119:0/854222288 learned_addr learned my addr 172.21.15.119:0/854222288 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.505+0000 7f3dce800640 1 -- 172.21.15.119:0/854222288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 msgr2=0x5583175103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.505+0000 7f3dce800640 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x5583175103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.505+0000 7f3dce800640 1 -- 172.21.15.119:0/854222288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 msgr2=0x55831750f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.505+0000 7f3dce800640 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.505+0000 7f3dce800640 1 -- 172.21.15.119:0/854222288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558317407690 con 0x558317519290 2024-03-28T18:46:44.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.506+0000 7f3dcde00640 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:46:44.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.506+0000 7f3dce800640 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009f60 tx=0x7f3dc4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5407fb5c6028841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.508+0000 7f3db7000640 1 -- 172.21.15.119:0/854222288 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4016070 con 0x558317519290 2024-03-28T18:46:44.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.509+0000 7f3db7000640 1 -- 172.21.15.119:0/854222288 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4012930 con 0x558317519290 2024-03-28T18:46:44.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.509+0000 7f3db7000640 1 -- 172.21.15.119:0/854222288 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4011aa0 con 0x558317519290 2024-03-28T18:46:44.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 -- 172.21.15.119:0/854222288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 msgr2=0x55831751b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009f60 tx=0x7f3dc4009f90 comp rx=0 tx=0).stop 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 -- 172.21.15.119:0/854222288 shutdown_connections 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317519290 0x55831751b680 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x5583175103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/854222288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317510c00 0x55831750f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.512+0000 7f3dd35e8740 1 -- 172.21.15.119:0/854222288 >> 172.21.15.119:0/854222288 conn(0x5583174d7200 msgr2=0x55831750a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.513+0000 7f3dd35e8740 1 -- 172.21.15.119:0/854222288 shutdown_connections 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.513+0000 7f3dd35e8740 1 -- 172.21.15.119:0/854222288 wait complete. 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.514+0000 7f3dd35e8740 1 -- start start 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.514+0000 7f3dd35e8740 1 Processor -- start 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 0x558317518820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317516160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831752d110 con 0x55831750ff20 2024-03-28T18:46:44.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831742fe80 con 0x558317519290 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831752d290 con 0x558317510c00 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.515+0000 7f3dd35e8740 10 client.0 _ll_register_callbacks cb 0x558317442130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.516+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.516+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39396/0 (socket says 172.21.15.119:39396) 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.516+0000 7f3dcd400640 1 -- 172.21.15.119:0/1228240292 learned_addr learned my addr 172.21.15.119:0/1228240292 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.516+0000 7f3dce800640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317516160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.517+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 0x558317518820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.517+0000 7f3dcd400640 1 -- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 msgr2=0x558317516160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.517+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317516160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.518+0000 7f3dcd400640 1 -- 172.21.15.119:0/1228240292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 msgr2=0x558317518820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.518+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 0x558317518820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.518+0000 7f3dcd400640 1 -- 172.21.15.119:0/1228240292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583172527d0 con 0x558317510c00 2024-03-28T18:46:44.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.518+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 0x558317518820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:46:44.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.519+0000 7f3dab400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:46:44.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.519+0000 7f3dcd400640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0058c0 tx=0x7f3dbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50126e2013af8dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.520+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc01de10 con 0x558317510c00 2024-03-28T18:46:44.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.520+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3dbc005020 con 0x558317510c00 2024-03-28T18:46:44.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.520+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3dc4012930 con 0x558317510c00 2024-03-28T18:46:44.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.521+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dbc005260 con 0x558317510c00 2024-03-28T18:46:44.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.521+0000 7f3dd35e8740 10 client.26197 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:44.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.521+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5583171883f0 con 0x558317510c00 2024-03-28T18:46:44.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4012930 con 0x558317510c00 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3dbc030630 con 0x558317510c00 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 10 client.26197 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 10 client.26197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 20 client.26197 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55831753a170 0x55831753c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 4 client.26197 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:46:44.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.523+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558317481790 con 0x55831753a170 2024-03-28T18:46:44.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.524+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55831753a170 0x55831753c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.524+0000 7f3dab400640 10 client.26197 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.524+0000 7f3dcde00640 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55831753a170 0x55831753c5a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe13b40b07083c3 server_cookie=fe4c8c5aadf6b8b in_seq=0 out_seq=0 2024-03-28T18:46:44.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.525+0000 7f3dab400640 1 -- 172.21.15.119:0/1228240292 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558317481790 con 0x55831753a170 2024-03-28T18:46:44.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.525+0000 7f3dab400640 10 client.26197 handle_command_reply: tid=0 2024-03-28T18:46:44.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.525+0000 7f3dd35e8740 1 client.26197 shutdown 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 msgr2=0x558317515c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0058c0 tx=0x7f3dbc0046c0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 shutdown_connections 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55831753a170 0x55831753c5a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317516160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317510c00 0x558317515c20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/1228240292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55831750ff20 0x558317518820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 >> 172.21.15.119:0/1228240292 conn(0x5583174d7200 msgr2=0x55831750c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dab400640 0 client.26197 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 shutdown_connections 2024-03-28T18:46:44.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 1 -- 172.21.15.119:0/1228240292 wait complete. 2024-03-28T18:46:44.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.526+0000 7f3dd35e8740 20 client.26197 trim_cache size 0 max 16384 2024-03-28T18:46:44.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.557+0000 7f3dd35e8740 1 Processor -- start 2024-03-28T18:46:44.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.557+0000 7f3dd35e8740 1 -- start start 2024-03-28T18:46:44.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55831750ff20 0x5583175151a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x55831751a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00e740 con 0x558317510c00 2024-03-28T18:46:44.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac0100c0 con 0x55831750ff20 2024-03-28T18:46:44.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.558+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00eda0 con 0x558317519290 2024-03-28T18:46:44.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.559+0000 7f3dce800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.559+0000 7f3dce800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:39410/0 (socket says 172.21.15.119:39410) 2024-03-28T18:46:44.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.559+0000 7f3dce800640 1 -- 172.21.15.119:0/125350871 learned_addr learned my addr 172.21.15.119:0/125350871 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dcde00640 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55831750ff20 0x5583175151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dcd400640 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x55831751a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dce800640 1 -- 172.21.15.119:0/125350871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55831750ff20 msgr2=0x5583175151a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dce800640 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55831750ff20 0x5583175151a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dce800640 1 -- 172.21.15.119:0/125350871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 msgr2=0x55831751a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.560+0000 7f3dce800640 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x55831751a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.561+0000 7f3dce800640 1 -- 172.21.15.119:0/125350871 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583171480f0 con 0x558317519290 2024-03-28T18:46:44.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.562+0000 7f3dce800640 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009f60 tx=0x7f3dc4009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c390a5d58caf894e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.563+0000 7f3db6600640 1 -- 172.21.15.119:0/125350871 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4016040 con 0x558317519290 2024-03-28T18:46:44.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.564+0000 7f3db6600640 1 -- 172.21.15.119:0/125350871 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4009b70 con 0x558317519290 2024-03-28T18:46:44.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.564+0000 7f3db6600640 1 -- 172.21.15.119:0/125350871 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4005350 con 0x558317519290 2024-03-28T18:46:44.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.565+0000 7f3dd35e8740 1 -- 172.21.15.119:0/125350871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 msgr2=0x55831751abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.565+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009f60 tx=0x7f3dc4009a00 comp rx=0 tx=0).stop 2024-03-28T18:46:44.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.565+0000 7f3dd35e8740 1 -- 172.21.15.119:0/125350871 shutdown_connections 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.565+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558317519290 0x55831751abc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.566+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x55831751a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.566+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/125350871 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55831750ff20 0x5583175151a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.566+0000 7f3dd35e8740 1 -- 172.21.15.119:0/125350871 >> 172.21.15.119:0/125350871 conn(0x5583174d7200 msgr2=0x55831750a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.566+0000 7f3dd35e8740 1 -- 172.21.15.119:0/125350871 shutdown_connections 2024-03-28T18:46:44.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.566+0000 7f3dd35e8740 1 -- 172.21.15.119:0/125350871 wait complete. 2024-03-28T18:46:44.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.567+0000 7f3dd35e8740 1 -- start start 2024-03-28T18:46:44.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.568+0000 7f3dd35e8740 1 Processor -- start 2024-03-28T18:46:44.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:44 smithi119 ceph-mon[31813]: pgmap v83: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:44.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:44 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:46:44.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:44 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:44.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:44 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:46:44.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:44 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:44.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.571+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x558317512910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x558317512e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dd35e8740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317594a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00fa60 con 0x558317510c00 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00f400 con 0x558317519290 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dd35e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00c900 con 0x55831750ff20 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dcd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x558317512e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dcde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x558317512910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.572+0000 7f3dce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317594a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.573+0000 7f3dce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317594a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:39996/0 (socket says 172.21.15.119:39996) 2024-03-28T18:46:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.573+0000 7f3dce800640 1 -- 172.21.15.119:0/3250812917 learned_addr learned my addr 172.21.15.119:0/3250812917 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:46:44.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.573+0000 7f3dd35e8740 10 client.0 _ll_register_callbacks cb 0x7f3dbc010780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:46:44.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 -- 172.21.15.119:0/3250812917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 msgr2=0x558317512910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x558317512910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 -- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 msgr2=0x558317594a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:46:44.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317594a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 -- 172.21.15.119:0/3250812917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558317594fb0 con 0x558317510c00 2024-03-28T18:46:44.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dcd400640 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x558317512e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc030a20 tx=0x7f3dbc01d8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f6c4f88c206f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:46:44.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.574+0000 7f3dab400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:46:44.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.575+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc041060 con 0x558317510c00 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.575+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3dc4009b70 con 0x558317510c00 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.575+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3dbc01dc10 con 0x558317510c00 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.576+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dbc036040 con 0x558317510c00 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.576+0000 7f3dd35e8740 10 client.16032 fetch_fsmap learned FSMap version 28 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.576+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558317596160 con 0x558317510c00 2024-03-28T18:46:44.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.576+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3dc4009b70 con 0x558317510c00 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3dbc004a00 con 0x558317510c00 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 10 client.16032 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 20 client.16032 populate_metadata read hostname 'smithi119' 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558317596780 0x558317598ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 4 client.16032 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.577+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55831747bae0 con 0x558317596780 2024-03-28T18:46:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.578+0000 7f3dcde00640 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558317596780 0x558317598ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:46:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.578+0000 7f3dcde00640 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558317596780 0x558317598ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d21e526440f502a server_cookie=15f9cf7d2f6491a in_seq=0 out_seq=0 2024-03-28T18:46:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.578+0000 7f3dab400640 10 client.16032 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dab400640 1 -- 172.21.15.119:0/3250812917 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55831747bae0 con 0x558317596780 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dab400640 10 client.16032 handle_command_reply: tid=0 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 client.16032 shutdown 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 msgr2=0x558317512e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x558317512e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc030a20 tx=0x7f3dbc01d8d0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 shutdown_connections 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558317596780 0x558317598ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558317519290 0x558317594a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558317510c00 0x558317512e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 --2- 172.21.15.119:0/3250812917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55831750ff20 0x558317512910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 >> 172.21.15.119:0/3250812917 conn(0x5583174d7200 msgr2=0x558317514470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.579+0000 7f3dab400640 0 client.16032 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:46:44.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.580+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 shutdown_connections 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.580+0000 7f3dd35e8740 1 -- 172.21.15.119:0/3250812917 wait complete. 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.580+0000 7f3dd35e8740 20 client.16032 trim_cache size 0 max 16384 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "f7943c71-d2cd-40e7-8dff-e8327817daa6": { 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "f7943c71-d2cd-40e7-8dff-e8327817daa6", 2024-03-28T18:46:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:46:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:46:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:46:44.582 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:46:44.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.584+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 msgr2=0x7f3db00a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.584+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 0x7f3db00a5000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005e50 tx=0x7f3dbc005de0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.584+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 msgr2=0x7f3dc810f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:46:44.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.584+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3db802c180 tx=0x7f3db802eb50 comp rx=0 tx=0).stop 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 shutdown_connections 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3db00a2b40 0x7f3db00a5000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3dc8109850 0x7f3dc810fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc8104200 0x7f3dc810f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 --2- 172.21.15.119:0/1015960010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dc8103840 0x7f3dc810f090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:46:44.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 >> 172.21.15.119:0/1015960010 conn(0x7f3dc80ff350 msgr2=0x7f3dc8101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:46:44.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 shutdown_connections 2024-03-28T18:46:44.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:46:44.585+0000 7f3dd0800640 1 -- 172.21.15.119:0/1015960010 wait complete. 2024-03-28T18:46:44.612 INFO:tasks.cephfs.filesystem:scrub status for tag:f7943c71-d2cd-40e7-8dff-e8327817daa6 - {'path': '/', 'tag': 'f7943c71-d2cd-40e7-8dff-e8327817daa6', 'options': 'recursive,force'} 2024-03-28T18:46:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:44 smithi196 ceph-mon[36300]: pgmap v83: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:44.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:44 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:46:44.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:44 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:44.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:44 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:46:44.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:44 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:44 smithi131 ceph-mon[36065]: pgmap v83: 97 pgs: 97 active+clean; 6.4 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 304 MiB/s wr, 87 op/s 2024-03-28T18:46:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:44 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:46:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:44 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:44 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:46:44.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:44 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:46:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:46 smithi196 ceph-mon[36300]: pgmap v84: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 479 MiB/s wr, 135 op/s 2024-03-28T18:46:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:46 smithi131 ceph-mon[36065]: pgmap v84: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 479 MiB/s wr, 135 op/s 2024-03-28T18:46:46.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:46 smithi119 ceph-mon[31813]: pgmap v84: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 479 MiB/s wr, 135 op/s 2024-03-28T18:46:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:48 smithi196 ceph-mon[36300]: pgmap v85: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 328 MiB/s wr, 92 op/s 2024-03-28T18:46:48.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:48 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:46:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:48 smithi131 ceph-mon[36065]: pgmap v85: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 328 MiB/s wr, 92 op/s 2024-03-28T18:46:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:48 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:46:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:48 smithi119 ceph-mon[31813]: pgmap v85: 97 pgs: 97 active+clean; 8.5 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 328 MiB/s wr, 92 op/s 2024-03-28T18:46:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:48 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:46:49.973 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 initial writers = 357709.66 kB/sec 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 initial writers = 357701.33 kB/sec 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 357709.66 kB/sec 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 357709.66 kB/sec 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 357709.66 kB/sec 2024-03-28T18:46:49.974 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 10485760.00 kB 2024-03-28T18:46:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:50 smithi131 ceph-mon[36065]: pgmap v86: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 509 MiB/s wr, 142 op/s 2024-03-28T18:46:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:50 smithi119 ceph-mon[31813]: pgmap v86: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 509 MiB/s wr, 142 op/s 2024-03-28T18:46:50.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:50 smithi196 ceph-mon[36300]: pgmap v86: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 509 MiB/s wr, 142 op/s 2024-03-28T18:46:52.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:52 smithi131 ceph-mon[36065]: pgmap v87: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 374 MiB/s wr, 102 op/s 2024-03-28T18:46:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:52 smithi119 ceph-mon[31813]: pgmap v87: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 374 MiB/s wr, 102 op/s 2024-03-28T18:46:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:52 smithi196 ceph-mon[36300]: pgmap v87: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 374 MiB/s wr, 102 op/s 2024-03-28T18:46:54.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:54 smithi119 ceph-mon[31813]: pgmap v88: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 357 MiB/s wr, 98 op/s 2024-03-28T18:46:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:54 smithi196 ceph-mon[36300]: pgmap v88: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 357 MiB/s wr, 98 op/s 2024-03-28T18:46:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:54 smithi131 ceph-mon[36065]: pgmap v88: 97 pgs: 97 active+clean; 11 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 357 MiB/s wr, 98 op/s 2024-03-28T18:46:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:56 smithi119 ceph-mon[31813]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 403 MiB/s wr, 114 op/s 2024-03-28T18:46:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:56 smithi196 ceph-mon[36300]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 403 MiB/s wr, 114 op/s 2024-03-28T18:46:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:56 smithi131 ceph-mon[36065]: pgmap v89: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 403 MiB/s wr, 114 op/s 2024-03-28T18:46:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:46:58 smithi119 ceph-mon[31813]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 228 MiB/s wr, 65 op/s 2024-03-28T18:46:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:46:58 smithi196 ceph-mon[36300]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 228 MiB/s wr, 65 op/s 2024-03-28T18:46:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:46:58 smithi131 ceph-mon[36065]: pgmap v90: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 228 MiB/s wr, 65 op/s 2024-03-28T18:47:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:00 smithi119 ceph-mon[31813]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 390 MiB/s wr, 111 op/s 2024-03-28T18:47:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:00 smithi196 ceph-mon[36300]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 390 MiB/s wr, 111 op/s 2024-03-28T18:47:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:00 smithi131 ceph-mon[36065]: pgmap v91: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 390 MiB/s wr, 111 op/s 2024-03-28T18:47:02.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:02 smithi119 ceph-mon[31813]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:02 smithi196 ceph-mon[36300]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:02 smithi131 ceph-mon[36065]: pgmap v92: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:03.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:03 smithi119 ceph-mon[31813]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:03.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:03 smithi196 ceph-mon[36300]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:03.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:03 smithi131 ceph-mon[36065]: pgmap v93: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 209 MiB/s wr, 61 op/s 2024-03-28T18:47:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:06 smithi196 ceph-mon[36300]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 375 MiB/s wr, 123 op/s 2024-03-28T18:47:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:06 smithi131 ceph-mon[36065]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 375 MiB/s wr, 123 op/s 2024-03-28T18:47:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:06 smithi119 ceph-mon[31813]: pgmap v94: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 375 MiB/s wr, 123 op/s 2024-03-28T18:47:08.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:08 smithi119 ceph-mon[31813]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 329 MiB/s wr, 108 op/s 2024-03-28T18:47:08.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:08 smithi196 ceph-mon[36300]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 329 MiB/s wr, 108 op/s 2024-03-28T18:47:08.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:08 smithi131 ceph-mon[36065]: pgmap v95: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 329 MiB/s wr, 108 op/s 2024-03-28T18:47:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:10 smithi196 ceph-mon[36300]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 508 MiB/s wr, 158 op/s 2024-03-28T18:47:10.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:10 smithi131 ceph-mon[36065]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 508 MiB/s wr, 158 op/s 2024-03-28T18:47:10.826 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:10 smithi119 ceph-mon[31813]: pgmap v96: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 508 MiB/s wr, 158 op/s 2024-03-28T18:47:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:12 smithi196 ceph-mon[36300]: pgmap v97: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:12 smithi131 ceph-mon[36065]: pgmap v97: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:12.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:12 smithi119 ceph-mon[31813]: pgmap v97: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:14.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:47:14.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:47:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:14 smithi196 ceph-mon[36300]: pgmap v98: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:14 smithi131 ceph-mon[36065]: pgmap v98: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:14.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:14 smithi119 ceph-mon[31813]: pgmap v98: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 346 MiB/s wr, 112 op/s 2024-03-28T18:47:14.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 -- 172.21.15.119:0/624115769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c1040e0 msgr2=0x7f5f6c1105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 --2- 172.21.15.119:0/624115769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c1105f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009cd0 tx=0x7f5f6402c120 comp rx=0 tx=0).stop 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 -- 172.21.15.119:0/624115769 shutdown_connections 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 --2- 172.21.15.119:0/624115769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c1105f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 --2- 172.21.15.119:0/624115769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 --2- 172.21.15.119:0/624115769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c109730 0x7f5f6c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 -- 172.21.15.119:0/624115769 >> 172.21.15.119:0/624115769 conn(0x7f5f6c0ff1b0 msgr2=0x7f5f6c1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:14.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 -- 172.21.15.119:0/624115769 shutdown_connections 2024-03-28T18:47:14.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.966+0000 7f5f6b200640 1 -- 172.21.15.119:0/624115769 wait complete. 2024-03-28T18:47:14.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.968+0000 7f5f6b200640 1 Processor -- start 2024-03-28T18:47:14.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.969+0000 7f5f6b200640 1 -- start start 2024-03-28T18:47:14.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.970+0000 7f5f6b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.970+0000 7f5f6b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.970+0000 7f5f6b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 0x7f5f6c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.971+0000 7f5f6a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 0x7f5f6c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.971+0000 7f5f6a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 0x7f5f6c06c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54998/0 (socket says 172.21.15.119:54998) 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.971+0000 7f5f63e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.971+0000 7f5f6a800640 1 -- 172.21.15.119:0/2600678488 learned_addr learned my addr 172.21.15.119:0/2600678488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.971+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113c00 con 0x7f5f6c109730 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.972+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113a80 con 0x7f5f6c103720 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.972+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c113d80 con 0x7f5f6c1040e0 2024-03-28T18:47:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.972+0000 7f5f69e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.973+0000 7f5f63e00640 1 -- 172.21.15.119:0/2600678488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 msgr2=0x7f5f6c072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.973+0000 7f5f63e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.973+0000 7f5f63e00640 1 -- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 msgr2=0x7f5f6c06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.973+0000 7f5f63e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 0x7f5f6c06c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.973+0000 7f5f63e00640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c06ca20 con 0x7f5f6c1040e0 2024-03-28T18:47:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.974+0000 7f5f69e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:14.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.975+0000 7f5f63e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c06bf50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009fd0 tx=0x7f5f5c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:14.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.976+0000 7f5f62a00640 1 -- 172.21.15.119:0/2600678488 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c015020 con 0x7f5f6c1040e0 2024-03-28T18:47:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.976+0000 7f5f62a00640 1 -- 172.21.15.119:0/2600678488 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f5c011d10 con 0x7f5f6c1040e0 2024-03-28T18:47:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.976+0000 7f5f62a00640 1 -- 172.21.15.119:0/2600678488 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c00f2e0 con 0x7f5f6c1040e0 2024-03-28T18:47:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.976+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c06cd10 con 0x7f5f6c1040e0 2024-03-28T18:47:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.976+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c070c70 con 0x7f5f6c1040e0 2024-03-28T18:47:14.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.986+0000 7f5f73c80740 1 Processor -- start 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.989+0000 7f5f73c80740 1 -- start start 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.989+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.989+0000 7f5f62a00640 1 -- 172.21.15.119:0/2600678488 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5f5c018020 con 0x7f5f6c1040e0 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.992+0000 7f5f62a00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 0x7f5f500a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.992+0000 7f5f62a00640 1 -- 172.21.15.119:0/2600678488 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5f5c0ce340 con 0x7f5f6c1040e0 2024-03-28T18:47:14.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec987a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f6a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f6a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54110/0 (socket says 172.21.15.119:54110) 2024-03-28T18:47:14.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9885850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f73c80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec97de250 con 0x55cec987a0f0 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f73c80740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec9786a30 con 0x55cec987add0 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f73c80740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec97da2f0 con 0x55cec9883460 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.994+0000 7f5f6a800640 1 -- 172.21.15.119:0/4231775212 learned_addr learned my addr 172.21.15.119:0/4231775212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9885850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 -- 172.21.15.119:0/4231775212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 msgr2=0x55cec9885850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9885850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 -- 172.21.15.119:0/4231775212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 msgr2=0x55cec987a570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec987a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.995+0000 7f5f6a800640 1 -- 172.21.15.119:0/4231775212 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cec9771690 con 0x55cec987add0 2024-03-28T18:47:14.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.996+0000 7f5f6a800640 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5f64005c40 tx=0x7f5f64009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6ca107edcdcdc76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:14.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.997+0000 7f5f69e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 0x7f5f500a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:14.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.998+0000 7f5f5b400640 1 -- 172.21.15.119:0/4231775212 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64059020 con 0x55cec987add0 2024-03-28T18:47:14.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.999+0000 7f5f5b400640 1 -- 172.21.15.119:0/4231775212 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f64032dd0 con 0x55cec987add0 2024-03-28T18:47:14.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:14.999+0000 7f5f5b400640 1 -- 172.21.15.119:0/4231775212 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64059020 con 0x55cec987add0 2024-03-28T18:47:15.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.001+0000 7f5f73c80740 1 -- 172.21.15.119:0/4231775212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 msgr2=0x55cec9879b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.001+0000 7f5f73c80740 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5f64005c40 tx=0x7f5f64009900 comp rx=0 tx=0).stop 2024-03-28T18:47:15.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.004+0000 7f5f73c80740 1 -- 172.21.15.119:0/4231775212 shutdown_connections 2024-03-28T18:47:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.005+0000 7f5f69e00640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 0x7f5f500a5000 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009920 tx=0x7f5f54008040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.005+0000 7f5f73c80740 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9885850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.005+0000 7f5f73c80740 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec987a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.005+0000 7f5f73c80740 1 --2- 172.21.15.119:0/4231775212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9879b20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.005+0000 7f5f73c80740 1 -- 172.21.15.119:0/4231775212 >> 172.21.15.119:0/4231775212 conn(0x55cec9841200 msgr2=0x55cec9874b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.008+0000 7f5f73c80740 1 -- 172.21.15.119:0/4231775212 shutdown_connections 2024-03-28T18:47:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.008+0000 7f5f73c80740 1 -- 172.21.15.119:0/4231775212 wait complete. 2024-03-28T18:47:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.009+0000 7f5f73c80740 1 -- start start 2024-03-28T18:47:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.009+0000 7f5f73c80740 1 Processor -- start 2024-03-28T18:47:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.010+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec9882c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9880510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54112/0 (socket says 172.21.15.119:54112) 2024-03-28T18:47:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f63e00640 1 -- 172.21.15.119:0/496021030 learned_addr learned my addr 172.21.15.119:0/496021030 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec9886270 con 0x55cec987a0f0 2024-03-28T18:47:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f6a800640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9880510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec9799e80 con 0x55cec987add0 2024-03-28T18:47:15.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.011+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cec98863f0 con 0x55cec9883460 2024-03-28T18:47:15.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f73c80740 10 client.0 _ll_register_callbacks cb 0x55cec97ac130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:15.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f6a800640 1 -- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 msgr2=0x55cec987ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f6a800640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f6a800640 1 -- 172.21.15.119:0/496021030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 msgr2=0x55cec9882c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f6a800640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec9882c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.012+0000 7f5f6a800640 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cec95bc7d0 con 0x55cec9883460 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.014+0000 7f5f6a800640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9880510 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5f640098d0 tx=0x7f5f64009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9088ba93bf020a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.014+0000 7f5f4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.015+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64059030 con 0x55cec9883460 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.015+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5f64004600 con 0x55cec9883460 2024-03-28T18:47:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.016+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f64032dd0 con 0x55cec9883460 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.016+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f64039040 con 0x55cec9883460 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.016+0000 7f5f73c80740 10 client.26212 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.016+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cec94f23f0 con 0x55cec9883460 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.016+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f64004800 con 0x55cec9883460 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.017+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5f64040050 con 0x55cec9883460 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 10 client.26212 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 20 client.26212 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98932d0 0x55cec98956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 4 client.26212 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f64032dd0 con 0x55cec98932d0 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.018+0000 7f5f69e00640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98932d0 0x55cec98956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.019+0000 7f5f69e00640 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98932d0 0x55cec98956c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71ad5f1093ee343 server_cookie=ddd45c784ab7d85d in_seq=0 out_seq=0 2024-03-28T18:47:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.019+0000 7f5f4fe00640 10 client.26212 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:15.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.020+0000 7f5f4fe00640 1 -- 172.21.15.119:0/496021030 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f5f64032dd0 con 0x55cec98932d0 2024-03-28T18:47:15.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.020+0000 7f5f4fe00640 10 client.26212 handle_command_reply: tid=0 2024-03-28T18:47:15.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.021+0000 7f5f73c80740 1 client.26212 shutdown 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 msgr2=0x55cec9880510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9880510 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5f640098d0 tx=0x7f5f64009360 comp rx=0 tx=0).stop 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 shutdown_connections 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98932d0 0x55cec98956c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9880510 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 --2- 172.21.15.119:0/496021030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec9882c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 >> 172.21.15.119:0/496021030 conn(0x55cec9841200 msgr2=0x55cec9876890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.023+0000 7f5f4fe00640 0 client.26212 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:15.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.033+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 shutdown_connections 2024-03-28T18:47:15.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.034+0000 7f5f73c80740 1 -- 172.21.15.119:0/496021030 wait complete. 2024-03-28T18:47:15.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.034+0000 7f5f73c80740 20 client.26212 trim_cache size 0 max 16384 2024-03-28T18:47:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.081+0000 7f5f73c80740 1 Processor -- start 2024-03-28T18:47:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.082+0000 7f5f73c80740 1 -- start start 2024-03-28T18:47:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.082+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.082+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9882700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9882c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f73c80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1b38c0 con 0x55cec987a0f0 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f73c80740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1b5240 con 0x55cec987add0 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f73c80740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1b3f20 con 0x55cec9883460 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55018/0 (socket says 172.21.15.119:55018) 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 -- 172.21.15.119:0/3457851825 learned_addr learned my addr 172.21.15.119:0/3457851825 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f6a800640 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9882c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f63e00640 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9882700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 -- 172.21.15.119:0/3457851825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 msgr2=0x55cec9882c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9882c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 -- 172.21.15.119:0/3457851825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 msgr2=0x55cec9882700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9882700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.083+0000 7f5f69e00640 1 -- 172.21.15.119:0/3457851825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cec94b20f0 con 0x55cec987a0f0 2024-03-28T18:47:15.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.084+0000 7f5f69e00640 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55cec9886390 tx=0x7f5f54011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7e9b0fbbf57443 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.085+0000 7f5f5aa00640 1 -- 172.21.15.119:0/3457851825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400f040 con 0x55cec987a0f0 2024-03-28T18:47:15.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.085+0000 7f5f5aa00640 1 -- 172.21.15.119:0/3457851825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f5402a070 con 0x55cec987a0f0 2024-03-28T18:47:15.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.085+0000 7f5f5aa00640 1 -- 172.21.15.119:0/3457851825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54015bf0 con 0x55cec987a0f0 2024-03-28T18:47:15.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 -- 172.21.15.119:0/3457851825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 msgr2=0x55cec988edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55cec9886390 tx=0x7f5f54011000 comp rx=0 tx=0).stop 2024-03-28T18:47:15.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 -- 172.21.15.119:0/3457851825 shutdown_connections 2024-03-28T18:47:15.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec9883460 0x55cec9882c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec9882700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 --2- 172.21.15.119:0/3457851825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec987a0f0 0x55cec988edc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 -- 172.21.15.119:0/3457851825 >> 172.21.15.119:0/3457851825 conn(0x55cec9841200 msgr2=0x55cec988e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 -- 172.21.15.119:0/3457851825 shutdown_connections 2024-03-28T18:47:15.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.086+0000 7f5f73c80740 1 -- 172.21.15.119:0/3457851825 wait complete. 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.087+0000 7f5f73c80740 1 -- start start 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.087+0000 7f5f73c80740 1 Processor -- start 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 0x55cec987c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec9883460 0x55cec987d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1b4be0 con 0x55cec9883460 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c1b4580 con 0x55cec987add0 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c079d40 con 0x55cec987a0f0 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f73c80740 10 client.0 _ll_register_callbacks cb 0x7f5f640326d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f69e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 0x55cec987c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54128/0 (socket says 172.21.15.119:54128) 2024-03-28T18:47:15.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f63e00640 1 -- 172.21.15.119:0/139570728 learned_addr learned my addr 172.21.15.119:0/139570728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f6a800640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec9883460 0x55cec987d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f69e00640 1 -- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 msgr2=0x55cec987ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f69e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f69e00640 1 -- 172.21.15.119:0/139570728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec9883460 msgr2=0x55cec987d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.088+0000 7f5f69e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec9883460 0x55cec987d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f69e00640 1 -- 172.21.15.119:0/139570728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cec98edac0 con 0x55cec987a0f0 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f63e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f69e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 0x55cec987c920 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c00 tx=0x7f5f54009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9867d67d30ff817 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54032070 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f5402a070 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.089+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5f54015640 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.090+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f540242e0 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.090+0000 7f5f73c80740 10 client.26218 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.090+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cec98eed70 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.090+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f5402a070 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5f54012030 con 0x55cec987a0f0 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 10 client.26218 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 10 client.26218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 20 client.26218 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98ef3f0 0x55cec98f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 4 client.26218 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cec97eb790 con 0x55cec98ef3f0 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f63e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98ef3f0 0x55cec98f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.091+0000 7f5f4fe00640 10 client.26218 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.092+0000 7f5f63e00640 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98ef3f0 0x55cec98f17e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a86949e98f4493ef server_cookie=742928ada3b84a0c in_seq=0 out_seq=0 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.092+0000 7f5f4fe00640 1 -- 172.21.15.119:0/139570728 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cec97eb790 con 0x55cec98ef3f0 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.092+0000 7f5f4fe00640 10 client.26218 handle_command_reply: tid=0 2024-03-28T18:47:15.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.093+0000 7f5f73c80740 1 client.26218 shutdown 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 msgr2=0x55cec987c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 0x55cec987c920 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c00 tx=0x7f5f54009230 comp rx=0 tx=0).stop 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 shutdown_connections 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55cec98ef3f0 0x55cec98f17e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cec9883460 0x55cec987d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cec987add0 0x55cec987ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 --2- 172.21.15.119:0/139570728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cec987a0f0 0x55cec987c920 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.094+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 >> 172.21.15.119:0/139570728 conn(0x55cec9841200 msgr2=0x55cec9875f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.095+0000 7f5f4fe00640 0 client.26218 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:15.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.095+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 shutdown_connections 2024-03-28T18:47:15.095 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:15.096 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:47:15.096 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:47:15.096 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:15.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.095+0000 7f5f73c80740 1 -- 172.21.15.119:0/139570728 wait complete. 2024-03-28T18:47:15.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.095+0000 7f5f73c80740 20 client.26218 trim_cache size 0 max 16384 2024-03-28T18:47:15.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.099+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 msgr2=0x7f5f500a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.099+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 0x7f5f500a5000 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009920 tx=0x7f5f54008040 comp rx=0 tx=0).stop 2024-03-28T18:47:15.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.099+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 msgr2=0x7f5f6c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.099+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c06bf50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c009fd0 tx=0x7f5f5c00b870 comp rx=0 tx=0).stop 2024-03-28T18:47:15.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.099+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 shutdown_connections 2024-03-28T18:47:15.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5f500a2b40 0x7f5f500a5000 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f6c109730 0x7f5f6c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f6c1040e0 0x7f5f6c06bf50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 --2- 172.21.15.119:0/2600678488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5f6c103720 0x7f5f6c072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 >> 172.21.15.119:0/2600678488 conn(0x7f5f6c0ff1b0 msgr2=0x7f5f6c0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.100+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 shutdown_connections 2024-03-28T18:47:15.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.101+0000 7f5f6b200640 1 -- 172.21.15.119:0/2600678488 wait complete. 2024-03-28T18:47:15.123 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:47:15.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:47:15.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.491+0000 7f80c1a00640 1 -- 172.21.15.119:0/3540756508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc107570 msgr2=0x7f80bc107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.491+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3540756508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc107570 0x7f80bc107950 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009930 tx=0x7f80ac02bc60 comp rx=0 tx=0).stop 2024-03-28T18:47:15.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.492+0000 7f80c1a00640 1 -- 172.21.15.119:0/3540756508 shutdown_connections 2024-03-28T18:47:15.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.492+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3540756508 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc101e70 0x7f80bc10e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.492+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3540756508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc1014b0 0x7f80bc101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.492+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3540756508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc107570 0x7f80bc107950 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.492+0000 7f80c1a00640 1 -- 172.21.15.119:0/3540756508 >> 172.21.15.119:0/3540756508 conn(0x7f80bc0fcf80 msgr2=0x7f80bc0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:15.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.493+0000 7f80c1a00640 1 -- 172.21.15.119:0/3540756508 shutdown_connections 2024-03-28T18:47:15.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.493+0000 7f80c1a00640 1 -- 172.21.15.119:0/3540756508 wait complete. 2024-03-28T18:47:15.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.494+0000 7f80c1a00640 1 Processor -- start 2024-03-28T18:47:15.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.494+0000 7f80c1a00640 1 -- start start 2024-03-28T18:47:15.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc1014b0 0x7f80bc19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc107570 0x7f80bc10b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc111900 con 0x7f80bc1014b0 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc111780 con 0x7f80bc107570 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80c1a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc111a80 con 0x7f80bc101e70 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.495+0000 7f80bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44882/0 (socket says 172.21.15.119:44882) 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 -- 172.21.15.119:0/3762957191 learned_addr learned my addr 172.21.15.119:0/3762957191 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80c1000640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc107570 0x7f80bc10b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 -- 172.21.15.119:0/3762957191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc107570 msgr2=0x7f80bc10b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc107570 0x7f80bc10b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 -- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc1014b0 msgr2=0x7f80bc19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc1014b0 0x7f80bc19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.496+0000 7f80bb400640 1 -- 172.21.15.119:0/3762957191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc10bb80 con 0x7f80bc101e70 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.497+0000 7f80bb400640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f80a800b4a0 tx=0x7f80a800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.497+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8014070 con 0x7f80bc101e70 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.497+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a8010af0 con 0x7f80bc101e70 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.497+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8017d40 con 0x7f80bc101e70 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.497+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc10be70 con 0x7f80bc101e70 2024-03-28T18:47:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.498+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc19a5f0 con 0x7f80bc101e70 2024-03-28T18:47:15.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.499+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc102bd0 con 0x7f80bc101e70 2024-03-28T18:47:15.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.506+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f80a8017070 con 0x7f80bc101e70 2024-03-28T18:47:15.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.513+0000 7f80b8c00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 0x7f80a00a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:15.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.514+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f80a80ce540 con 0x7f80bc101e70 2024-03-28T18:47:15.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.515+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80a80ce960 con 0x7f80bc101e70 2024-03-28T18:47:15.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.516+0000 7f80bbe00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 0x7f80a00a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:15.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:15.517+0000 7f80bbe00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 0x7f80a00a4f70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009230 tx=0x7f80ac000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:15 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:15.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:15 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:15.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:15 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:47:16.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.005+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80bc10c0a0 con 0x7f80bc101e70 2024-03-28T18:47:16.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.006+0000 7f80b8c00640 1 -- 172.21.15.119:0/3762957191 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f80a809b2f0 con 0x7f80bc101e70 2024-03-28T18:47:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:47:16.007 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:47:16.007 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:47:16.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 msgr2=0x7f80a00a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 0x7f80a00a4f70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009230 tx=0x7f80ac000bc0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 msgr2=0x7f80bc19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f80a800b4a0 tx=0x7f80a800b960 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 shutdown_connections 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f80a00a2ab0 0x7f80a00a4f70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f80bc107570 0x7f80bc10b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80bc101e70 0x7f80bc19ef10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 --2- 172.21.15.119:0/3762957191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80bc1014b0 0x7f80bc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.009+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 >> 172.21.15.119:0/3762957191 conn(0x7f80bc0fcf80 msgr2=0x7f80bc10c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.010+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 shutdown_connections 2024-03-28T18:47:16.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.010+0000 7f80c1a00640 1 -- 172.21.15.119:0/3762957191 wait complete. 2024-03-28T18:47:16.027 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:16.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:47:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.383+0000 7fbf57e00640 1 -- 172.21.15.119:0/3777574371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 msgr2=0x7fbf50100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.383+0000 7fbf57e00640 1 --2- 172.21.15.119:0/3777574371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf50100ab0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009900 tx=0x7fbf4402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.384+0000 7fbf57e00640 1 -- 172.21.15.119:0/3777574371 shutdown_connections 2024-03-28T18:47:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.384+0000 7fbf57e00640 1 --2- 172.21.15.119:0/3777574371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50105ad0 0x7fbf50105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.384+0000 7fbf57e00640 1 --2- 172.21.15.119:0/3777574371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50100ff0 0x7fbf501054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.384+0000 7fbf57e00640 1 --2- 172.21.15.119:0/3777574371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf50100ab0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.384+0000 7fbf57e00640 1 -- 172.21.15.119:0/3777574371 >> 172.21.15.119:0/3777574371 conn(0x7fbf50076920 msgr2=0x7fbf50076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.386+0000 7fbf57e00640 1 -- 172.21.15.119:0/3777574371 shutdown_connections 2024-03-28T18:47:16.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.386+0000 7fbf57e00640 1 -- 172.21.15.119:0/3777574371 wait complete. 2024-03-28T18:47:16.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.388+0000 7fbf57e00640 1 Processor -- start 2024-03-28T18:47:16.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 -- start start 2024-03-28T18:47:16.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50100ff0 0x7fbf5019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 0x7fbf501a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501180f0 con 0x7fbf50100ff0 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50117f70 con 0x7fbf501006d0 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.389+0000 7fbf57e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50118270 con 0x7fbf50105ad0 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 0x7fbf501a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54160/0 (socket says 172.21.15.119:54160) 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 -- 172.21.15.119:0/1487634282 learned_addr learned my addr 172.21.15.119:0/1487634282 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf54a00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50100ff0 0x7fbf5019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 -- 172.21.15.119:0/1487634282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 msgr2=0x7fbf501a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 0x7fbf501a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 -- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50100ff0 msgr2=0x7fbf5019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50100ff0 0x7fbf5019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55400640 1 -- 172.21.15.119:0/1487634282 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf501a35c0 con 0x7fbf501006d0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.390+0000 7fbf55e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 0x7fbf501a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf55400640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbf44000c00 tx=0x7fbf440043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf4e000640 1 -- 172.21.15.119:0/1487634282 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44042020 con 0x7fbf501006d0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf4e000640 1 -- 172.21.15.119:0/1487634282 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf440054e0 con 0x7fbf501006d0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf4e000640 1 -- 172.21.15.119:0/1487634282 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44003ab0 con 0x7fbf501006d0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf501a3850 con 0x7fbf501006d0 2024-03-28T18:47:16.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.391+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501ab140 con 0x7fbf501006d0 2024-03-28T18:47:16.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 Processor -- start 2024-03-28T18:47:16.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 -- start start 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf4e000640 1 -- 172.21.15.119:0/1487634282 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbf44045020 con 0x7fbf501006d0 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319d4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56553192c2c0 con 0x5655319c86d0 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655318d4c90 con 0x5655319c9080 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.396+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531928360 con 0x5655319d1c00 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54176/0 (socket says 172.21.15.119:54176) 2024-03-28T18:47:16.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 -- 172.21.15.119:0/89490021 learned_addr learned my addr 172.21.15.119:0/89490021 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55e00640 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 -- 172.21.15.119:0/89490021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 msgr2=0x5655319d4050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:16.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319d4050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 -- 172.21.15.119:0/89490021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 msgr2=0x5655319c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55400640 1 -- 172.21.15.119:0/89490021 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655318f9080 con 0x5655319c9080 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf55e00640 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.397+0000 7fbf4e000640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 0x7fbf380a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.398+0000 7fbf4e000640 1 -- 172.21.15.119:0/1487634282 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbf440f0980 con 0x7fbf501006d0 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.398+0000 7fbf55400640 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbf440ee0f0 tx=0x7fbf440b93f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6be562b9940f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.398+0000 7fbf55e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 0x7fbf380a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.399+0000 7fbf3e600640 1 -- 172.21.15.119:0/89490021 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440bce60 con 0x5655319c9080 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.399+0000 7fbf3e600640 1 -- 172.21.15.119:0/89490021 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf440bd790 con 0x5655319c9080 2024-03-28T18:47:16.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.399+0000 7fbf3e600640 1 -- 172.21.15.119:0/89490021 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440bc9f0 con 0x5655319c9080 2024-03-28T18:47:16.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 -- 172.21.15.119:0/89490021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 msgr2=0x5655319d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbf440ee0f0 tx=0x7fbf440b93f0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 -- 172.21.15.119:0/89490021 shutdown_connections 2024-03-28T18:47:16.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319d4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1630 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/89490021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 -- 172.21.15.119:0/89490021 >> 172.21.15.119:0/89490021 conn(0x5655319c2da0 msgr2=0x5655319c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf55e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 0x7fbf380a4ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbf48005fd0 tx=0x7fbf48009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 -- 172.21.15.119:0/89490021 shutdown_connections 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.400+0000 7fbf5ab86740 1 -- 172.21.15.119:0/89490021 wait complete. 2024-03-28T18:47:16.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.401+0000 7fbf5ab86740 1 -- start start 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.401+0000 7fbf5ab86740 1 Processor -- start 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 0x5655319d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x5655319ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319d1c00 0x5655319cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655319e5940 con 0x5655319c9080 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655318e80d0 con 0x5655319d1c00 2024-03-28T18:47:16.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655319e5ac0 con 0x5655319c86d0 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf5ab86740 10 client.0 _ll_register_callbacks cb 0x5655318ece80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf54a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x5655319ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf54a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x5655319ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55084/0 (socket says 172.21.15.119:55084) 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.402+0000 7fbf54a00640 1 -- 172.21.15.119:0/1373819691 learned_addr learned my addr 172.21.15.119:0/1373819691 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 0x5655319d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55e00640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319d1c00 0x5655319cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 -- 172.21.15.119:0/1373819691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319d1c00 msgr2=0x5655319cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319d1c00 0x5655319cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 -- 172.21.15.119:0/1373819691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 msgr2=0x5655319ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x5655319ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.403+0000 7fbf55400640 1 -- 172.21.15.119:0/1373819691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5655316403f0 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf55400640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 0x5655319d1520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5655319c81a0 tx=0x7fbf4402c1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38545f2a7516fedd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf32a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440bcbd0 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbf440b9700 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf440bd790 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf440bc9f0 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf5ab86740 10 client.26230 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.404+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56553170a7d0 con 0x5655319c86d0 2024-03-28T18:47:16.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.405+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf440bd790 con 0x5655319c86d0 2024-03-28T18:47:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.405+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbf440b8de0 con 0x5655319c86d0 2024-03-28T18:47:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 10 client.26230 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 10 client.26230 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 20 client.26230 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5655319f2610 0x5655319f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 4 client.26230 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565531939800 con 0x5655319f2610 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.406+0000 7fbf54a00640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5655319f2610 0x5655319f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.407+0000 7fbf54a00640 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5655319f2610 0x5655319f4a00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4283278bd7271ba5 server_cookie=a5d5bdfc5003963 in_seq=0 out_seq=0 2024-03-28T18:47:16.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.407+0000 7fbf32a00640 10 client.26230 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:16.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.408+0000 7fbf32a00640 1 -- 172.21.15.119:0/1373819691 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565531939800 con 0x5655319f2610 2024-03-28T18:47:16.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.408+0000 7fbf32a00640 10 client.26230 handle_command_reply: tid=0 2024-03-28T18:47:16.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.408+0000 7fbf5ab86740 1 client.26230 shutdown 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.408+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 msgr2=0x5655319d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.408+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 0x5655319d1520 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x5655319c81a0 tx=0x7fbf4402c1f0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 shutdown_connections 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5655319f2610 0x5655319f4a00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319d1c00 0x5655319cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x5655319ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/1373819691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319c86d0 0x5655319d1520 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 >> 172.21.15.119:0/1373819691 conn(0x5655319c2da0 msgr2=0x5655319c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf32a00640 0 client.26230 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 shutdown_connections 2024-03-28T18:47:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.409+0000 7fbf5ab86740 1 -- 172.21.15.119:0/1373819691 wait complete. 2024-03-28T18:47:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.412+0000 7fbf5ab86740 20 client.26230 trim_cache size 0 max 16384 2024-03-28T18:47:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.444+0000 7fbf5ab86740 1 Processor -- start 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 -- start start 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50110250 con 0x5655319c86d0 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501acae0 con 0x5655319c9080 2024-03-28T18:47:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.445+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501ab7c0 con 0x5655319d1c00 2024-03-28T18:47:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.448+0000 7fbf55e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.449+0000 7fbf55e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44912/0 (socket says 172.21.15.119:44912) 2024-03-28T18:47:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.449+0000 7fbf55e00640 1 -- 172.21.15.119:0/882563140 learned_addr learned my addr 172.21.15.119:0/882563140 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.449+0000 7fbf54a00640 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.448+0000 7fbf55400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.452+0000 7fbf54a00640 1 -- 172.21.15.119:0/882563140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 msgr2=0x5655319cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.452+0000 7fbf54a00640 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.452+0000 7fbf54a00640 1 -- 172.21.15.119:0/882563140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 msgr2=0x5655319d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.452+0000 7fbf54a00640 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.452+0000 7fbf54a00640 1 -- 172.21.15.119:0/882563140 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655319cf180 con 0x5655319c9080 2024-03-28T18:47:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.454+0000 7fbf54a00640 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1520 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x5655319f4f40 tx=0x7fbf40009110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c06a60580e23315 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.455+0000 7fbf3dc00640 1 -- 172.21.15.119:0/882563140 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40009d30 con 0x5655319c9080 2024-03-28T18:47:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.455+0000 7fbf3dc00640 1 -- 172.21.15.119:0/882563140 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf40016e30 con 0x5655319c9080 2024-03-28T18:47:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.455+0000 7fbf3dc00640 1 -- 172.21.15.119:0/882563140 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40014e50 con 0x5655319c9080 2024-03-28T18:47:16.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.457+0000 7fbf5ab86740 1 -- 172.21.15.119:0/882563140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 msgr2=0x5655319d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.457+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1520 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x5655319f4f40 tx=0x7fbf40009110 comp rx=0 tx=0).stop 2024-03-28T18:47:16.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.458+0000 7fbf5ab86740 1 -- 172.21.15.119:0/882563140 shutdown_connections 2024-03-28T18:47:16.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c9080 0x5655319d1520 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/882563140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c86d0 0x5655319d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 -- 172.21.15.119:0/882563140 >> 172.21.15.119:0/882563140 conn(0x5655319c2da0 msgr2=0x5655319c1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 -- 172.21.15.119:0/882563140 shutdown_connections 2024-03-28T18:47:16.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.459+0000 7fbf5ab86740 1 -- 172.21.15.119:0/882563140 wait complete. 2024-03-28T18:47:16.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.460+0000 7fbf5ab86740 1 -- start start 2024-03-28T18:47:16.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.460+0000 7fbf5ab86740 1 Processor -- start 2024-03-28T18:47:16.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.461+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x565531a47090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319ea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501ac480 con 0x5655319c9080 2024-03-28T18:47:16.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501abe20 con 0x5655319c86d0 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf50068920 con 0x5655319d1c00 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.462+0000 7fbf5ab86740 10 client.0 _ll_register_callbacks cb 0x7fbf440f68e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54214/0 (socket says 172.21.15.119:54214) 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 -- 172.21.15.119:0/431873993 learned_addr learned my addr 172.21.15.119:0/431873993 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 -- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 msgr2=0x5655319ea180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319ea180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 -- 172.21.15.119:0/431873993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 msgr2=0x565531a47090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:47:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x565531a47090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.463+0000 7fbf55400640 1 -- 172.21.15.119:0/431873993 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5655319ea990 con 0x5655319c86d0 2024-03-28T18:47:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.464+0000 7fbf55400640 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbf4402c2e0 tx=0x7fbf4402cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644fde06ddb53084 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.464+0000 7fbf32a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.464+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf440bce60 con 0x5655319c86d0 2024-03-28T18:47:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.464+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbf440c53a0 con 0x5655319c86d0 2024-03-28T18:47:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.465+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf40016e30 con 0x5655319c86d0 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.465+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf440bcbd0 con 0x5655319c86d0 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.465+0000 7fbf5ab86740 10 client.26252 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.465+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5655319ea990 con 0x5655319c86d0 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.465+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf40016e30 con 0x5655319c86d0 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbf440bd2f0 con 0x5655319c86d0 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 10 client.26252 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 10 client.26252 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 20 client.26252 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565531a5f990 0x565531a61df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 4 client.26252 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:16.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565531933b50 con 0x565531a5f990 2024-03-28T18:47:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.466+0000 7fbf54a00640 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565531a5f990 0x565531a61df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.467+0000 7fbf54a00640 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565531a5f990 0x565531a61df0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe1550a05a6f46a8 server_cookie=5b7b6725177565c7 in_seq=0 out_seq=0 2024-03-28T18:47:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.467+0000 7fbf32a00640 10 client.26252 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.467+0000 7fbf32a00640 1 -- 172.21.15.119:0/431873993 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565531933b50 con 0x565531a5f990 2024-03-28T18:47:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.467+0000 7fbf32a00640 10 client.26252 handle_command_reply: tid=0 2024-03-28T18:47:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.468+0000 7fbf5ab86740 1 client.26252 shutdown 2024-03-28T18:47:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.468+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 msgr2=0x565531a46b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.468+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbf4402c2e0 tx=0x7fbf4402cf00 comp rx=0 tx=0).stop 2024-03-28T18:47:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 shutdown_connections 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565531a5f990 0x565531a61df0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5655319d1c00 0x5655319ea180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5655319c9080 0x565531a47090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 --2- 172.21.15.119:0/431873993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5655319c86d0 0x565531a46b50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.469+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 >> 172.21.15.119:0/431873993 conn(0x5655319c2da0 msgr2=0x5655319c9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.470+0000 7fbf32a00640 0 client.26252 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.473+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 shutdown_connections 2024-03-28T18:47:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.475+0000 7fbf5ab86740 1 -- 172.21.15.119:0/431873993 wait complete. 2024-03-28T18:47:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.475+0000 7fbf5ab86740 20 client.26252 trim_cache size 0 max 16384 2024-03-28T18:47:16.479 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:16.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.480+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 msgr2=0x7fbf380a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.480+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 0x7fbf380a4ef0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbf48005fd0 tx=0x7fbf48009500 comp rx=0 tx=0).stop 2024-03-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.480+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 msgr2=0x7fbf5019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.480+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbf44000c00 tx=0x7fbf440043b0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 shutdown_connections 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbf380a2a30 0x7fbf380a4ef0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbf50105ad0 0x7fbf501a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf50100ff0 0x7fbf5019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 --2- 172.21.15.119:0/1487634282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbf501006d0 0x7fbf5019e5d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 >> 172.21.15.119:0/1487634282 conn(0x7fbf50076920 msgr2=0x7fbf50103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.482+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 shutdown_connections 2024-03-28T18:47:16.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.483+0000 7fbf57e00640 1 -- 172.21.15.119:0/1487634282 wait complete. 2024-03-28T18:47:16.510 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:16.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:47:16.648 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:16 smithi119 ceph-mon[31813]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 512 MiB/s wr, 158 op/s 2024-03-28T18:47:16.649 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:16 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3762957191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:16 smithi196 ceph-mon[36300]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 512 MiB/s wr, 158 op/s 2024-03-28T18:47:16.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:16 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3762957191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:16 smithi131 ceph-mon[36065]: pgmap v99: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 512 MiB/s wr, 158 op/s 2024-03-28T18:47:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:16 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3762957191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 -- 172.21.15.119:0/2593213362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb4100ff0 msgr2=0x7f5eb41054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/2593213362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb4100ff0 0x7f5eb41054b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009930 tx=0x7f5ea002bcb0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 -- 172.21.15.119:0/2593213362 shutdown_connections 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/2593213362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4105ad0 0x7f5eb4105f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/2593213362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb4100ff0 0x7f5eb41054b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/2593213362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb41006d0 0x7f5eb4100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 -- 172.21.15.119:0/2593213362 >> 172.21.15.119:0/2593213362 conn(0x7f5eb4076920 msgr2=0x7f5eb4076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 -- 172.21.15.119:0/2593213362 shutdown_connections 2024-03-28T18:47:16.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.807+0000 7f5ebbc00640 1 -- 172.21.15.119:0/2593213362 wait complete. 2024-03-28T18:47:16.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.808+0000 7f5ebbc00640 1 Processor -- start 2024-03-28T18:47:16.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.808+0000 7f5ebbc00640 1 -- start start 2024-03-28T18:47:16.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.808+0000 7f5ebbc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5ebbc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 0x7f5eb419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5ebbc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 0x7f5eb41a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5ebbc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4118050 con 0x7f5eb4105ad0 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5ebbc00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb4117ed0 con 0x7f5eb4100ff0 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5ebbc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41181d0 con 0x7f5eb41006d0 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44932/0 (socket says 172.21.15.119:44932) 2024-03-28T18:47:16.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 -- 172.21.15.119:0/3013487991 learned_addr learned my addr 172.21.15.119:0/3013487991 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 0x7f5eb419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 -- 172.21.15.119:0/3013487991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 msgr2=0x7f5eb419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 0x7f5eb419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 -- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 msgr2=0x7f5eb41a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 0x7f5eb41a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 0x7f5eb41a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9200640 1 -- 172.21.15.119:0/3013487991 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb41a3600 con 0x7f5eb41006d0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 0x7f5eb419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.809+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 0x7f5eb41a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800d950 tx=0x7f5ea800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5eb1600640 1 -- 172.21.15.119:0/3013487991 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8017070 con 0x7f5eb41006d0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5eb1600640 1 -- 172.21.15.119:0/3013487991 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea8009c30 con 0x7f5eb41006d0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5eb1600640 1 -- 172.21.15.119:0/3013487991 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea800f6b0 con 0x7f5eb41006d0 2024-03-28T18:47:16.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb41a38f0 con 0x7f5eb41006d0 2024-03-28T18:47:16.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.810+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb41ab180 con 0x7f5eb41006d0 2024-03-28T18:47:16.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.815+0000 7f5ebe987740 1 Processor -- start 2024-03-28T18:47:16.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.816+0000 7f5eb1600640 1 -- 172.21.15.119:0/3013487991 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5ea80109d0 con 0x7f5eb41006d0 2024-03-28T18:47:16.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 -- start start 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0bfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c8c00 0x563eea0cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eea0232c0 con 0x563eea0c8c00 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee9fcbc90 con 0x563eea0bf6d0 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.817+0000 7f5ebe987740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eea01f360 con 0x563eea0c0080 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.818+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.818+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44934/0 (socket says 172.21.15.119:44934) 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.818+0000 7f5eb9200640 1 -- 172.21.15.119:0/2428813901 learned_addr learned my addr 172.21.15.119:0/2428813901 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 -- 172.21.15.119:0/2428813901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 msgr2=0x563eea0bfab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0bfab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 -- 172.21.15.119:0/2428813901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c8c00 msgr2=0x563eea0cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb1600640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 0x7f5e9c0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c8c00 0x563eea0cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 -- 172.21.15.119:0/2428813901 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ee9ff0080 con 0x563eea0c0080 2024-03-28T18:47:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.819+0000 7f5eb9200640 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80cc1e0 tx=0x7f5ea80af1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8011450adb54a371 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.820+0000 7f5ea6000640 1 -- 172.21.15.119:0/2428813901 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea809af50 con 0x563eea0c0080 2024-03-28T18:47:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.820+0000 7f5eb1600640 1 -- 172.21.15.119:0/3013487991 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5ea80ceb80 con 0x7f5eb41006d0 2024-03-28T18:47:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.820+0000 7f5ea6000640 1 -- 172.21.15.119:0/2428813901 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea809b880 con 0x563eea0c0080 2024-03-28T18:47:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.820+0000 7f5ea6000640 1 -- 172.21.15.119:0/2428813901 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80c2dd0 con 0x563eea0c0080 2024-03-28T18:47:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.821+0000 7f5ebe987740 1 -- 172.21.15.119:0/2428813901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 msgr2=0x563eea0c8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.821+0000 7f5ebe987740 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80cc1e0 tx=0x7f5ea80af1a0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 0x7f5e9c0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 -- 172.21.15.119:0/2428813901 shutdown_connections 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c8c00 0x563eea0cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c0080 0x563eea0c8630 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 --2- 172.21.15.119:0/2428813901 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0bfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 -- 172.21.15.119:0/2428813901 >> 172.21.15.119:0/2428813901 conn(0x563eea0b9da0 msgr2=0x563eea0ba180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 -- 172.21.15.119:0/2428813901 shutdown_connections 2024-03-28T18:47:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.828+0000 7f5ebe987740 1 -- 172.21.15.119:0/2428813901 wait complete. 2024-03-28T18:47:16.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.829+0000 7f5ebe987740 1 -- start start 2024-03-28T18:47:16.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.829+0000 7f5ebe987740 1 Processor -- start 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eea0cbb70 con 0x563eea0c0080 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ee9fdf0d0 con 0x563eea0bf6d0 2024-03-28T18:47:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.830+0000 7f5ebe987740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563eea0cbcf0 con 0x563eea0c8c00 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5eb3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54266/0 (socket says 172.21.15.119:54266) 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5eb9200640 1 -- 172.21.15.119:0/3518450161 learned_addr learned my addr 172.21.15.119:0/3518450161 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.831+0000 7f5ebe987740 10 client.0 _ll_register_callbacks cb 0x563ee9fe3e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:16.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 -- 172.21.15.119:0/3518450161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 msgr2=0x563eea0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 -- 172.21.15.119:0/3518450161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 msgr2=0x563eea0c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 -- 172.21.15.119:0/3518450161 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ee9d373f0 con 0x563eea0bf6d0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 0x7f5e9c0b5ec0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5eb9200640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5ea809ae80 tx=0x7f5ea8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44aa81bd99aec64f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5e9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80b4040 con 0x563eea0bf6d0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.832+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5ea80d21e0 con 0x563eea0bf6d0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.833+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ea809b880 con 0x563eea0bf6d0 2024-03-28T18:47:16.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.833+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ea80c06b0 con 0x563eea0bf6d0 2024-03-28T18:47:16.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.834+0000 7f5ebe987740 10 client.26258 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:16.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.834+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563ee9e017d0 con 0x563eea0bf6d0 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.834+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea80bde50 con 0x563eea0bf6d0 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5ea80d5030 con 0x563eea0bf6d0 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 10 client.26258 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 10 client.26258 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 20 client.26258 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea0d89d0 0x563eea0dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 4 client.26258 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:16.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563eea030800 con 0x563eea0d89d0 2024-03-28T18:47:16.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.835+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea0d89d0 0x563eea0dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.836+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea0d89d0 0x563eea0dadc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d7f36c137caea623 server_cookie=f833a71e2d76bb0c in_seq=0 out_seq=0 2024-03-28T18:47:16.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.836+0000 7f5e9aa00640 10 client.26258 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:16.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.837+0000 7f5e9aa00640 1 -- 172.21.15.119:0/3518450161 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563eea030800 con 0x563eea0d89d0 2024-03-28T18:47:16.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.837+0000 7f5e9aa00640 10 client.26258 handle_command_reply: tid=0 2024-03-28T18:47:16.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.837+0000 7f5ebe987740 1 client.26258 shutdown 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 msgr2=0x563eea0c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5ea809ae80 tx=0x7f5ea8000bc0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 shutdown_connections 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea0d89d0 0x563eea0dadc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 --2- 172.21.15.119:0/3518450161 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0bf6d0 0x563eea0c85e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5e9aa00640 0 client.26258 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 >> 172.21.15.119:0/3518450161 conn(0x563eea0b9da0 msgr2=0x563eea0bbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 shutdown_connections 2024-03-28T18:47:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 1 -- 172.21.15.119:0/3518450161 wait complete. 2024-03-28T18:47:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.838+0000 7f5ebe987740 20 client.26258 trim_cache size 0 max 16384 2024-03-28T18:47:16.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.899+0000 7f5ebe987740 1 Processor -- start 2024-03-28T18:47:16.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 -- start start 2024-03-28T18:47:16.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c8c00 0x563eea0c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41ab3b0 con 0x563eea0c0080 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41acd00 con 0x563eea0c8c00 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.903+0000 7f5ebe987740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41ab9e0 con 0x563eea0bf6d0 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.904+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.905+0000 7f5eb9c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c8c00 0x563eea0c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.905+0000 7f5eb9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44948/0 (socket says 172.21.15.119:44948) 2024-03-28T18:47:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.905+0000 7f5eb9200640 1 -- 172.21.15.119:0/944648814 learned_addr learned my addr 172.21.15.119:0/944648814 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.914+0000 7f5eb9200640 1 -- 172.21.15.119:0/944648814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c8c00 msgr2=0x563eea0c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.914+0000 7f5eb9200640 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c8c00 0x563eea0c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.914+0000 7f5eb9200640 1 -- 172.21.15.119:0/944648814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 msgr2=0x563eea0bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:16.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.914+0000 7f5eb9200640 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.914+0000 7f5eb9200640 1 -- 172.21.15.119:0/944648814 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563eea0c6420 con 0x563eea0bf6d0 2024-03-28T18:47:16.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.915+0000 7f5eb9200640 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009610 tx=0x7f5ea809f170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a76513171daf5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.918+0000 7f5ea5600640 1 -- 172.21.15.119:0/944648814 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80b4040 con 0x563eea0bf6d0 2024-03-28T18:47:16.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.918+0000 7f5ea5600640 1 -- 172.21.15.119:0/944648814 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea80bde50 con 0x563eea0bf6d0 2024-03-28T18:47:16.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.918+0000 7f5ea5600640 1 -- 172.21.15.119:0/944648814 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea80bdab0 con 0x563eea0bf6d0 2024-03-28T18:47:16.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.921+0000 7f5ebe987740 1 -- 172.21.15.119:0/944648814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 msgr2=0x563eea0d4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.921+0000 7f5ebe987740 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009610 tx=0x7f5ea809f170 comp rx=0 tx=0).stop 2024-03-28T18:47:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.922+0000 7f5ebe987740 1 -- 172.21.15.119:0/944648814 shutdown_connections 2024-03-28T18:47:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.922+0000 7f5ebe987740 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c8c00 0x563eea0c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.922+0000 7f5ebe987740 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0c0080 0x563eea0bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.922+0000 7f5ebe987740 1 --2- 172.21.15.119:0/944648814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0bf6d0 0x563eea0d4f50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.923+0000 7f5ebe987740 1 -- 172.21.15.119:0/944648814 >> 172.21.15.119:0/944648814 conn(0x563eea0b9da0 msgr2=0x563eea0ba180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.923+0000 7f5ebe987740 1 -- 172.21.15.119:0/944648814 shutdown_connections 2024-03-28T18:47:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.923+0000 7f5ebe987740 1 -- 172.21.15.119:0/944648814 wait complete. 2024-03-28T18:47:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.923+0000 7f5ebe987740 1 -- start start 2024-03-28T18:47:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.924+0000 7f5ebe987740 1 Processor -- start 2024-03-28T18:47:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.926+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0bf6d0 0x563eea0ddb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.926+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5ebe987740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0db440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5eb3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5eb3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54298/0 (socket says 172.21.15.119:54298) 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5eb3e00640 1 -- 172.21.15.119:0/1900848386 learned_addr learned my addr 172.21.15.119:0/1900848386 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41ac6a0 con 0x563eea0bf6d0 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb41ac040 con 0x563eea0c0080 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb40688e0 con 0x563eea0c8c00 2024-03-28T18:47:16.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5ebe987740 10 client.0 _ll_register_callbacks cb 0x7f5ea80b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:16.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.927+0000 7f5eb9c00640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.928+0000 7f5eb3e00640 1 -- 172.21.15.119:0/1900848386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 msgr2=0x563eea0db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.928+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0db440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.928+0000 7f5eb3e00640 1 -- 172.21.15.119:0/1900848386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0bf6d0 msgr2=0x563eea0ddb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.928+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0bf6d0 0x563eea0ddb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.928+0000 7f5eb3e00640 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563eea0dbc10 con 0x563eea0c0080 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.929+0000 7f5e9aa00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.929+0000 7f5eb3e00640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4077270 tx=0x7f5ea0009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef2eafd607a666c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:16.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.929+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0046020 con 0x563eea0c0080 2024-03-28T18:47:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.929+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ea80bde50 con 0x563eea0c0080 2024-03-28T18:47:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.929+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5ea00040c0 con 0x563eea0c0080 2024-03-28T18:47:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.930+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ea0051050 con 0x563eea0c0080 2024-03-28T18:47:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.930+0000 7f5ebe987740 10 client.26264 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.930+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563eea133f30 con 0x563eea0c0080 2024-03-28T18:47:16.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.931+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ea80bde50 con 0x563eea0c0080 2024-03-28T18:47:16.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.931+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5ea0042ad0 con 0x563eea0c0080 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 10 client.26264 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 10 client.26264 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 20 client.26264 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea134690 0x563eea136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 4 client.26264 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.932+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563eea02ab50 con 0x563eea134690 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.933+0000 7f5eb9200640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea134690 0x563eea136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:16.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.934+0000 7f5e9aa00640 10 client.26264 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:16.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.935+0000 7f5eb9200640 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea134690 0x563eea136ab0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0ff0cfd2d61fae0 server_cookie=c8d574a9c869d3fd in_seq=0 out_seq=0 2024-03-28T18:47:16.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5e9aa00640 1 -- 172.21.15.119:0/1900848386 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563eea02ab50 con 0x563eea134690 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5e9aa00640 10 client.26264 handle_command_reply: tid=0 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 client.26264 shutdown 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 msgr2=0x563eea0daf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4077270 tx=0x7f5ea0009560 comp rx=0 tx=0).stop 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 shutdown_connections 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563eea0c8c00 0x563eea0db440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563eea0c0080 0x563eea0daf00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563eea134690 0x563eea136ab0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 --2- 172.21.15.119:0/1900848386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563eea0bf6d0 0x563eea0ddb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.937+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 >> 172.21.15.119:0/1900848386 conn(0x563eea0b9da0 msgr2=0x563eea0b8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.938+0000 7f5e9aa00640 0 client.26264 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.938+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 shutdown_connections 2024-03-28T18:47:16.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.938+0000 7f5ebe987740 1 -- 172.21.15.119:0/1900848386 wait complete. 2024-03-28T18:47:16.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.938+0000 7f5ebe987740 20 client.26264 trim_cache size 0 max 16384 2024-03-28T18:47:16.939 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:16.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.939+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 msgr2=0x7f5e9c0b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.939+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 0x7f5e9c0b5ec0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5eac005fd0 tx=0x7f5eac009500 comp rx=0 tx=0).stop 2024-03-28T18:47:16.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.939+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 msgr2=0x7f5eb419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:16.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.939+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800d950 tx=0x7f5ea800de10 comp rx=0 tx=0).stop 2024-03-28T18:47:16.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 shutdown_connections 2024-03-28T18:47:16.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5e9c0b3a00 0x7f5e9c0b5ec0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5eb4105ad0 0x7f5eb41a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5eb4100ff0 0x7f5eb419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 --2- 172.21.15.119:0/3013487991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5eb41006d0 0x7f5eb419e610 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:16.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 >> 172.21.15.119:0/3013487991 conn(0x7f5eb4076920 msgr2=0x7f5eb4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:16.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.943+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 shutdown_connections 2024-03-28T18:47:16.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:16.946+0000 7f5ebbc00640 1 -- 172.21.15.119:0/3013487991 wait complete. 2024-03-28T18:47:16.989 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:16.989 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:47:17.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.254+0000 7fec1de00640 1 -- 172.21.15.119:0/1955943212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 msgr2=0x7fec1811d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.254+0000 7fec1de00640 1 --2- 172.21.15.119:0/1955943212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 0x7fec1811d270 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec10009860 tx=0x7fec1002bbb0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.255+0000 7fec1de00640 1 -- 172.21.15.119:0/1955943212 shutdown_connections 2024-03-28T18:47:17.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.255+0000 7fec1de00640 1 --2- 172.21.15.119:0/1955943212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 0x7fec1811d270 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.255+0000 7fec1de00640 1 --2- 172.21.15.119:0/1955943212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec1810def0 0x7fec1810e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.255+0000 7fec1de00640 1 --2- 172.21.15.119:0/1955943212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec1810fec0 0x7fec1810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.255+0000 7fec1de00640 1 -- 172.21.15.119:0/1955943212 >> 172.21.15.119:0/1955943212 conn(0x7fec1806c780 msgr2=0x7fec1806cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.257+0000 7fec1de00640 1 -- 172.21.15.119:0/1955943212 shutdown_connections 2024-03-28T18:47:17.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.257+0000 7fec1de00640 1 -- 172.21.15.119:0/1955943212 wait complete. 2024-03-28T18:47:17.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.259+0000 7fec1de00640 1 Processor -- start 2024-03-28T18:47:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.259+0000 7fec1de00640 1 -- start start 2024-03-28T18:47:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.260+0000 7fec1de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 0x7fec181a7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.260+0000 7fec1de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec181a7570 0x7fec181ab8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.260+0000 7fec1de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec1de00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec181208e0 con 0x7fec181a7570 2024-03-28T18:47:17.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec1de00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18120760 con 0x7fec181abe30 2024-03-28T18:47:17.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec1de00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18120a60 con 0x7fec1811ae80 2024-03-28T18:47:17.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54320/0 (socket says 172.21.15.119:54320) 2024-03-28T18:47:17.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 -- 172.21.15.119:0/3455754162 learned_addr learned my addr 172.21.15.119:0/3455754162 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 -- 172.21.15.119:0/3455754162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 msgr2=0x7fec181a7030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 0x7fec181a7030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 -- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec181a7570 msgr2=0x7fec181ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec181a7570 0x7fec181ab8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.261+0000 7fec17400640 1 -- 172.21.15.119:0/3455754162 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec181b3ba0 con 0x7fec181abe30 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.262+0000 7fec17400640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fec100057c0 tx=0x7fec100349e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.263+0000 7fec1ca00640 1 -- 172.21.15.119:0/3455754162 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10041030 con 0x7fec181abe30 2024-03-28T18:47:17.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.263+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec181b3d70 con 0x7fec181abe30 2024-03-28T18:47:17.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.263+0000 7fec1ca00640 1 -- 172.21.15.119:0/3455754162 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec1002c550 con 0x7fec181abe30 2024-03-28T18:47:17.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.263+0000 7fec1ca00640 1 -- 172.21.15.119:0/3455754162 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec10039040 con 0x7fec181abe30 2024-03-28T18:47:17.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.263+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec181b4220 con 0x7fec181abe30 2024-03-28T18:47:17.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.272+0000 7fec1ca00640 1 -- 172.21.15.119:0/3455754162 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fec10044070 con 0x7fec181abe30 2024-03-28T18:47:17.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.273+0000 7fec20a0f740 1 Processor -- start 2024-03-28T18:47:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 -- start start 2024-03-28T18:47:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c19080 0x559dc1c21630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c21c00 0x559dc1c24050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1b7c2c0 con 0x559dc1c19080 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1b24c90 con 0x559dc1c21c00 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec20a0f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1b78360 con 0x559dc1c186d0 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec0e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec0e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:44968/0 (socket says 172.21.15.119:44968) 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.274+0000 7fec0e000640 1 -- 172.21.15.119:0/1717583128 learned_addr learned my addr 172.21.15.119:0/1717583128 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c21c00 0x559dc1c24050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 -- 172.21.15.119:0/1717583128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c21c00 msgr2=0x559dc1c24050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c21c00 0x559dc1c24050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 -- 172.21.15.119:0/1717583128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c19080 msgr2=0x559dc1c21630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c19080 0x559dc1c21630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 -- 172.21.15.119:0/1717583128 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dc1b49080 con 0x559dc1c186d0 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec1ca00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec0e000640 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec00001b70 tx=0x7fec00001100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbf034fcdebb3ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.275+0000 7fec1ca00640 1 -- 172.21.15.119:0/3455754162 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fec100f0360 con 0x7fec181abe30 2024-03-28T18:47:17.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.276+0000 7fec07e00640 1 -- 172.21.15.119:0/1717583128 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0000d050 con 0x559dc1c186d0 2024-03-28T18:47:17.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.276+0000 7fec07e00640 1 -- 172.21.15.119:0/1717583128 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec00011040 con 0x559dc1c186d0 2024-03-28T18:47:17.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.276+0000 7fec07e00640 1 -- 172.21.15.119:0/1717583128 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00010040 con 0x559dc1c186d0 2024-03-28T18:47:17.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 -- 172.21.15.119:0/1717583128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 msgr2=0x559dc1c18ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec00001b70 tx=0x7fec00001100 comp rx=0 tx=0).stop 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 -- 172.21.15.119:0/1717583128 shutdown_connections 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c21c00 0x559dc1c24050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c19080 0x559dc1c21630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1717583128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c18ab0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 -- 172.21.15.119:0/1717583128 >> 172.21.15.119:0/1717583128 conn(0x559dc1c12da0 msgr2=0x559dc1c13180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 -- 172.21.15.119:0/1717583128 shutdown_connections 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.278+0000 7fec20a0f740 1 -- 172.21.15.119:0/1717583128 wait complete. 2024-03-28T18:47:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.279+0000 7fec20a0f740 1 -- start start 2024-03-28T18:47:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.279+0000 7fec20a0f740 1 Processor -- start 2024-03-28T18:47:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.280+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 0x559dc1c1e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.280+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c1ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.280+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 0x559dc1c1f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.280+0000 7fec20a0f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1c24b70 con 0x559dc1c186d0 2024-03-28T18:47:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.281+0000 7fec20a0f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1b380d0 con 0x559dc1c19080 2024-03-28T18:47:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.281+0000 7fec20a0f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dc1c24cf0 con 0x559dc1c21c00 2024-03-28T18:47:17.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.281+0000 7fec20a0f740 10 client.0 _ll_register_callbacks cb 0x559dc1b3ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:17.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.281+0000 7fec16a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 0x559dc1c1e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.282+0000 7fec16a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 0x559dc1c1e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55170/0 (socket says 172.21.15.119:55170) 2024-03-28T18:47:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.282+0000 7fec16a00640 1 -- 172.21.15.119:0/2782196211 learned_addr learned my addr 172.21.15.119:0/2782196211 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.283+0000 7fec0e000640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c1ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.284+0000 7fec17400640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 0x559dc1c1f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.284+0000 7fec16a00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.286+0000 7fec17400640 1 -- 172.21.15.119:0/2782196211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 msgr2=0x559dc1c1ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.286+0000 7fec17400640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c1ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.286+0000 7fec17400640 1 -- 172.21.15.119:0/2782196211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 msgr2=0x559dc1c1e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.286+0000 7fec17400640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 0x559dc1c1e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.286+0000 7fec17400640 1 -- 172.21.15.119:0/2782196211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dc18903f0 con 0x559dc1c21c00 2024-03-28T18:47:17.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.290+0000 7fec17400640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 0x559dc1c1f470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fec100edb60 tx=0x7fec100bd170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae0e3e819a5ae2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.293+0000 7fec04c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:17.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.293+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec100c5210 con 0x559dc1c21c00 2024-03-28T18:47:17.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.293+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fec100edcd0 con 0x559dc1c21c00 2024-03-28T18:47:17.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.293+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec00011040 con 0x559dc1c21c00 2024-03-28T18:47:17.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.293+0000 7fec16a00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 0x7febfc0a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fec180738a0 tx=0x7fec08004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.294+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec100b90e0 con 0x559dc1c21c00 2024-03-28T18:47:17.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.294+0000 7fec20a0f740 10 client.26260 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:17.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.294+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559dc195a7d0 con 0x559dc1c21c00 2024-03-28T18:47:17.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.294+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec00011040 con 0x559dc1c21c00 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fec1004e060 con 0x559dc1c21c00 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 10 client.26260 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 10 client.26260 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 20 client.26260 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c31870 0x559dc1c33c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 4 client.26260 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559dc1b89800 con 0x559dc1c31870 2024-03-28T18:47:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.295+0000 7fec16a00640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c31870 0x559dc1c33c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.296+0000 7fec04c00640 10 client.26260 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:17.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.297+0000 7fec16a00640 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c31870 0x559dc1c33c60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c69814db3abaff0 server_cookie=27c90a39e0590d3a in_seq=0 out_seq=0 2024-03-28T18:47:17.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.298+0000 7fec04c00640 1 -- 172.21.15.119:0/2782196211 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559dc1b89800 con 0x559dc1c31870 2024-03-28T18:47:17.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.298+0000 7fec04c00640 10 client.26260 handle_command_reply: tid=0 2024-03-28T18:47:17.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.298+0000 7fec20a0f740 1 client.26260 shutdown 2024-03-28T18:47:17.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.299+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 msgr2=0x559dc1c1f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.299+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 0x559dc1c1f470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fec100edb60 tx=0x7fec100bd170 comp rx=0 tx=0).stop 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 shutdown_connections 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c31870 0x559dc1c33c60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c21c00 0x559dc1c1f470 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c1ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 --2- 172.21.15.119:0/2782196211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c186d0 0x559dc1c1e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.300+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 >> 172.21.15.119:0/2782196211 conn(0x559dc1c12da0 msgr2=0x559dc1c14af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.301+0000 7fec04c00640 0 client.26260 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:17.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.303+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 shutdown_connections 2024-03-28T18:47:17.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.304+0000 7fec20a0f740 1 -- 172.21.15.119:0/2782196211 wait complete. 2024-03-28T18:47:17.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.305+0000 7fec20a0f740 20 client.26260 trim_cache size 0 max 16384 2024-03-28T18:47:17.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.361+0000 7fec20a0f740 1 Processor -- start 2024-03-28T18:47:17.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.363+0000 7fec20a0f740 1 -- start start 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.363+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 0x559dc1c21120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.363+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 0x559dc1c11620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.363+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c1f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec20a0f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1807c0a0 con 0x559dc1c21c00 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec20a0f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1807da20 con 0x559dc1c186d0 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec20a0f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1807c700 con 0x559dc1c19080 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec0e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 0x559dc1c11620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec16a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 0x559dc1c21120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec16a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 0x559dc1c21120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54352/0 (socket says 172.21.15.119:54352) 2024-03-28T18:47:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.364+0000 7fec16a00640 1 -- 172.21.15.119:0/1812435704 learned_addr learned my addr 172.21.15.119:0/1812435704 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.365+0000 7fec0e000640 1 -- 172.21.15.119:0/1812435704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 msgr2=0x559dc1c21120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.365+0000 7fec0e000640 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 0x559dc1c21120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.365+0000 7fec0e000640 1 -- 172.21.15.119:0/1812435704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 msgr2=0x559dc1c1f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.365+0000 7fec0e000640 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c1f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.365+0000 7fec0e000640 1 -- 172.21.15.119:0/1812435704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dc1c1a5e0 con 0x559dc1c19080 2024-03-28T18:47:17.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.366+0000 7fec0e000640 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 0x559dc1c11620 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x559dc1c1ffc0 tx=0x7fec00001dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0fe4af929e1c94e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.366+0000 7fec07400640 1 -- 172.21.15.119:0/1812435704 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0000d070 con 0x559dc1c19080 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.368+0000 7fec07400640 1 -- 172.21.15.119:0/1812435704 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec00004510 con 0x559dc1c19080 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.368+0000 7fec07400640 1 -- 172.21.15.119:0/1812435704 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0000d070 con 0x559dc1c19080 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.369+0000 7fec20a0f740 1 -- 172.21.15.119:0/1812435704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 msgr2=0x559dc1c11620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.369+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 0x559dc1c11620 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x559dc1c1ffc0 tx=0x7fec00001dd0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.370+0000 7fec20a0f740 1 -- 172.21.15.119:0/1812435704 shutdown_connections 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.370+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c1f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.370+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c19080 0x559dc1c11620 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.370+0000 7fec20a0f740 1 --2- 172.21.15.119:0/1812435704 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c186d0 0x559dc1c21120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.370+0000 7fec20a0f740 1 -- 172.21.15.119:0/1812435704 >> 172.21.15.119:0/1812435704 conn(0x559dc1c12da0 msgr2=0x559dc1c14910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.371+0000 7fec20a0f740 1 -- 172.21.15.119:0/1812435704 shutdown_connections 2024-03-28T18:47:17.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.371+0000 7fec20a0f740 1 -- 172.21.15.119:0/1812435704 wait complete. 2024-03-28T18:47:17.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.372+0000 7fec20a0f740 1 -- start start 2024-03-28T18:47:17.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.372+0000 7fec20a0f740 1 Processor -- start 2024-03-28T18:47:17.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c29c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c27b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1807d3c0 con 0x559dc1c21c00 2024-03-28T18:47:17.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1807cd60 con 0x559dc1c19080 2024-03-28T18:47:17.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec181a8380 con 0x559dc1c186d0 2024-03-28T18:47:17.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.373+0000 7fec20a0f740 10 client.0 _ll_register_callbacks cb 0x7fec100bc5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:17.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.375+0000 7fec16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.375+0000 7fec16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45010/0 (socket says 172.21.15.119:45010) 2024-03-28T18:47:17.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.375+0000 7fec16a00640 1 -- 172.21.15.119:0/3221713738 learned_addr learned my addr 172.21.15.119:0/3221713738 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.376+0000 7fec17400640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c27b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.380+0000 7fec16a00640 1 -- 172.21.15.119:0/3221713738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 msgr2=0x559dc1c29c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.377+0000 7fec0e000640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c29c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.381+0000 7fec16a00640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c29c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.381+0000 7fec16a00640 1 -- 172.21.15.119:0/3221713738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 msgr2=0x559dc1c27b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.381+0000 7fec16a00640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c27b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.382+0000 7fec16a00640 1 -- 172.21.15.119:0/3221713738 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dc1c282a0 con 0x559dc1c186d0 2024-03-28T18:47:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.382+0000 7fec0e000640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c29c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.382+0000 7fec17400640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c27b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:17.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.383+0000 7fec16a00640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fec0800ae40 tx=0x7fec08004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec10fd356e7f563f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.383+0000 7fec04c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:17.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec08004b20 con 0x559dc1c186d0 2024-03-28T18:47:17.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec00004510 con 0x559dc1c186d0 2024-03-28T18:47:17.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fec0802f020 con 0x559dc1c186d0 2024-03-28T18:47:17.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec08037050 con 0x559dc1c186d0 2024-03-28T18:47:17.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec20a0f740 10 client.26266 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:17.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.384+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559dc1c8dbf0 con 0x559dc1c186d0 2024-03-28T18:47:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.385+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec00004510 con 0x559dc1c186d0 2024-03-28T18:47:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fec1002c700 con 0x559dc1c186d0 2024-03-28T18:47:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec20a0f740 10 client.26266 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec20a0f740 10 client.26266 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec20a0f740 20 client.26266 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c8e320 0x559dc1c90740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.388+0000 7fec20a0f740 4 client.26266 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.389+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559dc1b83b50 con 0x559dc1c8e320 2024-03-28T18:47:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.393+0000 7fec0e000640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c8e320 0x559dc1c90740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.394+0000 7fec0e000640 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c8e320 0x559dc1c90740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67049a952270e1b server_cookie=3ecbc4a26c86c22f in_seq=0 out_seq=0 2024-03-28T18:47:17.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.394+0000 7fec04c00640 10 client.26266 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:17.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.396+0000 7fec04c00640 1 -- 172.21.15.119:0/3221713738 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559dc1b83b50 con 0x559dc1c8e320 2024-03-28T18:47:17.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.396+0000 7fec04c00640 10 client.26266 handle_command_reply: tid=0 2024-03-28T18:47:17.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.396+0000 7fec20a0f740 1 client.26266 shutdown 2024-03-28T18:47:17.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 msgr2=0x559dc1c296c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fec0800ae40 tx=0x7fec08004ab0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 shutdown_connections 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x559dc1c8e320 0x559dc1c90740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559dc1c21c00 0x559dc1c27b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559dc1c19080 0x559dc1c29c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 --2- 172.21.15.119:0/3221713738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559dc1c186d0 0x559dc1c296c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 >> 172.21.15.119:0/3221713738 conn(0x559dc1c12da0 msgr2=0x559dc1c13ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec04c00640 0 client.26266 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:17.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.397+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 shutdown_connections 2024-03-28T18:47:17.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.398+0000 7fec20a0f740 1 -- 172.21.15.119:0/3221713738 wait complete. 2024-03-28T18:47:17.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.398+0000 7fec20a0f740 20 client.26266 trim_cache size 0 max 16384 2024-03-28T18:47:17.399 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 msgr2=0x7febfc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 0x7febfc0a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fec180738a0 tx=0x7fec08004a40 comp rx=0 tx=0).stop 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 msgr2=0x7fec181ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fec100057c0 tx=0x7fec100349e0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 shutdown_connections 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fec181abe30 0x7fec181ac2e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec181a7570 0x7fec181ab8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 --2- 172.21.15.119:0/3455754162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec1811ae80 0x7fec181a7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.404+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 >> 172.21.15.119:0/3455754162 conn(0x7fec1806c780 msgr2=0x7fec18072170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.405+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 shutdown_connections 2024-03-28T18:47:17.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.405+0000 7fec1de00640 1 -- 172.21.15.119:0/3455754162 wait complete. 2024-03-28T18:47:17.422 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:17.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:47:17.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:17 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:17.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:17 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:17.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:17 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:47:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:17 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:17 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:17 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:47:17.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 -- 172.21.15.119:0/748030520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 msgr2=0x7f38f8110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 --2- 172.21.15.119:0/748030520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f8110710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009cd0 tx=0x7f38f002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 -- 172.21.15.119:0/748030520 shutdown_connections 2024-03-28T18:47:17.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 --2- 172.21.15.119:0/748030520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f8110710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 --2- 172.21.15.119:0/748030520 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8103840 0x7f38f8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 --2- 172.21.15.119:0/748030520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8109850 0x7f38f8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 -- 172.21.15.119:0/748030520 >> 172.21.15.119:0/748030520 conn(0x7f38f80ff310 msgr2=0x7f38f8101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 -- 172.21.15.119:0/748030520 shutdown_connections 2024-03-28T18:47:17.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.704+0000 7f3900200640 1 -- 172.21.15.119:0/748030520 wait complete. 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.705+0000 7f3900200640 1 Processor -- start 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.705+0000 7f3900200640 1 -- start start 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.705+0000 7f3900200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8103840 0x7f38f819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f3900200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f3900200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f3900200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8113c60 con 0x7f38f8104200 2024-03-28T18:47:17.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f3900200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8113ae0 con 0x7f38f8109850 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f3900200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8113de0 con 0x7f38f8103840 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fd800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8103840 0x7f38f819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54366/0 (socket says 172.21.15.119:54366) 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 -- 172.21.15.119:0/688725774 learned_addr learned my addr 172.21.15.119:0/688725774 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 -- 172.21.15.119:0/688725774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8103840 msgr2=0x7f38f819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8103840 0x7f38f819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 -- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 msgr2=0x7f38f819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.706+0000 7f38fe200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38fe200640 1 -- 172.21.15.119:0/688725774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f819f0e0 con 0x7f38f8109850 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38fce00640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38fe200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f38f00348f0 tx=0x7f38f0032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38f6000640 1 -- 172.21.15.119:0/688725774 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0041020 con 0x7f38f8109850 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f819f370 con 0x7f38f8109850 2024-03-28T18:47:17.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38f6000640 1 -- 172.21.15.119:0/688725774 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38f0034cd0 con 0x7f38f8109850 2024-03-28T18:47:17.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f38f6000640 1 -- 172.21.15.119:0/688725774 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0039040 con 0x7f38f8109850 2024-03-28T18:47:17.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.707+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f81a6c60 con 0x7f38f8109850 2024-03-28T18:47:17.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.710+0000 7f3903037740 1 Processor -- start 2024-03-28T18:47:17.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.710+0000 7f38f6000640 1 -- 172.21.15.119:0/688725774 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f38f0044020 con 0x7f38f8109850 2024-03-28T18:47:17.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 -- start start 2024-03-28T18:47:17.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd571ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd57d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd4d52c0 con 0x559bfd5716d0 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd47dc90 con 0x559bfd572080 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f3903037740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd4d1360 con 0x559bfd57ac00 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f38fd800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f38fd800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54378/0 (socket says 172.21.15.119:54378) 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f38fd800640 1 -- 172.21.15.119:0/1153725893 learned_addr learned my addr 172.21.15.119:0/1153725893 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.711+0000 7f38fce00640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd571ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fce00640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd57d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 -- 172.21.15.119:0/1153725893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 msgr2=0x559bfd57d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd57d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 -- 172.21.15.119:0/1153725893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 msgr2=0x559bfd571ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd571ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 -- 172.21.15.119:0/1153725893 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bfd4a2080 con 0x559bfd572080 2024-03-28T18:47:17.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fce00640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd571ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38fd800640 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f38e400b710 tx=0x7f38e4004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=108fefae714269f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38f6000640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 0x7f38e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38eaa00640 1 -- 172.21.15.119:0/1153725893 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4003bd0 con 0x559bfd572080 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38eaa00640 1 -- 172.21.15.119:0/1153725893 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e4015070 con 0x559bfd572080 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38eaa00640 1 -- 172.21.15.119:0/1153725893 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4004e60 con 0x559bfd572080 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.712+0000 7f38f6000640 1 -- 172.21.15.119:0/688725774 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f38f00f0c10 con 0x7f38f8109850 2024-03-28T18:47:17.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f38fce00640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 0x7f38e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 -- 172.21.15.119:0/1153725893 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 msgr2=0x559bfd57a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f38e400b710 tx=0x7f38e4004610 comp rx=0 tx=0).stop 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f38fce00640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 0x7f38e00a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00b620 tx=0x7f38ec014040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 -- 172.21.15.119:0/1153725893 shutdown_connections 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd57d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a630 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 --2- 172.21.15.119:0/1153725893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd571ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 -- 172.21.15.119:0/1153725893 >> 172.21.15.119:0/1153725893 conn(0x559bfd56bda0 msgr2=0x559bfd56c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 -- 172.21.15.119:0/1153725893 shutdown_connections 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.713+0000 7f3903037740 1 -- 172.21.15.119:0/1153725893 wait complete. 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 -- start start 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 Processor -- start 2024-03-28T18:47:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 0x559bfd57a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd57ac00 0x559bfd577f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.714+0000 7f3903037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd57dbc0 con 0x559bfd57ac00 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f3903037740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd4910d0 con 0x559bfd5716d0 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f3903037740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bfd57dd40 con 0x559bfd572080 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f3903037740 10 client.0 _ll_register_callbacks cb 0x559bfd495e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45048/0 (socket says 172.21.15.119:45048) 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 -- 172.21.15.119:0/2753392390 learned_addr learned my addr 172.21.15.119:0/2753392390 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fd800640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 0x559bfd57a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fe200640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd57ac00 0x559bfd577f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 -- 172.21.15.119:0/2753392390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 msgr2=0x559bfd57a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 0x559bfd57a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 -- 172.21.15.119:0/2753392390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd57ac00 msgr2=0x559bfd577f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd57ac00 0x559bfd577f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 -- 172.21.15.119:0/2753392390 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559bfd1e93f0 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fd800640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 0x559bfd57a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38fce00640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011c80 tx=0x7f38ec00bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffe0ea428ee7d8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.715+0000 7f38df400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec018030 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f38ec011cb0 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38e4015070 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38ec021440 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f3903037740 10 client.26278 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559bfd2b37d0 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.716+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38e4015070 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f38ec01e8a0 con 0x559bfd572080 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 10 client.26278 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:17.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 10 client.26278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:17.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 20 client.26278 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:17.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd58ab90 0x559bfd58cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 4 client.26278 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:17.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559bfd4e2800 con 0x559bfd58ab90 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.717+0000 7f38fd800640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd58ab90 0x559bfd58cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.718+0000 7f38fd800640 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd58ab90 0x559bfd58cfd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e644f808dadfca4a server_cookie=adc44f3faf4f2649 in_seq=0 out_seq=0 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.718+0000 7f38df400640 10 client.26278 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.718+0000 7f38df400640 1 -- 172.21.15.119:0/2753392390 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559bfd4e2800 con 0x559bfd58ab90 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.718+0000 7f38df400640 10 client.26278 handle_command_reply: tid=0 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.718+0000 7f3903037740 1 client.26278 shutdown 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 msgr2=0x559bfd577920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011c80 tx=0x7f38ec00bb20 comp rx=0 tx=0).stop 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 shutdown_connections 2024-03-28T18:47:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd58ab90 0x559bfd58cfd0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd57ac00 0x559bfd577f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd572080 0x559bfd577920 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f38df400640 0 client.26278 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 --2- 172.21.15.119:0/2753392390 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd5716d0 0x559bfd57a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 >> 172.21.15.119:0/2753392390 conn(0x559bfd56bda0 msgr2=0x559bfd56daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 shutdown_connections 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.719+0000 7f3903037740 1 -- 172.21.15.119:0/2753392390 wait complete. 2024-03-28T18:47:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.720+0000 7f3903037740 20 client.26278 trim_cache size 0 max 16384 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.742+0000 7f3903037740 1 Processor -- start 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.742+0000 7f3903037740 1 -- start start 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd577200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd574ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81af570 con 0x559bfd5716d0 2024-03-28T18:47:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b0ef0 con 0x559bfd572080 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.743+0000 7f3903037740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81afbd0 con 0x559bfd57ac00 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.744+0000 7f38fd800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd577200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.744+0000 7f38fce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.744+0000 7f38fce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54400/0 (socket says 172.21.15.119:54400) 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.744+0000 7f38fce00640 1 -- 172.21.15.119:0/1998147653 learned_addr learned my addr 172.21.15.119:0/1998147653 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.744+0000 7f38fe200640 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd574ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fce00640 1 -- 172.21.15.119:0/1998147653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 msgr2=0x559bfd574ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fce00640 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd574ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fce00640 1 -- 172.21.15.119:0/1998147653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 msgr2=0x559bfd577200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fce00640 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd577200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fce00640 1 -- 172.21.15.119:0/1998147653 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bfd5709f0 con 0x559bfd572080 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.745+0000 7f38fd800640 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd577200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.746+0000 7f38fce00640 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011820 tx=0x7f38ec00b500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f2acc5a89163dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.747+0000 7f38ea000640 1 -- 172.21.15.119:0/1998147653 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec020b70 con 0x559bfd572080 2024-03-28T18:47:17.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.747+0000 7f38ea000640 1 -- 172.21.15.119:0/1998147653 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ec01a070 con 0x559bfd572080 2024-03-28T18:47:17.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.747+0000 7f38ea000640 1 -- 172.21.15.119:0/1998147653 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec01e8f0 con 0x559bfd572080 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.748+0000 7f3903037740 1 -- 172.21.15.119:0/1998147653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 msgr2=0x559bfd57a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.748+0000 7f3903037740 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011820 tx=0x7f38ec00b500 comp rx=0 tx=0).stop 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 -- 172.21.15.119:0/1998147653 shutdown_connections 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd574ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd57a050 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 --2- 172.21.15.119:0/1998147653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd577200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 -- 172.21.15.119:0/1998147653 >> 172.21.15.119:0/1998147653 conn(0x559bfd56bda0 msgr2=0x559bfd572d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 -- 172.21.15.119:0/1998147653 shutdown_connections 2024-03-28T18:47:17.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.749+0000 7f3903037740 1 -- 172.21.15.119:0/1998147653 wait complete. 2024-03-28T18:47:17.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.750+0000 7f3903037740 1 -- start start 2024-03-28T18:47:17.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.750+0000 7f3903037740 1 Processor -- start 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd581110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd5f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b0890 con 0x559bfd5716d0 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81b0230 con 0x559bfd572080 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f81ad730 con 0x559bfd57ac00 2024-03-28T18:47:17.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.751+0000 7f3903037740 10 client.0 _ll_register_callbacks cb 0x7f38ec0124d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:17.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.752+0000 7f38fd800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.752+0000 7f38fe200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd5f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.752+0000 7f38fd800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55258/0 (socket says 172.21.15.119:55258) 2024-03-28T18:47:17.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.752+0000 7f38fd800640 1 -- 172.21.15.119:0/3446029469 learned_addr learned my addr 172.21.15.119:0/3446029469 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:17.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.752+0000 7f38fce00640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd581110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.754+0000 7f38fce00640 1 -- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 msgr2=0x559bfd5f8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.754+0000 7f38fce00640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd5f8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.754+0000 7f38fce00640 1 -- 172.21.15.119:0/3446029469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 msgr2=0x559bfd580bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:17.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.754+0000 7f38fce00640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.755+0000 7f38fce00640 1 -- 172.21.15.119:0/3446029469 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559bfd5f9250 con 0x559bfd572080 2024-03-28T18:47:17.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.755+0000 7f38fe200640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd5f8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:17.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.755+0000 7f38fd800640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:17.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.758+0000 7f38fce00640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd581110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011820 tx=0x7f38ec01e430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d9d1b5a746ba4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:17.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.758+0000 7f38df400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:17.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.758+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec020b70 con 0x559bfd572080 2024-03-28T18:47:17.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.759+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f38ec021a80 con 0x559bfd572080 2024-03-28T18:47:17.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.759+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38ec01a070 con 0x559bfd572080 2024-03-28T18:47:17.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.760+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38ec021c80 con 0x559bfd572080 2024-03-28T18:47:17.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.760+0000 7f3903037740 10 client.26297 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:17.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.760+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559bfd6005d0 con 0x559bfd572080 2024-03-28T18:47:17.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.762+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ec01a070 con 0x559bfd572080 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.762+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f38ec038020 con 0x559bfd572080 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 10 client.26297 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 10 client.26297 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 20 client.26297 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd600c00 0x559bfd603020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 4 client.26297 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:17.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.763+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559bfd4dcb50 con 0x559bfd600c00 2024-03-28T18:47:17.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.764+0000 7f38fd800640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd600c00 0x559bfd603020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:17.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.767+0000 7f38fd800640 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd600c00 0x559bfd603020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b34d5043e46ec4c4 server_cookie=ad359ef581da5253 in_seq=0 out_seq=0 2024-03-28T18:47:17.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.767+0000 7f38df400640 10 client.26297 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:17.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.768+0000 7f38df400640 1 -- 172.21.15.119:0/3446029469 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559bfd4dcb50 con 0x559bfd600c00 2024-03-28T18:47:17.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.768+0000 7f38df400640 10 client.26297 handle_command_reply: tid=0 2024-03-28T18:47:17.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.768+0000 7f3903037740 1 client.26297 shutdown 2024-03-28T18:47:17.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 msgr2=0x559bfd581110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd581110 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f38ec011820 tx=0x7f38ec01e430 comp rx=0 tx=0).stop 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 shutdown_connections 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559bfd600c00 0x559bfd603020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559bfd57ac00 0x559bfd5f8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559bfd572080 0x559bfd581110 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 --2- 172.21.15.119:0/3446029469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559bfd5716d0 0x559bfd580bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 >> 172.21.15.119:0/3446029469 conn(0x559bfd56bda0 msgr2=0x559bfd56ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.769+0000 7f38df400640 0 client.26297 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.771+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 shutdown_connections 2024-03-28T18:47:17.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.771+0000 7f3903037740 1 -- 172.21.15.119:0/3446029469 wait complete. 2024-03-28T18:47:17.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.771+0000 7f3903037740 20 client.26297 trim_cache size 0 max 16384 2024-03-28T18:47:17.772 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:17.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.773+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 msgr2=0x7f38e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.773+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 0x7f38e00a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00b620 tx=0x7f38ec014040 comp rx=0 tx=0).stop 2024-03-28T18:47:17.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.774+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 msgr2=0x7f38f819e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:17.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.774+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f38f00348f0 tx=0x7f38f0032ec0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 shutdown_connections 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f38e00a2970 0x7f38e00a4e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38f8109850 0x7f38f819e9b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38f8104200 0x7f38f819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 --2- 172.21.15.119:0/688725774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38f8103840 0x7f38f819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:17.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.776+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 >> 172.21.15.119:0/688725774 conn(0x7f38f80ff310 msgr2=0x7f38f80ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:17.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.780+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 shutdown_connections 2024-03-28T18:47:17.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:17.785+0000 7f3900200640 1 -- 172.21.15.119:0/688725774 wait complete. 2024-03-28T18:47:17.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:17 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:47:17.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:17 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:47:17.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:17 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:47:17.828 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:17.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 -- 172.21.15.119:0/734580100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0101240 msgr2=0x7f6fb010d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 --2- 172.21.15.119:0/734580100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0101240 0x7f6fb010d3b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009cd0 tx=0x7f6fa802e390 comp rx=0 tx=0).stop 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 -- 172.21.15.119:0/734580100 shutdown_connections 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 --2- 172.21.15.119:0/734580100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0101240 0x7f6fb010d3b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 --2- 172.21.15.119:0/734580100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0100880 0x7f6fb0100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 --2- 172.21.15.119:0/734580100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0106750 0x7f6fb0106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 -- 172.21.15.119:0/734580100 >> 172.21.15.119:0/734580100 conn(0x7f6fb0077580 msgr2=0x7f6fb0077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 -- 172.21.15.119:0/734580100 shutdown_connections 2024-03-28T18:47:18.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.127+0000 7f6fb6400640 1 -- 172.21.15.119:0/734580100 wait complete. 2024-03-28T18:47:18.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.128+0000 7f6fb6400640 1 Processor -- start 2024-03-28T18:47:18.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.128+0000 7f6fb6400640 1 -- start start 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0101240 0x7f6fb019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0106750 0x7f6fb019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0113d10 con 0x7f6fb0106750 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0113b90 con 0x7f6fb0101240 2024-03-28T18:47:18.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fb6400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0113e90 con 0x7f6fb0100880 2024-03-28T18:47:18.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6faf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6faf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45082/0 (socket says 172.21.15.119:45082) 2024-03-28T18:47:18.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6faf400640 1 -- 172.21.15.119:0/1202773857 learned_addr learned my addr 172.21.15.119:0/1202773857 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:18.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0101240 0x7f6fb019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0106750 0x7f6fb019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.129+0000 7f6faf400640 1 -- 172.21.15.119:0/1202773857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0101240 msgr2=0x7f6fb019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6faf400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0101240 0x7f6fb019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6faf400640 1 -- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0106750 msgr2=0x7f6fb019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6faf400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0106750 0x7f6fb019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6faf400640 1 -- 172.21.15.119:0/1202773857 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb019f120 con 0x7f6fb0100880 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6faf400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800e9f0 tx=0x7f6f9800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6facc00640 1 -- 172.21.15.119:0/1202773857 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9800cd80 con 0x7f6fb0100880 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6facc00640 1 -- 172.21.15.119:0/1202773857 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f98010910 con 0x7f6fb0100880 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.130+0000 7f6facc00640 1 -- 172.21.15.119:0/1202773857 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f980188f0 con 0x7f6fb0100880 2024-03-28T18:47:18.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.131+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb019f410 con 0x7f6fb0100880 2024-03-28T18:47:18.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.131+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb01a6ca0 con 0x7f6fb0100880 2024-03-28T18:47:18.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.135+0000 7f6fb91d1740 1 Processor -- start 2024-03-28T18:47:18.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.135+0000 7f6fb91d1740 1 -- start start 2024-03-28T18:47:18.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 0x561d27b64910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b7a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b7b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27ac82c0 con 0x561d27b7ac40 2024-03-28T18:47:18.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27a70c90 con 0x561d27b64530 2024-03-28T18:47:18.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.136+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27ac4360 con 0x561d27b64ee0 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 0x561d27b64910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b7b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 0x561d27b64910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:54426/0 (socket says 172.21.15.119:54426) 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fafe00640 1 -- 172.21.15.119:0/311359700 learned_addr learned my addr 172.21.15.119:0/311359700 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6facc00640 1 -- 172.21.15.119:0/1202773857 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6f98010ac0 con 0x7f6fb0100880 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 -- 172.21.15.119:0/311359700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 msgr2=0x561d27b7a620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b7a620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 -- 172.21.15.119:0/311359700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 msgr2=0x561d27b64910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 0x561d27b64910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.137+0000 7f6fa7e00640 1 -- 172.21.15.119:0/311359700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d27a95080 con 0x561d27b7ac40 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.138+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b7b0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000bc10 tx=0x7f6fa00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe8059f33f510a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.138+0000 7f6fa4c00640 1 -- 172.21.15.119:0/311359700 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0004030 con 0x561d27b7ac40 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.139+0000 7f6fa4c00640 1 -- 172.21.15.119:0/311359700 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fa00041d0 con 0x561d27b7ac40 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.139+0000 7f6fa4c00640 1 -- 172.21.15.119:0/311359700 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0011750 con 0x561d27b7ac40 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.139+0000 7f6facc00640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 0x7f6f940a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.139+0000 7f6facc00640 1 -- 172.21.15.119:0/1202773857 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6f980cf340 con 0x7f6fb0100880 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.140+0000 7f6fb91d1740 1 -- 172.21.15.119:0/311359700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 msgr2=0x561d27b7b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.140+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b7b0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000bc10 tx=0x7f6fa00107a0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.140+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 0x7f6f940a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 -- 172.21.15.119:0/311359700 shutdown_connections 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b7b0c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b7a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/311359700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64530 0x561d27b64910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 -- 172.21.15.119:0/311359700 >> 172.21.15.119:0/311359700 conn(0x561d27b5eda0 msgr2=0x561d27b5f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 -- 172.21.15.119:0/311359700 shutdown_connections 2024-03-28T18:47:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 -- 172.21.15.119:0/311359700 wait complete. 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fb91d1740 1 -- start start 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.141+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 0x7f6f940a4eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800e550 tx=0x7f6fa8034000 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 Processor -- start 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b810d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b7e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27b81bc0 con 0x561d27b64530 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27a840d0 con 0x561d27b64ee0 2024-03-28T18:47:18.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27b81d40 con 0x561d27b7ac40 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.142+0000 7f6fb91d1740 10 client.0 _ll_register_callbacks cb 0x561d27a88e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45088/0 (socket says 172.21.15.119:45088) 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 -- 172.21.15.119:0/3991258088 learned_addr learned my addr 172.21.15.119:0/3991258088 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b7e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 -- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 msgr2=0x561d27b7e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b7e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 -- 172.21.15.119:0/3991258088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 msgr2=0x561d27b810d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b810d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fafe00640 1 -- 172.21.15.119:0/3991258088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d277dc3f0 con 0x561d27b7ac40 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.143+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b7e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80356b0 tx=0x7f6fa80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75abb1dcdfce7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6f9d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8058060 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6fa8004170 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fa00041d0 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fa8004370 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6fb91d1740 10 client.26293 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.144+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d278a67d0 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fa8033510 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6fa8052030 con 0x561d27b7ac40 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 10 client.26293 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 20 client.26293 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b97240 0x561d27b99680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 4 client.26293 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:18.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fa00041d0 con 0x561d27b97240 2024-03-28T18:47:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.145+0000 7f6faf400640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b97240 0x561d27b99680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.146+0000 7f6f9d600640 10 client.26293 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.146+0000 7f6faf400640 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b97240 0x561d27b99680 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=167538df79fd27cb server_cookie=bcf4b687d829379 in_seq=0 out_seq=0 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.146+0000 7f6f9d600640 1 -- 172.21.15.119:0/3991258088 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f6fa00041d0 con 0x561d27b97240 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6f9d600640 10 client.26293 handle_command_reply: tid=0 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 client.26293 shutdown 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 msgr2=0x561d27b7ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80356b0 tx=0x7f6fa80095c0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 shutdown_connections 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b97240 0x561d27b99680 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b7ac40 0x561d27b7ea70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b7e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3991258088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b810d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 >> 172.21.15.119:0/3991258088 conn(0x561d27b5eda0 msgr2=0x561d27b60af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.147+0000 7f6f9d600640 0 client.26293 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.148+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 shutdown_connections 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.148+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3991258088 wait complete. 2024-03-28T18:47:18.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.148+0000 7f6fb91d1740 20 client.26293 trim_cache size 0 max 16384 2024-03-28T18:47:18.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.173+0000 7f6fb91d1740 1 Processor -- start 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 -- start start 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b7c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b80b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 0x561d27b810d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006f370 con 0x561d27b64530 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0070cf0 con 0x561d27b7ac40 2024-03-28T18:47:18.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.174+0000 7f6fb91d1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006f9d0 con 0x561d27b64ee0 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.175+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b80b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.176+0000 7f6fa7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b80b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:45102/0 (socket says 172.21.15.119:45102) 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.176+0000 7f6fa7e00640 1 -- 172.21.15.119:0/3225369480 learned_addr learned my addr 172.21.15.119:0/3225369480 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.176+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 0x561d27b810d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.176+0000 7f6faf400640 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b7c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6fafe00640 1 -- 172.21.15.119:0/3225369480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 msgr2=0x561d27b80b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b80b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6fafe00640 1 -- 172.21.15.119:0/3225369480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 msgr2=0x561d27b7c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b7c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6fafe00640 1 -- 172.21.15.119:0/3225369480 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d27b63760 con 0x561d27b7ac40 2024-03-28T18:47:18.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.177+0000 7f6faf400640 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:18.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.178+0000 7f6fafe00640 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 0x561d27b810d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009bf0 tx=0x7f6fa80357a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142dc3dac07404a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.179+0000 7f6f9fe00640 1 -- 172.21.15.119:0/3225369480 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8058040 con 0x561d27b7ac40 2024-03-28T18:47:18.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.179+0000 7f6f9fe00640 1 -- 172.21.15.119:0/3225369480 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fa8003f70 con 0x561d27b7ac40 2024-03-28T18:47:18.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.179+0000 7f6f9fe00640 1 -- 172.21.15.119:0/3225369480 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8033800 con 0x561d27b7ac40 2024-03-28T18:47:18.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.180+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3225369480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 msgr2=0x561d27b810d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.180+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 0x561d27b810d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009bf0 tx=0x7f6fa80357a0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3225369480 shutdown_connections 2024-03-28T18:47:18.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b7ac40 0x561d27b810d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64ee0 0x561d27b80b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/3225369480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b64530 0x561d27b7c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3225369480 >> 172.21.15.119:0/3225369480 conn(0x561d27b5eda0 msgr2=0x561d27b5d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3225369480 shutdown_connections 2024-03-28T18:47:18.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.181+0000 7f6fb91d1740 1 -- 172.21.15.119:0/3225369480 wait complete. 2024-03-28T18:47:18.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.182+0000 7f6fb91d1740 1 -- start start 2024-03-28T18:47:18.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.182+0000 7f6fb91d1740 1 Processor -- start 2024-03-28T18:47:18.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.183+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64530 0x561d27b81f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.183+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b824d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.183+0000 7f6fb91d1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.184+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.184+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55304/0 (socket says 172.21.15.119:55304) 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.184+0000 7f6fafe00640 1 -- 172.21.15.119:0/1336198974 learned_addr learned my addr 172.21.15.119:0/1336198974 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.184+0000 7f6faf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64530 0x561d27b81f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.185+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0070690 con 0x561d27b7ac40 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.185+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0070030 con 0x561d27b64ee0 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.185+0000 7f6fa7e00640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b824d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.185+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb006d550 con 0x561d27b64530 2024-03-28T18:47:18.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.186+0000 7f6fafe00640 1 -- 172.21.15.119:0/1336198974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64530 msgr2=0x561d27b81f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.186+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64530 0x561d27b81f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.186+0000 7f6fafe00640 1 -- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 msgr2=0x561d27b824d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.186+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b824d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.186+0000 7f6fafe00640 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d27b8d290 con 0x561d27b7ac40 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.187+0000 7f6fb91d1740 10 client.0 _ll_register_callbacks cb 0x7f6fa8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.187+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fa8003f70 con 0x561d27b7ac40 2024-03-28T18:47:18.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.187+0000 7f6fafe00640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8035650 tx=0x7f6fa8033ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1a05f5dc92f789 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:18.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.187+0000 7f6f9d600640 10 client.16095 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:18.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.188+0000 7f6f9d600640 1 -- 172.21.15.119:0/1336198974 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8058040 con 0x561d27b7ac40 2024-03-28T18:47:18.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.188+0000 7f6f9d600640 1 -- 172.21.15.119:0/1336198974 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6fa8033710 con 0x561d27b7ac40 2024-03-28T18:47:18.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6f9d600640 1 -- 172.21.15.119:0/1336198974 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fa8033e40 con 0x561d27b7ac40 2024-03-28T18:47:18.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 10 client.16095 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:18.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=28}) v3 -- 0x561d27b8e270 con 0x561d27b7ac40 2024-03-28T18:47:18.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6f9d600640 1 -- 172.21.15.119:0/1336198974 <== mon.0 v2:172.21.15.119:3300/0 4 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6fa8062020 con 0x561d27b7ac40 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 10 client.16095 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 20 client.16095 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.189+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b8e840 0x561d27b8ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.190+0000 7f6fb91d1740 4 client.16095 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:18.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.190+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d27ad5800 con 0x561d27b8e840 2024-03-28T18:47:18.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.191+0000 7f6faf400640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b8e840 0x561d27b8ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:18.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.193+0000 7f6f9d600640 10 client.16095 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:18.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.193+0000 7f6faf400640 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b8e840 0x561d27b8ec90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82e074ccb0d4d5f server_cookie=da5ce2d926f24fd9 in_seq=0 out_seq=0 2024-03-28T18:47:18.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.194+0000 7f6f9d600640 1 -- 172.21.15.119:0/1336198974 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d27ad5800 con 0x561d27b8e840 2024-03-28T18:47:18.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.194+0000 7f6f9d600640 10 client.16095 handle_command_reply: tid=0 2024-03-28T18:47:18.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.194+0000 7f6fb91d1740 1 client.16095 shutdown 2024-03-28T18:47:18.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.194+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 msgr2=0x561d27b82a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.194+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8035650 tx=0x7f6fa8033ac0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.195+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 shutdown_connections 2024-03-28T18:47:18.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.195+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d27b8e840 0x561d27b8ec90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.195+0000 7f6f9d600640 0 client.16095 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.195+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d27b7ac40 0x561d27b82a10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.195+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d27b64ee0 0x561d27b824d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.196+0000 7f6fb91d1740 1 --2- 172.21.15.119:0/1336198974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d27b64530 0x561d27b81f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.196+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 >> 172.21.15.119:0/1336198974 conn(0x561d27b5eda0 msgr2=0x561d27b5deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.196+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 shutdown_connections 2024-03-28T18:47:18.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.196+0000 7f6fb91d1740 1 -- 172.21.15.119:0/1336198974 wait complete. 2024-03-28T18:47:18.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.196+0000 7f6fb91d1740 20 client.16095 trim_cache size 0 max 16384 2024-03-28T18:47:18.197 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:18.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.197+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 msgr2=0x7f6f940a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.197+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 0x7f6f940a4eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800e550 tx=0x7f6fa8034000 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.197+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 msgr2=0x7f6fb019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.197+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800e9f0 tx=0x7f6f9800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 shutdown_connections 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6f940a29f0 0x7f6f940a4eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb0106750 0x7f6fb019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6fb0101240 0x7f6fb019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 --2- 172.21.15.119:0/1202773857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fb0100880 0x7f6fb019a130 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:18.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.198+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 >> 172.21.15.119:0/1202773857 conn(0x7f6fb0077580 msgr2=0x7f6fb00ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:18.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.199+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 shutdown_connections 2024-03-28T18:47:18.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:18.199+0000 7f6fb6400640 1 -- 172.21.15.119:0/1202773857 wait complete. 2024-03-28T18:47:18.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:18 smithi119 ceph-mon[31813]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 345 MiB/s wr, 96 op/s 2024-03-28T18:47:18.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:18 smithi196 ceph-mon[36300]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 345 MiB/s wr, 96 op/s 2024-03-28T18:47:18.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:18 smithi131 ceph-mon[36065]: pgmap v100: 97 pgs: 97 active+clean; 12 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 345 MiB/s wr, 96 op/s 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 rewriters = 390217.97 kB/sec 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 rewriters = 390202.79 kB/sec 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 390217.97 kB/sec 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 390217.97 kB/sec 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 390217.97 kB/sec 2024-03-28T18:47:19.239 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 10485760.00 kB 2024-03-28T18:47:19.243 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:47:19.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:47:19.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:47:19.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 -- 172.21.15.119:0/397757291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0109850 msgr2=0x7fabd0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 --2- 172.21.15.119:0/397757291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0109850 0x7fabd0109c30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009860 tx=0x7fabc002bb70 comp rx=0 tx=0).stop 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 -- 172.21.15.119:0/397757291 shutdown_connections 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 --2- 172.21.15.119:0/397757291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 0x7fabd0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 --2- 172.21.15.119:0/397757291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0103840 0x7fabd0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 --2- 172.21.15.119:0/397757291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0109850 0x7fabd0109c30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 -- 172.21.15.119:0/397757291 >> 172.21.15.119:0/397757291 conn(0x7fabd00ff2f0 msgr2=0x7fabd0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 -- 172.21.15.119:0/397757291 shutdown_connections 2024-03-28T18:47:19.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.425+0000 7fabd5e00640 1 -- 172.21.15.119:0/397757291 wait complete. 2024-03-28T18:47:19.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.426+0000 7fabd5e00640 1 Processor -- start 2024-03-28T18:47:19.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.426+0000 7fabd5e00640 1 -- start start 2024-03-28T18:47:19.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 0x7fabd019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 0x7fabd019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0113c20 con 0x7fabd0109850 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0113aa0 con 0x7fabd0103840 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabd5e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0113da0 con 0x7fabd0104200 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49168/0 (socket says 172.21.15.119:49168) 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 -- 172.21.15.119:0/3098827666 learned_addr learned my addr 172.21.15.119:0/3098827666 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabc6000640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 0x7fabd019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcf400640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 0x7fabd019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 -- 172.21.15.119:0/3098827666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 msgr2=0x7fabd019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 0x7fabd019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 -- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 msgr2=0x7fabd019ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 0x7fabd019ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcea00640 1 -- 172.21.15.119:0/3098827666 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd019f140 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.427+0000 7fabcf400640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 0x7fabd019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabcea00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fabc0037040 tx=0x7fabc00045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabd4a00640 1 -- 172.21.15.119:0/3098827666 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0041020 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabd4a00640 1 -- 172.21.15.119:0/3098827666 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc002c9f0 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd019f3d0 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabd4a00640 1 -- 172.21.15.119:0/3098827666 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0039040 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.428+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd01a6cc0 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd4a00640 1 -- 172.21.15.119:0/3098827666 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fabc0044020 con 0x7fabd0103840 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 Processor -- start 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 -- start start 2024-03-28T18:47:19.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985fe6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985f41eb0 con 0x559985fdd9a0 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985eeaa60 con 0x559985fdd080 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.431+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985f3df50 con 0x559985fe6580 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabc6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49178/0 (socket says 172.21.15.119:49178) 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 -- 172.21.15.119:0/3869644131 learned_addr learned my addr 172.21.15.119:0/3869644131 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985fe6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 -- 172.21.15.119:0/3869644131 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 msgr2=0x559985fe8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 -- 172.21.15.119:0/3869644131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 msgr2=0x559985fe6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985fe6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 -- 172.21.15.119:0/3869644131 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559985d1e7e0 con 0x559985fdd080 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabcf400640 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005f40 tx=0x7fabc8010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72110545ae9fd3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabbfe00640 1 -- 172.21.15.119:0/3869644131 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc8011700 con 0x559985fdd080 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabbfe00640 1 -- 172.21.15.119:0/3869644131 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc8011d80 con 0x559985fdd080 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.432+0000 7fabbfe00640 1 -- 172.21.15.119:0/3869644131 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc800e3c0 con 0x559985fdd080 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd4a00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 0x7fabb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabcea00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 0x7fabb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd4a00640 1 -- 172.21.15.119:0/3098827666 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fabc00f0680 con 0x7fabd0103840 2024-03-28T18:47:19.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3869644131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 msgr2=0x559985fdd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fabc8005f40 tx=0x7fabc8010860 comp rx=0 tx=0).stop 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabcea00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 0x7fabb40a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x559985fdcb50 tx=0x7fabc00f6530 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3869644131 shutdown_connections 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985fe6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3869644131 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fdd460 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3869644131 >> 172.21.15.119:0/3869644131 conn(0x559985fd77b0 msgr2=0x559985fd7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.433+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3869644131 shutdown_connections 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3869644131 wait complete. 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 -- start start 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 Processor -- start 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 0x559985fe5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 0x559985fe3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.434+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985fe9370 con 0x559985fdd080 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985efdeb0 con 0x559985fdd9a0 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559985fe94f0 con 0x559985fe6580 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabd8bb2740 10 client.0 _ll_register_callbacks cb 0x559985ef2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabcf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 0x559985fe5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabcf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 0x559985fe5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56312/0 (socket says 172.21.15.119:56312) 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabcf400640 1 -- 172.21.15.119:0/3926289762 learned_addr learned my addr 172.21.15.119:0/3926289762 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 0x559985fe3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabcf400640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 -- 172.21.15.119:0/3926289762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 msgr2=0x559985fe3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 -- 172.21.15.119:0/3926289762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 msgr2=0x559985fe5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 0x559985fe5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 -- 172.21.15.119:0/3926289762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559985d7ef70 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabcf400640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabc6000640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 0x559985fe3290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fabd0108ee0 tx=0x7fabb8011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4442c08563d1020f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.435+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb801e020 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabc8011d80 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fabb800fbc0 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb8026050 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 10 client.26324 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559985c16170 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc8011d80 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fabb8019070 con 0x559985fdd9a0 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 10 client.26324 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:19.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 20 client.26324 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559985ff6640 0x559985ff8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 4 client.26324 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.436+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559985f4f3f0 con 0x559985ff6640 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.437+0000 7fabcf400640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559985ff6640 0x559985ff8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.437+0000 7fabcf400640 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559985ff6640 0x559985ff8a60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0dccc027c3bb75d server_cookie=3364e6db7d4f919 in_seq=0 out_seq=0 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.437+0000 7fabbcc00640 10 client.26324 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabbcc00640 1 -- 172.21.15.119:0/3926289762 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559985f4f3f0 con 0x559985ff6640 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabbcc00640 10 client.26324 handle_command_reply: tid=0 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 client.26324 shutdown 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 msgr2=0x559985fe3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 0x559985fe3290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fabd0108ee0 tx=0x7fabb8011b50 comp rx=0 tx=0).stop 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 shutdown_connections 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559985ff6640 0x559985ff8a60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fe3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd9a0 0x559985fe3290 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabbcc00640 0 client.26324 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/3926289762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd080 0x559985fe5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 >> 172.21.15.119:0/3926289762 conn(0x559985fd77b0 msgr2=0x559985fd9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.438+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 shutdown_connections 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.439+0000 7fabd8bb2740 1 -- 172.21.15.119:0/3926289762 wait complete. 2024-03-28T18:47:19.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.439+0000 7fabd8bb2740 20 client.26324 trim_cache size 0 max 16384 2024-03-28T18:47:19.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.455+0000 7fabd8bb2740 1 Processor -- start 2024-03-28T18:47:19.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.455+0000 7fabd8bb2740 1 -- start start 2024-03-28T18:47:19.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.455+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fe59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.455+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fe6580 0x559985ff1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01af5d0 con 0x559985fe6580 2024-03-28T18:47:19.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01b0f50 con 0x559985fdd080 2024-03-28T18:47:19.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01afc30 con 0x559985fdd9a0 2024-03-28T18:47:19.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabc6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabc6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:55006/0 (socket says 172.21.15.119:55006) 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fe59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabc6000640 1 -- 172.21.15.119:0/243607102 learned_addr learned my addr 172.21.15.119:0/243607102 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fe6580 0x559985ff1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 -- 172.21.15.119:0/243607102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 msgr2=0x559985fe5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 -- 172.21.15.119:0/243607102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fe6580 msgr2=0x559985ff1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fe6580 0x559985ff1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabcf400640 1 -- 172.21.15.119:0/243607102 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559985fd2c20 con 0x559985fdd080 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.456+0000 7fabc6000640 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabcf400640 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fe59c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fabc8011fd0 tx=0x7fabc8013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a9ed09096a3f0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabbf400640 1 -- 172.21.15.119:0/243607102 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc801b4f0 con 0x559985fdd080 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabbf400640 1 -- 172.21.15.119:0/243607102 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc8024040 con 0x559985fdd080 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabbf400640 1 -- 172.21.15.119:0/243607102 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc802c9b0 con 0x559985fdd080 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabd8bb2740 1 -- 172.21.15.119:0/243607102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 msgr2=0x559985fe59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.457+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fe59c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fabc8011fd0 tx=0x7fabc8013040 comp rx=0 tx=0).stop 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 -- 172.21.15.119:0/243607102 shutdown_connections 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fe6580 0x559985ff1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fdd9a0 0x559985fe5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/243607102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985fe59c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 -- 172.21.15.119:0/243607102 >> 172.21.15.119:0/243607102 conn(0x559985fd77b0 msgr2=0x559985fd9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 -- 172.21.15.119:0/243607102 shutdown_connections 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 -- 172.21.15.119:0/243607102 wait complete. 2024-03-28T18:47:19.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 -- start start 2024-03-28T18:47:19.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.458+0000 7fabd8bb2740 1 Processor -- start 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985febc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fec190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01b08f0 con 0x559985fdd9a0 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01b0290 con 0x559985fdd080 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01ad790 con 0x559985fe6580 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabd8bb2740 10 client.0 _ll_register_callbacks cb 0x7fabb8009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49202/0 (socket says 172.21.15.119:49202) 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 -- 172.21.15.119:0/2710288424 learned_addr learned my addr 172.21.15.119:0/2710288424 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabc6000640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985febc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fec190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 -- 172.21.15.119:0/2710288424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 msgr2=0x559985fec190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fec190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 -- 172.21.15.119:0/2710288424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 msgr2=0x559985febc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985febc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 -- 172.21.15.119:0/2710288424 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559985fe12b0 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabcf400640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fabc801bc10 tx=0x7fabc802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92a8bd82bdc332d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.459+0000 7fabbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc802c750 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabc8024040 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fabc802c8f0 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabc802cb30 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 10 client.26330 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559986052ff0 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabc8024040 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fabc8029040 con 0x559985fdd080 2024-03-28T18:47:19.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 10 client.26330 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 20 client.26330 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559986053460 0x559986055850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 4 client.26330 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.460+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559985f49740 con 0x559986053460 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.461+0000 7fabc6000640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559986053460 0x559986055850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.461+0000 7fabc6000640 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559986053460 0x559986055850 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ba12ade95c2f1a server_cookie=8c9ba0ac4837133 in_seq=0 out_seq=0 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.461+0000 7fabbcc00640 10 client.26330 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.462+0000 7fabbcc00640 1 -- 172.21.15.119:0/2710288424 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559985f49740 con 0x559986053460 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.462+0000 7fabbcc00640 10 client.26330 handle_command_reply: tid=0 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.462+0000 7fabd8bb2740 1 client.26330 shutdown 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.462+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 msgr2=0x559985feb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.462+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fabc801bc10 tx=0x7fabc802e040 comp rx=0 tx=0).stop 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 shutdown_connections 2024-03-28T18:47:19.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559986053460 0x559986055850 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559985fe6580 0x559985fec190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559985fdd9a0 0x559985febc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 --2- 172.21.15.119:0/2710288424 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559985fdd080 0x559985feb710 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabbcc00640 0 client.26330 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 >> 172.21.15.119:0/2710288424 conn(0x559985fd77b0 msgr2=0x559985fd5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 shutdown_connections 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 1 -- 172.21.15.119:0/2710288424 wait complete. 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd8bb2740 20 client.26330 trim_cache size 0 max 16384 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:19.466 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:47:19.467 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "09ad7f11-4a97-44fb-a6bb-97b6b4eb8d5e", 2024-03-28T18:47:19.467 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:47:19.467 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:19.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 msgr2=0x7fabb40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 0x7fabb40a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x559985fdcb50 tx=0x7fabc00f6530 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 msgr2=0x7fabd019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fabc0037040 tx=0x7fabc00045b0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 shutdown_connections 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabb40a2970 0x7fabb40a4e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd0109850 0x7fabd019ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabd0104200 0x7fabd019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 --2- 172.21.15.119:0/3098827666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabd0103840 0x7fabd019a1a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.463+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 >> 172.21.15.119:0/3098827666 conn(0x7fabd00ff2f0 msgr2=0x7fabd00ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.464+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 shutdown_connections 2024-03-28T18:47:19.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.464+0000 7fabd5e00640 1 -- 172.21.15.119:0/3098827666 wait complete. 2024-03-28T18:47:19.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:47:19.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 -- 172.21.15.119:0/2293064233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99200764f0 msgr2=0x7f9920111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 --2- 172.21.15.119:0/2293064233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99200764f0 0x7f9920111fa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9910009930 tx=0x7f991002bc20 comp rx=0 tx=0).stop 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 -- 172.21.15.119:0/2293064233 shutdown_connections 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 --2- 172.21.15.119:0/2293064233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99201124e0 0x7f99201148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 --2- 172.21.15.119:0/2293064233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99200764f0 0x7f9920111fa0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 --2- 172.21.15.119:0/2293064233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f9920075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.683+0000 7f9927600640 1 -- 172.21.15.119:0/2293064233 >> 172.21.15.119:0/2293064233 conn(0x7f99200ff2f0 msgr2=0x7f9920101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.684+0000 7f9927600640 1 -- 172.21.15.119:0/2293064233 shutdown_connections 2024-03-28T18:47:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.684+0000 7f9927600640 1 -- 172.21.15.119:0/2293064233 wait complete. 2024-03-28T18:47:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.684+0000 7f9927600640 1 Processor -- start 2024-03-28T18:47:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 -- start start 2024-03-28T18:47:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200764f0 0x7f992019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 0x7f99201a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920118090 con 0x7f99200764f0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920117f10 con 0x7f9920075bd0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9927600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920118210 con 0x7f99201124e0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49222/0 (socket says 172.21.15.119:49222) 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9924c00640 1 -- 172.21.15.119:0/120774087 learned_addr learned my addr 172.21.15.119:0/120774087 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f9925600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 0x7f99201a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.685+0000 7f991fe00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200764f0 0x7f992019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 -- 172.21.15.119:0/120774087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 msgr2=0x7f99201a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 0x7f99201a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 -- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200764f0 msgr2=0x7f992019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200764f0 0x7f992019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 -- 172.21.15.119:0/120774087 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99201a35c0 con 0x7f9920075bd0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9925600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 0x7f99201a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9924c00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f991400e9f0 tx=0x7f991400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f991d600640 1 -- 172.21.15.119:0/120774087 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991400cd80 con 0x7f9920075bd0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99201a38b0 con 0x7f9920075bd0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f991d600640 1 -- 172.21.15.119:0/120774087 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9914005500 con 0x7f9920075bd0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f991d600640 1 -- 172.21.15.119:0/120774087 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914010690 con 0x7f9920075bd0 2024-03-28T18:47:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.686+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99201ab140 con 0x7f9920075bd0 2024-03-28T18:47:19.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 Processor -- start 2024-03-28T18:47:19.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 -- start start 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c579683a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 0x555c57973680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c578cc250 con 0x555c57968c00 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c57874a30 con 0x555c57971290 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.690+0000 7f992a366740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c578c82f0 con 0x555c57967f20 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56348/0 (socket says 172.21.15.119:56348) 2024-03-28T18:47:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 -- 172.21.15.119:0/971403297 learned_addr learned my addr 172.21.15.119:0/971403297 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f991fe00640 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 0x555c57973680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f991d600640 1 -- 172.21.15.119:0/120774087 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f99140046c0 con 0x7f9920075bd0 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 -- 172.21.15.119:0/971403297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 msgr2=0x555c579683a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c579683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 -- 172.21.15.119:0/971403297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 msgr2=0x555c57973680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 0x555c57973680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 -- 172.21.15.119:0/971403297 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c5785f690 con 0x555c57968c00 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f991fe00640 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 0x555c57973680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.691+0000 7f9925600640 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9918011760 tx=0x7f9918011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bd46dcc3f9fb3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f990dc00640 1 -- 172.21.15.119:0/971403297 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918004d20 con 0x555c57968c00 2024-03-28T18:47:19.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f990dc00640 1 -- 172.21.15.119:0/971403297 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f991800f9a0 con 0x555c57968c00 2024-03-28T18:47:19.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f990dc00640 1 -- 172.21.15.119:0/971403297 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918004470 con 0x555c57968c00 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 -- 172.21.15.119:0/971403297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 msgr2=0x555c57967950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9918011760 tx=0x7f9918011c20 comp rx=0 tx=0).stop 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 -- 172.21.15.119:0/971403297 shutdown_connections 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57971290 0x555c57973680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c579683a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 --2- 172.21.15.119:0/971403297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c57967950 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 -- 172.21.15.119:0/971403297 >> 172.21.15.119:0/971403297 conn(0x555c5792f200 msgr2=0x555c57962970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 -- 172.21.15.119:0/971403297 shutdown_connections 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f991d600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 0x7f99080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.692+0000 7f992a366740 1 -- 172.21.15.119:0/971403297 wait complete. 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f991d600640 1 -- 172.21.15.119:0/120774087 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f99140cf310 con 0x7f9920075bd0 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f991fe00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 0x7f99080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 -- start start 2024-03-28T18:47:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 Processor -- start 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f991fe00640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 0x7f99080a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9910009900 tx=0x7f9910005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57968c00 0x555c5796dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c5796e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.693+0000 7f992a366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c57974100 con 0x555c57971290 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f992a366740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c57887e80 con 0x555c57967f20 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f992a366740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c57974280 con 0x555c57968c00 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f992a366740 10 client.0 _ll_register_callbacks cb 0x555c5789a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f991fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57968c00 0x555c5796dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49234/0 (socket says 172.21.15.119:49234) 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 -- 172.21.15.119:0/929808759 learned_addr learned my addr 172.21.15.119:0/929808759 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9925600640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c5796e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 -- 172.21.15.119:0/929808759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57968c00 msgr2=0x555c5796dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57968c00 0x555c5796dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 -- 172.21.15.119:0/929808759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 msgr2=0x555c5796e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c5796e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 -- 172.21.15.119:0/929808759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c576aa7d0 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9924c00640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f991400e9c0 tx=0x7f991409b210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b3efff83d8464 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9902000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140cce20 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9914086070 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.694+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f991800f9a0 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99140b9050 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 10 client.26342 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555c575e03f0 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f991800f9a0 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f99140a6080 con 0x555c57967f20 2024-03-28T18:47:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 10 client.26342 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 10 client.26342 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 20 client.26342 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c57981200 0x555c579835f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 4 client.26342 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.695+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c578d9790 con 0x555c57981200 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.696+0000 7f991fe00640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c57981200 0x555c579835f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.696+0000 7f991fe00640 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c57981200 0x555c579835f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23631af57171d16a server_cookie=e07eff1a6143c3ed in_seq=0 out_seq=0 2024-03-28T18:47:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.696+0000 7f9902000640 10 client.26342 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f9902000640 1 -- 172.21.15.119:0/929808759 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555c578d9790 con 0x555c57981200 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f9902000640 10 client.26342 handle_command_reply: tid=0 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 client.26342 shutdown 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 msgr2=0x555c57970980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f991400e9c0 tx=0x7f991409b210 comp rx=0 tx=0).stop 2024-03-28T18:47:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 shutdown_connections 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c57981200 0x555c579835f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c5796e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f9902000640 0 client.26342 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57968c00 0x555c5796dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 --2- 172.21.15.119:0/929808759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57970980 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 >> 172.21.15.119:0/929808759 conn(0x555c5792f200 msgr2=0x555c579646c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 shutdown_connections 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.697+0000 7f992a366740 1 -- 172.21.15.119:0/929808759 wait complete. 2024-03-28T18:47:19.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.698+0000 7f992a366740 20 client.26342 trim_cache size 0 max 16384 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.714+0000 7f992a366740 1 Processor -- start 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.714+0000 7f992a366740 1 -- start start 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.714+0000 7f992a366740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c5796cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f992a366740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f992a366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c57970980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f992a366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b3a60 con 0x555c57971290 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f992a366740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b53e0 con 0x555c57968c00 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f992a366740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b40c0 con 0x555c57967f20 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49244/0 (socket says 172.21.15.119:49244) 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 -- 172.21.15.119:0/3033854008 learned_addr learned my addr 172.21.15.119:0/3033854008 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f9924c00640 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c5796cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f9925600640 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c57970980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 -- 172.21.15.119:0/3033854008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 msgr2=0x555c5796cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c5796cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 -- 172.21.15.119:0/3033854008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 msgr2=0x555c57970980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c57970980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 -- 172.21.15.119:0/3033854008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c575a00f0 con 0x555c57968c00 2024-03-28T18:47:19.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.715+0000 7f991fe00640 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9910005ab0 tx=0x7f9910045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9be61f32527e478e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f990d200640 1 -- 172.21.15.119:0/3033854008 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910004400 con 0x555c57968c00 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f990d200640 1 -- 172.21.15.119:0/3033854008 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9910004ba0 con 0x555c57968c00 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f990d200640 1 -- 172.21.15.119:0/3033854008 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99100035f0 con 0x555c57968c00 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 -- 172.21.15.119:0/3033854008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 msgr2=0x555c57970440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9910005ab0 tx=0x7f9910045000 comp rx=0 tx=0).stop 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 -- 172.21.15.119:0/3033854008 shutdown_connections 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57971290 0x555c57970980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57968c00 0x555c57970440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 --2- 172.21.15.119:0/3033854008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57967f20 0x555c5796cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 -- 172.21.15.119:0/3033854008 >> 172.21.15.119:0/3033854008 conn(0x555c5792f200 msgr2=0x555c579645e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 -- 172.21.15.119:0/3033854008 shutdown_connections 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.716+0000 7f992a366740 1 -- 172.21.15.119:0/3033854008 wait complete. 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 -- start start 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 Processor -- start 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c5796adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 0x555c5796b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b4d80 con 0x555c57968c00 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b4720 con 0x555c57967f20 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201b1c20 con 0x555c57971290 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.717+0000 7f992a366740 10 client.0 _ll_register_callbacks cb 0x7f991400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:19.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:49256/0 (socket says 172.21.15.119:49256) 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 -- 172.21.15.119:0/1875726222 learned_addr learned my addr 172.21.15.119:0/1875726222 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f991fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c5796adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9925600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 0x555c5796b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 -- 172.21.15.119:0/1875726222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 msgr2=0x555c5796b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 0x555c5796b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 -- 172.21.15.119:0/1875726222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 msgr2=0x555c5796adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c5796adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 -- 172.21.15.119:0/1875726222 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c5796b840 con 0x555c57967f20 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9925600640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 0x555c5796b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9924c00640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f991409b5b0 tx=0x7f991409ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f18bbb1dd3283a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9902000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140a3bc0 con 0x555c57967f20 2024-03-28T18:47:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f991409ad00 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.718+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9910004ba0 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f991408b050 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 10 client.26348 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555c579dd730 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9910004ba0 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f991409a770 con 0x555c57967f20 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 10 client.26348 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 10 client.26348 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 20 client.26348 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c579dde80 0x555c579e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 4 client.26348 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555c578d3ae0 con 0x555c579dde80 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.719+0000 7f991fe00640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c579dde80 0x555c579e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:19.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f991fe00640 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c579dde80 0x555c579e02a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c30e428eaf76ee server_cookie=9fd7ef0233bbcdba in_seq=0 out_seq=0 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f9902000640 10 client.26348 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f9902000640 1 -- 172.21.15.119:0/1875726222 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555c578d3ae0 con 0x555c579dde80 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f9902000640 10 client.26348 handle_command_reply: tid=0 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 client.26348 shutdown 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 msgr2=0x555c57978fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f991409b5b0 tx=0x7f991409ba30 comp rx=0 tx=0).stop 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 shutdown_connections 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555c579dde80 0x555c579e02a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c57971290 0x555c5796b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555c57968c00 0x555c5796adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 --2- 172.21.15.119:0/1875726222 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555c57967f20 0x555c57978fb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f9902000640 0 client.26348 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 >> 172.21.15.119:0/1875726222 conn(0x555c5792f200 msgr2=0x555c57961200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 shutdown_connections 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 1 -- 172.21.15.119:0/1875726222 wait complete. 2024-03-28T18:47:19.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.720+0000 7f992a366740 20 client.26348 trim_cache size 0 max 16384 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "09ad7f11-4a97-44fb-a6bb-97b6b4eb8d5e": { 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "09ad7f11-4a97-44fb-a6bb-97b6b4eb8d5e", 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:47:19.726 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 msgr2=0x7f99080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 0x7f99080a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9910009900 tx=0x7f9910005a20 comp rx=0 tx=0).stop 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 msgr2=0x7f992019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f991400e9f0 tx=0x7f991400edf0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 shutdown_connections 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99080a2970 0x7f99080a4e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99201124e0 0x7f99201a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200764f0 0x7f992019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 --2- 172.21.15.119:0/120774087 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9920075bd0 0x7f992019e5d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 >> 172.21.15.119:0/120774087 conn(0x7f99200ff2f0 msgr2=0x7f9920113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 shutdown_connections 2024-03-28T18:47:19.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:19.721+0000 7f9927600640 1 -- 172.21.15.119:0/120774087 wait complete. 2024-03-28T18:47:19.736 INFO:tasks.cephfs.filesystem:scrub status for tag:09ad7f11-4a97-44fb-a6bb-97b6b4eb8d5e - {'path': '/', 'tag': '09ad7f11-4a97-44fb-a6bb-97b6b4eb8d5e', 'options': 'recursive,force'} 2024-03-28T18:47:20.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:20 smithi196 ceph-mon[36300]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 507 MiB/s wr, 140 op/s 2024-03-28T18:47:20.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:20 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:47:20.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:20 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:20.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:20 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:47:20.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:20 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:20 smithi131 ceph-mon[36065]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 507 MiB/s wr, 140 op/s 2024-03-28T18:47:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:20 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:47:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:20 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:20 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:47:20.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:20 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:20 smithi119 ceph-mon[31813]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 507 MiB/s wr, 140 op/s 2024-03-28T18:47:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:20 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:47:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:20 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:20 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:47:20.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:20 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:22.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:22 smithi196 ceph-mon[36300]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:22.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:22 smithi131 ceph-mon[36065]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:22 smithi119 ceph-mon[31813]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:23.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:23 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:47:23.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:23 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:47:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:23 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:47:24.165 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:24.165 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 readers = 5550555.50 kB/sec 2024-03-28T18:47:24.165 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 readers = 5548132.97 kB/sec 2024-03-28T18:47:24.165 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 5550555.50 kB/sec 2024-03-28T18:47:24.165 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 5550555.50 kB/sec 2024-03-28T18:47:24.166 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 5550555.50 kB/sec 2024-03-28T18:47:24.166 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 10485760.00 kB 2024-03-28T18:47:24.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:24 smithi196 ceph-mon[36300]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:24.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:24 smithi131 ceph-mon[36065]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:24 smithi119 ceph-mon[31813]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 328 MiB/s wr, 90 op/s 2024-03-28T18:47:26.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:26 smithi196 ceph-mon[36300]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 330 MiB/s wr, 95 op/s 2024-03-28T18:47:26.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:26 smithi131 ceph-mon[36065]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 330 MiB/s wr, 95 op/s 2024-03-28T18:47:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:26 smithi119 ceph-mon[31813]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 330 MiB/s wr, 95 op/s 2024-03-28T18:47:28.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:28 smithi196 ceph-mon[36300]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 164 MiB/s wr, 49 op/s 2024-03-28T18:47:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:28 smithi131 ceph-mon[36065]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 164 MiB/s wr, 49 op/s 2024-03-28T18:47:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:28 smithi119 ceph-mon[31813]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 164 MiB/s wr, 49 op/s 2024-03-28T18:47:29.008 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:29.008 INFO:tasks.workunit.client.0.smithi119.stdout: Children see throughput for 1 re-readers = 3802161.25 kB/sec 2024-03-28T18:47:29.009 INFO:tasks.workunit.client.0.smithi119.stdout: Parent sees throughput for 1 re-readers = 3801022.62 kB/sec 2024-03-28T18:47:29.009 INFO:tasks.workunit.client.0.smithi119.stdout: Min throughput per process = 3802161.25 kB/sec 2024-03-28T18:47:29.009 INFO:tasks.workunit.client.0.smithi119.stdout: Max throughput per process = 3802161.25 kB/sec 2024-03-28T18:47:29.009 INFO:tasks.workunit.client.0.smithi119.stdout: Avg throughput per process = 3802161.25 kB/sec 2024-03-28T18:47:29.009 INFO:tasks.workunit.client.0.smithi119.stdout: Min xfer = 10485760.00 kB 2024-03-28T18:47:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:30 smithi196 ceph-mon[36300]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 164 MiB/s wr, 53 op/s 2024-03-28T18:47:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:30 smithi131 ceph-mon[36065]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 164 MiB/s wr, 53 op/s 2024-03-28T18:47:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:30 smithi119 ceph-mon[31813]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 164 MiB/s wr, 53 op/s 2024-03-28T18:47:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:32 smithi196 ceph-mon[36300]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:32 smithi131 ceph-mon[36065]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:32 smithi119 ceph-mon[31813]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:34 smithi131 ceph-mon[36065]: pgmap v108: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:34 smithi119 ceph-mon[31813]: pgmap v108: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:34.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:34 smithi196 ceph-mon[36300]: pgmap v108: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-03-28T18:47:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:36 smithi119 ceph-mon[31813]: pgmap v109: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-03-28T18:47:36.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:36 smithi196 ceph-mon[36300]: pgmap v109: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-03-28T18:47:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:36 smithi131 ceph-mon[36065]: pgmap v109: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 1.7 MiB/s wr, 13 op/s 2024-03-28T18:47:38.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:38 smithi196 ceph-mon[36300]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T18:47:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:38 smithi119 ceph-mon[31813]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T18:47:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:38 smithi131 ceph-mon[36065]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-28T18:47:39.768 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:39.768 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:39.768 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-03-28T18:47:39.768 INFO:tasks.workunit.client.0.smithi119.stdout:iozone test complete. 2024-03-28T18:47:39.772 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T18:47:39.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T18:47:39.883 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-03-28T18:47:39.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T18:47:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:40 smithi119 ceph-mon[31813]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-03-28T18:47:40.848 DEBUG:teuthology.parallel:result is None 2024-03-28T18:47:40.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T18:47:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:40 smithi196 ceph-mon[36300]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-03-28T18:47:40.942 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T18:47:40.942 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T18:47:40.953 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T18:47:40.953 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T18:47:40.953 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:47:40.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T18:47:40.954 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:40 smithi131 ceph-mon[36065]: pgmap v111: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 226 op/s 2024-03-28T18:47:41.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T18:47:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:42 smithi119 ceph-mon[31813]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:42 smithi196 ceph-mon[36300]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:42.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:42 smithi131 ceph-mon[36065]: pgmap v112: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:44 smithi119 ceph-mon[31813]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:44 smithi196 ceph-mon[36300]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:44 smithi131 ceph-mon[36065]: pgmap v113: 97 pgs: 97 active+clean; 11 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.2 KiB/s wr, 222 op/s 2024-03-28T18:47:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:46 smithi119 ceph-mon[31813]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 227 op/s 2024-03-28T18:47:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:46 smithi196 ceph-mon[36300]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 227 op/s 2024-03-28T18:47:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:46 smithi131 ceph-mon[36065]: pgmap v114: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 227 op/s 2024-03-28T18:47:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:48 smithi119 ceph-mon[31813]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 223 op/s 2024-03-28T18:47:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:48 smithi196 ceph-mon[36300]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 223 op/s 2024-03-28T18:47:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:48 smithi131 ceph-mon[36065]: pgmap v115: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.8 KiB/s wr, 223 op/s 2024-03-28T18:47:49.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:47:49.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:47:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 -- 172.21.15.119:0/4185089760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63880743f0 msgr2=0x7f638810d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 --2- 172.21.15.119:0/4185089760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63880743f0 0x7f638810d930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6370009900 tx=0x7f637002bcc0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 -- 172.21.15.119:0/4185089760 shutdown_connections 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 --2- 172.21.15.119:0/4185089760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 0x7f63881102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 --2- 172.21.15.119:0/4185089760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63880743f0 0x7f638810d930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 --2- 172.21.15.119:0/4185089760 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6388073ad0 0x7f6388073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 -- 172.21.15.119:0/4185089760 >> 172.21.15.119:0/4185089760 conn(0x7f6388078aa0 msgr2=0x7f6388078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 -- 172.21.15.119:0/4185089760 shutdown_connections 2024-03-28T18:47:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.969+0000 7f638ee00640 1 -- 172.21.15.119:0/4185089760 wait complete. 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.970+0000 7f638ee00640 1 Processor -- start 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 -- start start 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 0x7f638819e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388113c60 con 0x7f638810de70 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388113ae0 con 0x7f63880743f0 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388113de0 con 0x7f6388073ad0 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f6387e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f637fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f6387e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46420/0 (socket says 172.21.15.119:46420) 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f6387e00640 1 -- 172.21.15.119:0/2207157158 learned_addr learned my addr 172.21.15.119:0/2207157158 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f637fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45744/0 (socket says 172.21.15.119:45744) 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.971+0000 7f638ce00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 0x7f638819e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 -- 172.21.15.119:0/2207157158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 msgr2=0x7f638819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 -- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 msgr2=0x7f638819e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 0x7f638819e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 -- 172.21.15.119:0/2207157158 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638819f110 con 0x7f63880743f0 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f6387e00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f637fe00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f637002c200 tx=0x7f6370004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f6385600640 1 -- 172.21.15.119:0/2207157158 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370042020 con 0x7f63880743f0 2024-03-28T18:47:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f6385600640 1 -- 172.21.15.119:0/2207157158 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6370005000 con 0x7f63880743f0 2024-03-28T18:47:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f6385600640 1 -- 172.21.15.119:0/2207157158 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370003c60 con 0x7f63880743f0 2024-03-28T18:47:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.972+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638819f3a0 con 0x7f63880743f0 2024-03-28T18:47:49.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.973+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63881a6c90 con 0x7f63880743f0 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.977+0000 7f6385600640 1 -- 172.21.15.119:0/2207157158 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6370045020 con 0x7f63880743f0 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.977+0000 7f6391bc5740 1 Processor -- start 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.977+0000 7f6391bc5740 1 -- start start 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611e25250 con 0x55b611ec0f20 2024-03-28T18:47:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611dcda30 con 0x55b611ec1c00 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6391bc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611e212f0 con 0x55b611eca290 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f6387e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45754/0 (socket says 172.21.15.119:45754) 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 -- 172.21.15.119:0/771321116 learned_addr learned my addr 172.21.15.119:0/771321116 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f637fe00640 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 -- 172.21.15.119:0/771321116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 msgr2=0x55b611ecc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 -- 172.21.15.119:0/771321116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 msgr2=0x55b611ec13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.978+0000 7f638ce00640 1 -- 172.21.15.119:0/771321116 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b611db8690 con 0x55b611ec1c00 2024-03-28T18:47:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f638ce00640 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6380009f70 tx=0x7f6380010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7c98e34a41ad9e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f637d600640 1 -- 172.21.15.119:0/771321116 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380016070 con 0x55b611ec1c00 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f637d600640 1 -- 172.21.15.119:0/771321116 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6380004d20 con 0x55b611ec1c00 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f637d600640 1 -- 172.21.15.119:0/771321116 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380012870 con 0x55b611ec1c00 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f6385600640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 0x7f636c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f6385600640 1 -- 172.21.15.119:0/2207157158 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f63700f0960 con 0x7f63880743f0 2024-03-28T18:47:49.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.979+0000 7f6387e00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 0x7f636c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- 172.21.15.119:0/771321116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 msgr2=0x55b611ec0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6380009f70 tx=0x7f6380010740 comp rx=0 tx=0).stop 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- 172.21.15.119:0/771321116 shutdown_connections 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 --2- 172.21.15.119:0/771321116 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec0950 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- 172.21.15.119:0/771321116 >> 172.21.15.119:0/771321116 conn(0x55b611e88200 msgr2=0x55b611ebb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- 172.21.15.119:0/771321116 shutdown_connections 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- 172.21.15.119:0/771321116 wait complete. 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6387e00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 0x7f636c0a4eb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f637800bfd0 tx=0x7f637800b4a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 -- start start 2024-03-28T18:47:49.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.980+0000 7f6391bc5740 1 Processor -- start 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611ecd100 con 0x55b611ec0f20 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611de0e80 con 0x55b611ec1c00 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b611ecd280 con 0x55b611eca290 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6391bc5740 10 client.0 _ll_register_callbacks cb 0x55b611df3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f637fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f6387e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f638ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f637fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45770/0 (socket says 172.21.15.119:45770) 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.981+0000 7f637fe00640 1 -- 172.21.15.119:0/3469060720 learned_addr learned my addr 172.21.15.119:0/3469060720 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:49.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 -- 172.21.15.119:0/3469060720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 msgr2=0x55b611ec71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 -- 172.21.15.119:0/3469060720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 msgr2=0x55b611ec9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 -- 172.21.15.119:0/3469060720 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b611c037d0 con 0x55b611ec1c00 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f638ce00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f637fe00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f63700ee1c0 tx=0x7f6370009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98ee4a7e2502ac7b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6376000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63700bce40 con 0x55b611ec1c00 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f63700ee2d0 con 0x55b611ec1c00 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6380004d20 con 0x55b611ec1c00 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63700bc9d0 con 0x55b611ec1c00 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6391bc5740 10 client.26360 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:49.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b611b393f0 con 0x55b611ec1c00 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.982+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6380004d20 con 0x55b611ec1c00 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f63700b7be0 con 0x55b611ec1c00 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 10 client.26360 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 10 client.26360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 20 client.26360 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611eda010 0x55b611edc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 4 client.26360 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b611e32790 con 0x55b611eda010 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.983+0000 7f6387e00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611eda010 0x55b611edc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6387e00640 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611eda010 0x55b611edc430 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d17368ce9a1a3e7 server_cookie=e8b2f5e557930405 in_seq=0 out_seq=0 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6376000640 10 client.26360 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:49.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6376000640 1 -- 172.21.15.119:0/3469060720 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b611e32790 con 0x55b611eda010 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6376000640 10 client.26360 handle_command_reply: tid=0 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6391bc5740 1 client.26360 shutdown 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 msgr2=0x55b611ec6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.984+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f63700ee1c0 tx=0x7f6370009290 comp rx=0 tx=0).stop 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 shutdown_connections 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611eda010 0x55b611edc430 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec1c00 0x55b611ec6c90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6376000640 0 client.26360 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 --2- 172.21.15.119:0/3469060720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec0f20 0x55b611ec9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 >> 172.21.15.119:0/3469060720 conn(0x55b611e88200 msgr2=0x55b611ebd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 shutdown_connections 2024-03-28T18:47:49.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 1 -- 172.21.15.119:0/3469060720 wait complete. 2024-03-28T18:47:49.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:49.985+0000 7f6391bc5740 20 client.26360 trim_cache size 0 max 16384 2024-03-28T18:47:50.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.002+0000 7f6391bc5740 1 Processor -- start 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.002+0000 7f6391bc5740 1 -- start start 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.002+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.002+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6391bc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881af550 con 0x55b611ec1c00 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6391bc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881b0ed0 con 0x55b611ec0f20 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6391bc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881afbb0 con 0x55b611eca290 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f638ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45784/0 (socket says 172.21.15.119:45784) 2024-03-28T18:47:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 -- 172.21.15.119:0/1585004862 learned_addr learned my addr 172.21.15.119:0/1585004862 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f637fe00640 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 -- 172.21.15.119:0/1585004862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 msgr2=0x55b611ecb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 -- 172.21.15.119:0/1585004862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 msgr2=0x55b611ec9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 -- 172.21.15.119:0/1585004862 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b611af90f0 con 0x55b611ec0f20 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.003+0000 7f6387e00640 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55b611edcca0 tx=0x7f6378019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd8be977e2ef033 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f637cc00640 1 -- 172.21.15.119:0/1585004862 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637802f070 con 0x55b611ec0f20 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f637cc00640 1 -- 172.21.15.119:0/1585004862 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f637801e7f0 con 0x55b611ec0f20 2024-03-28T18:47:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f637cc00640 1 -- 172.21.15.119:0/1585004862 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637802a9c0 con 0x55b611ec0f20 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 -- 172.21.15.119:0/1585004862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 msgr2=0x55b611ec9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x55b611edcca0 tx=0x7f6378019000 comp rx=0 tx=0).stop 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 -- 172.21.15.119:0/1585004862 shutdown_connections 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ecb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 --2- 172.21.15.119:0/1585004862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec9350 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 -- 172.21.15.119:0/1585004862 >> 172.21.15.119:0/1585004862 conn(0x55b611e88200 msgr2=0x55b611ebb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 -- 172.21.15.119:0/1585004862 shutdown_connections 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.004+0000 7f6391bc5740 1 -- 172.21.15.119:0/1585004862 wait complete. 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 -- start start 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 Processor -- start 2024-03-28T18:47:50.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881b0870 con 0x55b611ec1c00 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881b0210 con 0x55b611ec0f20 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63881ad710 con 0x55b611eca290 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.005+0000 7f6391bc5740 10 client.0 _ll_register_callbacks cb 0x7f6370009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f638ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f637fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f638ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46436/0 (socket says 172.21.15.119:46436) 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45798/0 (socket says 172.21.15.119:45798) 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f638ce00640 1 -- 172.21.15.119:0/2502652140 learned_addr learned my addr 172.21.15.119:0/2502652140 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 -- 172.21.15.119:0/2502652140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 msgr2=0x55b611ec3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 -- 172.21.15.119:0/2502652140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 msgr2=0x55b611ec3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 -- 172.21.15.119:0/2502652140 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b611f34a70 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6387e00640 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6378004650 tx=0x7f63780108d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eaa99ab89f58bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.006+0000 7f6376000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637802f070 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f637801ed60 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f637801e7f0 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6378041050 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 10 client.26366 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b611f35d40 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f637801e7f0 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f637803c030 con 0x55b611ec0f20 2024-03-28T18:47:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 10 client.26366 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 10 client.26366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 20 client.26366 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611f36350 0x55b611f38740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 4 client.26366 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.007+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b611e2cae0 con 0x55b611f36350 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f637fe00640 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611f36350 0x55b611f38740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f637fe00640 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611f36350 0x55b611f38740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be253a6783a8a8a6 server_cookie=72ee8f0c4c2a7d40 in_seq=0 out_seq=0 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6376000640 10 client.26366 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6376000640 1 -- 172.21.15.119:0/2502652140 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b611e2cae0 con 0x55b611f36350 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6376000640 10 client.26366 handle_command_reply: tid=0 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 client.26366 shutdown 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 msgr2=0x55b611ec3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6378004650 tx=0x7f63780108d0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 shutdown_connections 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b611f36350 0x55b611f38740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b611eca290 0x55b611ec3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b611ec1c00 0x55b611ec3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 --2- 172.21.15.119:0/2502652140 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b611ec0f20 0x55b611ec3500 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6376000640 0 client.26366 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.008+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 >> 172.21.15.119:0/2502652140 conn(0x55b611e88200 msgr2=0x55b611ec5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 shutdown_connections 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f6391bc5740 1 -- 172.21.15.119:0/2502652140 wait complete. 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f6391bc5740 20 client.26366 trim_cache size 0 max 16384 2024-03-28T18:47:50.014 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:50.015 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:47:50.015 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:47:50.015 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:50.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 msgr2=0x7f636c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 0x7f636c0a4eb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f637800bfd0 tx=0x7f637800b4a0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 msgr2=0x7f638819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f637002c200 tx=0x7f6370004430 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 shutdown_connections 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f636c0a29f0 0x7f636c0a4eb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f638810de70 0x7f638819e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f63880743f0 0x7f638819a660 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 --2- 172.21.15.119:0/2207157158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6388073ad0 0x7f638819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.009+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 >> 172.21.15.119:0/2207157158 conn(0x7f6388078aa0 msgr2=0x7f63880fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.010+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 shutdown_connections 2024-03-28T18:47:50.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.010+0000 7f638ee00640 1 -- 172.21.15.119:0/2207157158 wait complete. 2024-03-28T18:47:50.026 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:47:50.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:47:50.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 -- 172.21.15.119:0/1296914011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 msgr2=0x7f3a801148e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 --2- 172.21.15.119:0/1296914011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801148e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009cd0 tx=0x7f3a7802c0e0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 -- 172.21.15.119:0/1296914011 shutdown_connections 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 --2- 172.21.15.119:0/1296914011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801148e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 --2- 172.21.15.119:0/1296914011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a800764f0 0x7f3a80111fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 --2- 172.21.15.119:0/1296914011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a80075bd0 0x7f3a80075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 -- 172.21.15.119:0/1296914011 >> 172.21.15.119:0/1296914011 conn(0x7f3a800ff2e0 msgr2=0x7f3a80101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 -- 172.21.15.119:0/1296914011 shutdown_connections 2024-03-28T18:47:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.227+0000 7f3a85800640 1 -- 172.21.15.119:0/1296914011 wait complete. 2024-03-28T18:47:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.228+0000 7f3a85800640 1 Processor -- start 2024-03-28T18:47:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.228+0000 7f3a85800640 1 -- start start 2024-03-28T18:47:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 0x7f3a8019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a80118050 con 0x7f3a800764f0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a80117ed0 con 0x7f3a80075bd0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a85800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a801181d0 con 0x7f3a801124f0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 0x7f3a8019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 0x7f3a8019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45814/0 (socket says 172.21.15.119:45814) 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36600/0 (socket says 172.21.15.119:36600) 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7ea00640 1 -- 172.21.15.119:0/4002136911 learned_addr learned my addr 172.21.15.119:0/4002136911 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7f400640 1 -- 172.21.15.119:0/4002136911 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 msgr2=0x7f3a8019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7f400640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 0x7f3a8019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7f400640 1 -- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 msgr2=0x7f3a8019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.229+0000 7f3a7f400640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a7f400640 1 -- 172.21.15.119:0/4002136911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a801a3580 con 0x7f3a801124f0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a7e000640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a7f400640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801a2e50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3a78037040 tx=0x7f3a780324e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78039040 con 0x7f3a801124f0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a78044070 con 0x7f3a801124f0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78004280 con 0x7f3a801124f0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a801a3810 con 0x7f3a801124f0 2024-03-28T18:47:50.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.230+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a801a3c80 con 0x7f3a801124f0 2024-03-28T18:47:50.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.231+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a80068490 con 0x7f3a801124f0 2024-03-28T18:47:50.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.233+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3a78003970 con 0x7f3a801124f0 2024-03-28T18:47:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.235+0000 7f3a77400640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 0x7f3a640a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.235+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3a780f1cf0 con 0x7f3a801124f0 2024-03-28T18:47:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.235+0000 7f3a7ea00640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 0x7f3a640a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.236+0000 7f3a7ea00640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 0x7f3a640a4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a700095f0 tx=0x7f3a70005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.241+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a801a3c80 con 0x7f3a801124f0 2024-03-28T18:47:50.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.522+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a8002d070 con 0x7f3a801124f0 2024-03-28T18:47:50.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.523+0000 7f3a77400640 1 -- 172.21.15.119:0/4002136911 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f3a780beaa0 con 0x7f3a801124f0 2024-03-28T18:47:50.523 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:47:50.523 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:47:50.523 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 msgr2=0x7f3a640a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 0x7f3a640a4eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a700095f0 tx=0x7f3a70005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 msgr2=0x7f3a801a2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801a2e50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3a78037040 tx=0x7f3a780324e0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 shutdown_connections 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a640a29f0 0x7f3a640a4eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a801124f0 0x7f3a801a2e50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a800764f0 0x7f3a8019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 --2- 172.21.15.119:0/4002136911 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a80075bd0 0x7f3a8019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.525+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 >> 172.21.15.119:0/4002136911 conn(0x7f3a800ff2e0 msgr2=0x7f3a80113c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.526+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 shutdown_connections 2024-03-28T18:47:50.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.526+0000 7f3a85800640 1 -- 172.21.15.119:0/4002136911 wait complete. 2024-03-28T18:47:50.544 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:50.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:47:50.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.751+0000 7f1222000640 1 -- 172.21.15.119:0/362842740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 msgr2=0x7f121c114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.751+0000 7f1222000640 1 --2- 172.21.15.119:0/362842740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c114b50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1214009ca0 tx=0x7f121402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 -- 172.21.15.119:0/362842740 shutdown_connections 2024-03-28T18:47:50.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 --2- 172.21.15.119:0/362842740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c114b50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 --2- 172.21.15.119:0/362842740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c108860 0x7f121c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 --2- 172.21.15.119:0/362842740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c107eb0 0x7f121c108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 -- 172.21.15.119:0/362842740 >> 172.21.15.119:0/362842740 conn(0x7f121c075be0 msgr2=0x7f121c078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 -- 172.21.15.119:0/362842740 shutdown_connections 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.752+0000 7f1222000640 1 -- 172.21.15.119:0/362842740 wait complete. 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 Processor -- start 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 -- start start 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c107eb0 0x7f121c19e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c1a2dc0 0x7f121c1a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c118030 con 0x7f121c107eb0 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c117eb0 con 0x7f121c10cf80 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1222000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1181b0 con 0x7f121c1a2dc0 2024-03-28T18:47:50.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1213e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f121b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c107eb0 0x7f121c19e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1213e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45834/0 (socket says 172.21.15.119:45834) 2024-03-28T18:47:50.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f1213e00640 1 -- 172.21.15.119:0/1949432933 learned_addr learned my addr 172.21.15.119:0/1949432933 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.753+0000 7f121be00640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c1a2dc0 0x7f121c1a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 -- 172.21.15.119:0/1949432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c1a2dc0 msgr2=0x7f121c1a3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c1a2dc0 0x7f121c1a3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 -- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c107eb0 msgr2=0x7f121c19e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c107eb0 0x7f121c19e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 -- 172.21.15.119:0/1949432933 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c1a38b0 con 0x7f121c10cf80 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1213e00640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1204005fd0 tx=0x7f120400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1218c00640 1 -- 172.21.15.119:0/1949432933 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204017070 con 0x7f121c10cf80 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1218c00640 1 -- 172.21.15.119:0/1949432933 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1204009c30 con 0x7f121c10cf80 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c1a3ba0 con 0x7f121c10cf80 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1218c00640 1 -- 172.21.15.119:0/1949432933 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204010460 con 0x7f121c10cf80 2024-03-28T18:47:50.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.754+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c1ab3e0 con 0x7f121c10cf80 2024-03-28T18:47:50.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.757+0000 7f1224c0d740 1 Processor -- start 2024-03-28T18:47:50.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.757+0000 7f1218c00640 1 -- 172.21.15.119:0/1949432933 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1204003e70 con 0x7f121c10cf80 2024-03-28T18:47:50.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 -- start start 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee4f92c0 con 0x563cee596080 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee4a1c90 con 0x563cee59ec00 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1224c0d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee4f5360 con 0x563cee5956d0 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f121be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f121be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46480/0 (socket says 172.21.15.119:46480) 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f121be00640 1 -- 172.21.15.119:0/1159564348 learned_addr learned my addr 172.21.15.119:0/1159564348 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f121b400640 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.758+0000 7f1213e00640 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 -- 172.21.15.119:0/1159564348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 msgr2=0x563cee5a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 -- 172.21.15.119:0/1159564348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 msgr2=0x563cee59e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 -- 172.21.15.119:0/1159564348 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cee4c6080 con 0x563cee5956d0 2024-03-28T18:47:50.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f121be00640 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f121402c700 tx=0x7f1214005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d93e1ff7966db69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f1218c00640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f120be00640 1 -- 172.21.15.119:0/1159564348 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214043070 con 0x563cee5956d0 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f120be00640 1 -- 172.21.15.119:0/1159564348 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f121402cca0 con 0x563cee5956d0 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f120be00640 1 -- 172.21.15.119:0/1159564348 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214039040 con 0x563cee5956d0 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.759+0000 7f1218c00640 1 -- 172.21.15.119:0/1949432933 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f12040ce9f0 con 0x7f121c10cf80 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f121b400640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 -- 172.21.15.119:0/1159564348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 msgr2=0x563cee595ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f121402c700 tx=0x7f1214005f80 comp rx=0 tx=0).stop 2024-03-28T18:47:50.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 -- 172.21.15.119:0/1159564348 shutdown_connections 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1159564348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee595ab0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 -- 172.21.15.119:0/1159564348 >> 172.21.15.119:0/1159564348 conn(0x563cee58fda0 msgr2=0x563cee590180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 -- 172.21.15.119:0/1159564348 shutdown_connections 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f1224c0d740 1 -- 172.21.15.119:0/1159564348 wait complete. 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.760+0000 7f121b400640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 0x7f12000a4eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x563cee590e10 tx=0x7f120c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 -- start start 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 Processor -- start 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 0x563cee59e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 0x563cee59bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee5a1b10 con 0x563cee5956d0 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee4b50d0 con 0x563cee596080 2024-03-28T18:47:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.761+0000 7f1224c0d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cee5a1c90 con 0x563cee59ec00 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1224c0d740 10 client.0 _ll_register_callbacks cb 0x563cee4b9e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f121b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 0x563cee59e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f121be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 0x563cee59bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f121b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 0x563cee59e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36638/0 (socket says 172.21.15.119:36638) 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f121b400640 1 -- 172.21.15.119:0/3241836317 learned_addr learned my addr 172.21.15.119:0/3241836317 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45852/0 (socket says 172.21.15.119:45852) 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 -- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 msgr2=0x563cee59bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 0x563cee59bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 -- 172.21.15.119:0/3241836317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 msgr2=0x563cee59e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 0x563cee59e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 -- 172.21.15.119:0/3241836317 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cee20d3f0 con 0x563cee596080 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f121be00640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 0x563cee59bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1213e00640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12040cc080 tx=0x7f120409a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8db94472c8264 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.762+0000 7f1208c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040cc500 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f120409a3e0 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f121402cca0 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f120409af90 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 10 client.26384 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563cee2d77d0 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f121402cca0 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1204095b00 con 0x563cee596080 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 10 client.26384 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 10 client.26384 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 20 client.26384 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee5ae990 0x563cee5b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 4 client.26384 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.763+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cee506800 con 0x563cee5ae990 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.764+0000 7f121b400640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee5ae990 0x563cee5b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.764+0000 7f121b400640 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee5ae990 0x563cee5b0d80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f5740e5156f492d server_cookie=47454dcd90d9f59b in_seq=0 out_seq=0 2024-03-28T18:47:50.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.764+0000 7f1208c00640 10 client.26384 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1208c00640 1 -- 172.21.15.119:0/3241836317 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563cee506800 con 0x563cee5ae990 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1208c00640 10 client.26384 handle_command_reply: tid=0 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1224c0d740 1 client.26384 shutdown 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 msgr2=0x563cee59b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f12040cc080 tx=0x7f120409a8f0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 shutdown_connections 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.765+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee5ae990 0x563cee5b0d80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee59ec00 0x563cee59bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1208c00640 0 client.26384 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee596080 0x563cee59b7b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 --2- 172.21.15.119:0/3241836317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee5956d0 0x563cee59e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 >> 172.21.15.119:0/3241836317 conn(0x563cee58fda0 msgr2=0x563cee591af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 shutdown_connections 2024-03-28T18:47:50.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 1 -- 172.21.15.119:0/3241836317 wait complete. 2024-03-28T18:47:50.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.766+0000 7f1224c0d740 20 client.26384 trim_cache size 0 max 16384 2024-03-28T18:47:50.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.779+0000 7f1224c0d740 1 Processor -- start 2024-03-28T18:47:50.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.779+0000 7f1224c0d740 1 -- start start 2024-03-28T18:47:50.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.779+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.779+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5aa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1224c0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b3980 con 0x563cee596080 2024-03-28T18:47:50.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1224c0d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b5300 con 0x563cee59ec00 2024-03-28T18:47:50.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1224c0d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b3fe0 con 0x563cee5956d0 2024-03-28T18:47:50.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46492/0 (socket says 172.21.15.119:46492) 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121b400640 1 -- 172.21.15.119:0/1022771400 learned_addr learned my addr 172.21.15.119:0/1022771400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1213e00640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 -- 172.21.15.119:0/1022771400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 msgr2=0x563cee59dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 -- 172.21.15.119:0/1022771400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 msgr2=0x563cee59e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 -- 172.21.15.119:0/1022771400 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cee595e20 con 0x563cee59ec00 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f1213e00640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121b400640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.780+0000 7f121be00640 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5aa400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1214037040 tx=0x7f1214003ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5428d3c138a7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f120b400640 1 -- 172.21.15.119:0/1022771400 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214039040 con 0x563cee59ec00 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f120b400640 1 -- 172.21.15.119:0/1022771400 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f121404c070 con 0x563cee59ec00 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f120b400640 1 -- 172.21.15.119:0/1022771400 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214043040 con 0x563cee59ec00 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 -- 172.21.15.119:0/1022771400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 msgr2=0x563cee5aa400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5aa400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1214037040 tx=0x7f1214003ab0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 -- 172.21.15.119:0/1022771400 shutdown_connections 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee59ec00 0x563cee5aa400 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee596080 0x563cee59e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1022771400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee5956d0 0x563cee59dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 -- 172.21.15.119:0/1022771400 >> 172.21.15.119:0/1022771400 conn(0x563cee58fda0 msgr2=0x563cee58eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 -- 172.21.15.119:0/1022771400 shutdown_connections 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.781+0000 7f1224c0d740 1 -- 172.21.15.119:0/1022771400 wait complete. 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 -- start start 2024-03-28T18:47:50.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 Processor -- start 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee5956d0 0x563cee5a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 0x563cee5a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b4ca0 con 0x563cee59ec00 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b4640 con 0x563cee5956d0 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c1b1b40 con 0x563cee596080 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.782+0000 7f1224c0d740 10 client.0 _ll_register_callbacks cb 0x7f120409ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f121be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 0x563cee5a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f121be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36656/0 (socket says 172.21.15.119:36656) 2024-03-28T18:47:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f121be00640 1 -- 172.21.15.119:0/1921536969 learned_addr learned my addr 172.21.15.119:0/1921536969 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f121b400640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee5956d0 0x563cee5a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 -- 172.21.15.119:0/1921536969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee5956d0 msgr2=0x563cee5a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee5956d0 0x563cee5a2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 -- 172.21.15.119:0/1921536969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 msgr2=0x563cee6091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 -- 172.21.15.119:0/1921536969 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cee6096e0 con 0x563cee596080 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f121be00640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1213e00640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 0x563cee5a2f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1204097960 tx=0x7f12040cc630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf090dfd7aedcdd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1208c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040d4140 con 0x563cee596080 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f121404c070 con 0x563cee596080 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.783+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1204004160 con 0x563cee596080 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12040a3730 con 0x563cee596080 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 10 client.26374 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:50.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563cee60a980 con 0x563cee596080 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f121404c070 con 0x563cee596080 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1204099e50 con 0x563cee596080 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 10 client.26374 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 10 client.26374 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 20 client.26374 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee60b020 0x563cee60d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 4 client.26374 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563cee500b50 con 0x563cee60b020 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.784+0000 7f121b400640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee60b020 0x563cee60d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1208c00640 10 client.26374 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f121b400640 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee60b020 0x563cee60d440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f3eac1fa2322cc99 server_cookie=5f351e8fb3ffede2 in_seq=0 out_seq=0 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1208c00640 1 -- 172.21.15.119:0/1921536969 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563cee500b50 con 0x563cee60b020 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1208c00640 10 client.26374 handle_command_reply: tid=0 2024-03-28T18:47:50.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 client.26374 shutdown 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 msgr2=0x563cee5a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 0x563cee5a2f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1204097960 tx=0x7f12040cc630 comp rx=0 tx=0).stop 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 shutdown_connections 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x563cee60b020 0x563cee60d440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563cee59ec00 0x563cee6091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1208c00640 0 client.26374 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563cee596080 0x563cee5a2f40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 --2- 172.21.15.119:0/1921536969 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563cee5956d0 0x563cee5a2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 >> 172.21.15.119:0/1921536969 conn(0x563cee58fda0 msgr2=0x563cee596460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 shutdown_connections 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.785+0000 7f1224c0d740 1 -- 172.21.15.119:0/1921536969 wait complete. 2024-03-28T18:47:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1224c0d740 20 client.26374 trim_cache size 0 max 16384 2024-03-28T18:47:50.791 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 msgr2=0x7f12000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 0x7f12000a4eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x563cee590e10 tx=0x7f120c005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 msgr2=0x7f121c19ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1204005fd0 tx=0x7f120400dde0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 shutdown_connections 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f12000a29f0 0x7f12000a4eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f121c1a2dc0 0x7f121c1a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f121c10cf80 0x7f121c19ea90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 --2- 172.21.15.119:0/1949432933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f121c107eb0 0x7f121c19e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 >> 172.21.15.119:0/1949432933 conn(0x7f121c075be0 msgr2=0x7f121c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 shutdown_connections 2024-03-28T18:47:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:50.786+0000 7f1222000640 1 -- 172.21.15.119:0/1949432933 wait complete. 2024-03-28T18:47:50.801 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:50 smithi119 ceph-mon[31813]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.8 KiB/s wr, 227 op/s 2024-03-28T18:47:50.805 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:50.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:47:50.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:50 smithi196 ceph-mon[36300]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.8 KiB/s wr, 227 op/s 2024-03-28T18:47:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:50 smithi131 ceph-mon[36065]: pgmap v116: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.8 KiB/s wr, 227 op/s 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.012+0000 7fdb58200640 1 -- 172.21.15.119:0/2373317648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50112640 msgr2=0x7fdb50114a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.012+0000 7fdb58200640 1 --2- 172.21.15.119:0/2373317648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50112640 0x7fdb50114a30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009900 tx=0x7fdb4802bcc0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.012+0000 7fdb58200640 1 -- 172.21.15.119:0/2373317648 shutdown_connections 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.012+0000 7fdb58200640 1 --2- 172.21.15.119:0/2373317648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50112640 0x7fdb50114a30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.012+0000 7fdb58200640 1 --2- 172.21.15.119:0/2373317648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb50112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.013+0000 7fdb58200640 1 --2- 172.21.15.119:0/2373317648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50073a40 0x7fdb50073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.013+0000 7fdb58200640 1 -- 172.21.15.119:0/2373317648 >> 172.21.15.119:0/2373317648 conn(0x7fdb50078aa0 msgr2=0x7fdb50078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.013+0000 7fdb58200640 1 -- 172.21.15.119:0/2373317648 shutdown_connections 2024-03-28T18:47:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.013+0000 7fdb58200640 1 -- 172.21.15.119:0/2373317648 wait complete. 2024-03-28T18:47:51.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.013+0000 7fdb58200640 1 Processor -- start 2024-03-28T18:47:51.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 -- start start 2024-03-28T18:47:51.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb5019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50112640 0x7fdb501a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50118090 con 0x7fdb50074360 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50117f10 con 0x7fdb50112640 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb58200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50118210 con 0x7fdb50073a40 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb55800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb55800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46534/0 (socket says 172.21.15.119:46534) 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb55800640 1 -- 172.21.15.119:0/3242107492 learned_addr learned my addr 172.21.15.119:0/3242107492 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb54e00640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb5019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.014+0000 7fdb56200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50112640 0x7fdb501a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 -- 172.21.15.119:0/3242107492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50112640 msgr2=0x7fdb501a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50112640 0x7fdb501a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 -- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 msgr2=0x7fdb5019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb5019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 -- 172.21.15.119:0/3242107492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb501a3540 con 0x7fdb50073a40 2024-03-28T18:47:51.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb54e00640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb5019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb55800640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400e9f0 tx=0x7fdb4400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb4e000640 1 -- 172.21.15.119:0/3242107492 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4400cd80 con 0x7fdb50073a40 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb4e000640 1 -- 172.21.15.119:0/3242107492 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb44005500 con 0x7fdb50073a40 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb4e000640 1 -- 172.21.15.119:0/3242107492 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb44010690 con 0x7fdb50073a40 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb501a3830 con 0x7fdb50073a40 2024-03-28T18:47:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.015+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb501ab0c0 con 0x7fdb50073a40 2024-03-28T18:47:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.018+0000 7fdb5ae79740 1 Processor -- start 2024-03-28T18:47:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.018+0000 7fdb4e000640 1 -- 172.21.15.119:0/3242107492 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fdb440046c0 con 0x7fdb50073a40 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.018+0000 7fdb5ae79740 1 -- start start 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d0cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d15630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 0x55e827d18050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827c702c0 con 0x55e827d0c6d0 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827c18c90 con 0x55e827d15c00 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827c6c360 con 0x55e827d0d080 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb56200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d0cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb55800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d15630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb56200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d0cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36690/0 (socket says 172.21.15.119:36690) 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 0x55e827d18050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb56200640 1 -- 172.21.15.119:0/901466595 learned_addr learned my addr 172.21.15.119:0/901466595 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 -- 172.21.15.119:0/901466595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 msgr2=0x55e827d15630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d15630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 -- 172.21.15.119:0/901466595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 msgr2=0x55e827d0cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d0cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.019+0000 7fdb54e00640 1 -- 172.21.15.119:0/901466595 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e827c3d080 con 0x55e827d15c00 2024-03-28T18:47:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb54e00640 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 0x55e827d18050 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c005fd0 tx=0x7fdb3c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2419bf7d9ca65d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb4e000640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 0x7fdb380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb42a00640 1 -- 172.21.15.119:0/901466595 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c003b70 con 0x55e827d15c00 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb42a00640 1 -- 172.21.15.119:0/901466595 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c003d10 con 0x55e827d15c00 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb42a00640 1 -- 172.21.15.119:0/901466595 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c011a90 con 0x55e827d15c00 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb4e000640 1 -- 172.21.15.119:0/3242107492 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdb440cf360 con 0x7fdb50073a40 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.020+0000 7fdb56200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 0x7fdb380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- 172.21.15.119:0/901466595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 msgr2=0x55e827d18050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 0x55e827d18050 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c005fd0 tx=0x7fdb3c009f90 comp rx=0 tx=0).stop 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- 172.21.15.119:0/901466595 shutdown_connections 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d15c00 0x55e827d18050 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d15630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/901466595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d0cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- 172.21.15.119:0/901466595 >> 172.21.15.119:0/901466595 conn(0x55e827d06da0 msgr2=0x55e827d07180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- 172.21.15.119:0/901466595 shutdown_connections 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb56200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 0x7fdb380a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb480096c0 tx=0x7fdb4802e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- 172.21.15.119:0/901466595 wait complete. 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.021+0000 7fdb5ae79740 1 -- start start 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 Processor -- start 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d12890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d12e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827d18b70 con 0x55e827d15c00 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827c2c0d0 con 0x55e827d0d080 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e827d18cf0 con 0x55e827d0c6d0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb5ae79740 10 client.0 _ll_register_callbacks cb 0x55e827c30e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb55800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb55800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46548/0 (socket says 172.21.15.119:46548) 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb54e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d12890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb55800640 1 -- 172.21.15.119:0/1446968628 learned_addr learned my addr 172.21.15.119:0/1446968628 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.022+0000 7fdb56200640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d12e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 -- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 msgr2=0x55e827d12890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d12890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 -- 172.21.15.119:0/1446968628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 msgr2=0x55e827d12e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d12e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 -- 172.21.15.119:0/1446968628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8279843f0 con 0x55e827d0c6d0 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d12890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb56200640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d12e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb55800640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c6a0 tx=0x7fdb440056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fbf9801182bcbb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb37400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4409b4a0 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdb440862f0 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb3c003d10 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb4409b2c0 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb5ae79740 10 client.26386 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.023+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e827a4e7d0 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c003d10 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdb440a3c70 con 0x55e827d0c6d0 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 10 client.26386 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 10 client.26386 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 20 client.26386 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d25a10 0x55e827d27e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 4 client.26386 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e827c7d800 con 0x55e827d25a10 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.024+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d25a10 0x55e827d27e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.025+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d25a10 0x55e827d27e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e40eec340f329f19 server_cookie=6d33e5bb1f90d150 in_seq=0 out_seq=0 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.025+0000 7fdb37400640 10 client.26386 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.025+0000 7fdb37400640 1 -- 172.21.15.119:0/1446968628 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e827c7d800 con 0x55e827d25a10 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.025+0000 7fdb37400640 10 client.26386 handle_command_reply: tid=0 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.025+0000 7fdb5ae79740 1 client.26386 shutdown 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 msgr2=0x55e827d15500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400c6a0 tx=0x7fdb440056b0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 shutdown_connections 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d25a10 0x55e827d27e00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d12e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d12890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1446968628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0c6d0 0x55e827d15500 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb37400640 0 client.26386 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 >> 172.21.15.119:0/1446968628 conn(0x55e827d06da0 msgr2=0x55e827d08af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 shutdown_connections 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1446968628 wait complete. 2024-03-28T18:47:51.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.026+0000 7fdb5ae79740 20 client.26386 trim_cache size 0 max 16384 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.039+0000 7fdb5ae79740 1 Processor -- start 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.039+0000 7fdb5ae79740 1 -- start start 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d08900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d15c00 0x55e827d15500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b3950 con 0x55e827d0c6d0 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b52d0 con 0x55e827d0d080 2024-03-28T18:47:51.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b3fb0 con 0x55e827d15c00 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb55800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d08900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45938/0 (socket says 172.21.15.119:45938) 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb56200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d15c00 0x55e827d15500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb55800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d08900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36700/0 (socket says 172.21.15.119:36700) 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 -- 172.21.15.119:0/1310909616 learned_addr learned my addr 172.21.15.119:0/1310909616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 -- 172.21.15.119:0/1310909616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d15c00 msgr2=0x55e827d15500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d15c00 0x55e827d15500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 -- 172.21.15.119:0/1310909616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 msgr2=0x55e827d08900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d08900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.040+0000 7fdb54e00640 1 -- 172.21.15.119:0/1310909616 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e827d0e470 con 0x55e827d0d080 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb54e00640 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0116f0 tx=0x7fdb3c005c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e619fe12d08aa702 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb42000640 1 -- 172.21.15.119:0/1310909616 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c029020 con 0x55e827d0d080 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb42000640 1 -- 172.21.15.119:0/1310909616 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c009480 con 0x55e827d0d080 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb42000640 1 -- 172.21.15.119:0/1310909616 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c011b90 con 0x55e827d0d080 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1310909616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 msgr2=0x55e827d14fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0116f0 tx=0x7fdb3c005c10 comp rx=0 tx=0).stop 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1310909616 shutdown_connections 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d15c00 0x55e827d15500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0d080 0x55e827d14fc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/1310909616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d0c6d0 0x55e827d08900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.041+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1310909616 >> 172.21.15.119:0/1310909616 conn(0x55e827d06da0 msgr2=0x55e827d081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1310909616 shutdown_connections 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- 172.21.15.119:0/1310909616 wait complete. 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- start start 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 Processor -- start 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d1be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d1c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b4c70 con 0x55e827d15c00 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b4610 con 0x55e827d0c6d0 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.042+0000 7fdb5ae79740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb501b1b10 con 0x55e827d0d080 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb5ae79740 10 client.0 _ll_register_callbacks cb 0x7fdb4400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45946/0 (socket says 172.21.15.119:45946) 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb56200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d1c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 -- 172.21.15.119:0/451388322 learned_addr learned my addr 172.21.15.119:0/451388322 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb54e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d1be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 -- 172.21.15.119:0/451388322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 msgr2=0x55e827d1be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d1be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 -- 172.21.15.119:0/451388322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 msgr2=0x55e827d1c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d1c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 -- 172.21.15.119:0/451388322 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e827d1c9c0 con 0x55e827d0c6d0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb55800640 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdb440a3bd0 tx=0x7fdb4409b0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f1293155ca3dc6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.043+0000 7fdb37400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440b9060 con 0x55e827d0c6d0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdb440976b0 con 0x55e827d0c6d0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb3c009480 con 0x55e827d0c6d0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb4409b4a0 con 0x55e827d0c6d0 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 10 client.26408 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e827d82080 con 0x55e827d0c6d0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c009480 con 0x55e827d0c6d0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdb440978b0 con 0x55e827d0c6d0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 10 client.26408 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 10 client.26408 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 20 client.26408 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d824b0 0x55e827d848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 4 client.26408 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e827c77b50 con 0x55e827d824b0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.044+0000 7fdb54e00640 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d824b0 0x55e827d848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb54e00640 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d824b0 0x55e827d848a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eda0f4d20dd20096 server_cookie=c532f9921675b85 in_seq=0 out_seq=0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb37400640 10 client.26408 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb37400640 1 -- 172.21.15.119:0/451388322 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e827c77b50 con 0x55e827d824b0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb37400640 10 client.26408 handle_command_reply: tid=0 2024-03-28T18:47:51.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 client.26408 shutdown 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 msgr2=0x55e827d1daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdb440a3bd0 tx=0x7fdb4409b0b0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 shutdown_connections 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e827d824b0 0x55e827d848a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e827d15c00 0x55e827d1c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e827d0d080 0x55e827d1be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 --2- 172.21.15.119:0/451388322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e827d0c6d0 0x55e827d1daf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb37400640 0 client.26408 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.045+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 >> 172.21.15.119:0/451388322 conn(0x55e827d06da0 msgr2=0x55e827d0dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 shutdown_connections 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb5ae79740 1 -- 172.21.15.119:0/451388322 wait complete. 2024-03-28T18:47:51.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb5ae79740 20 client.26408 trim_cache size 0 max 16384 2024-03-28T18:47:51.051 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:51.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 msgr2=0x7fdb380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 0x7fdb380a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb480096c0 tx=0x7fdb4802e040 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 msgr2=0x7fdb5019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400e9f0 tx=0x7fdb4400edf0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 shutdown_connections 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdb380a2970 0x7fdb380a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdb50112640 0x7fdb501a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50074360 0x7fdb5019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 --2- 172.21.15.119:0/3242107492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdb50073a40 0x7fdb5019e550 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 >> 172.21.15.119:0/3242107492 conn(0x7fdb50078aa0 msgr2=0x7fdb5010e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.046+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 shutdown_connections 2024-03-28T18:47:51.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.047+0000 7fdb58200640 1 -- 172.21.15.119:0/3242107492 wait complete. 2024-03-28T18:47:51.063 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:51.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:47:51.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 -- 172.21.15.119:0/2406569396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c108860 msgr2=0x7fad4c10c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 --2- 172.21.15.119:0/2406569396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c108860 0x7fad4c10c8a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fad34009900 tx=0x7fad3402bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 -- 172.21.15.119:0/2406569396 shutdown_connections 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 --2- 172.21.15.119:0/2406569396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c10cf80 0x7fad4c114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 --2- 172.21.15.119:0/2406569396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c108860 0x7fad4c10c8a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 --2- 172.21.15.119:0/2406569396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 -- 172.21.15.119:0/2406569396 >> 172.21.15.119:0/2406569396 conn(0x7fad4c075be0 msgr2=0x7fad4c078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 -- 172.21.15.119:0/2406569396 shutdown_connections 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.266+0000 7fad51800640 1 -- 172.21.15.119:0/2406569396 wait complete. 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.267+0000 7fad51800640 1 Processor -- start 2024-03-28T18:47:51.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 -- start start 2024-03-28T18:47:51.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c10cf80 0x7fad4c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c1a2e40 0x7fad4c1a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c118080 con 0x7fad4c1a2e40 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c117f00 con 0x7fad4c10cf80 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.268+0000 7fad51800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c118200 con 0x7fad4c107eb0 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c10cf80 0x7fad4c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46590/0 (socket says 172.21.15.119:46590) 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 -- 172.21.15.119:0/2777197348 learned_addr learned my addr 172.21.15.119:0/2777197348 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4b400640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c1a2e40 0x7fad4c1a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 -- 172.21.15.119:0/2777197348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c10cf80 msgr2=0x7fad4c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c10cf80 0x7fad4c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 -- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c1a2e40 msgr2=0x7fad4c1a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c1a2e40 0x7fad4c1a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 -- 172.21.15.119:0/2777197348 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c1a3930 con 0x7fad4c107eb0 2024-03-28T18:47:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e9f0 tx=0x7fad3c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.269+0000 7fad43400640 1 -- 172.21.15.119:0/2777197348 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00cd80 con 0x7fad4c107eb0 2024-03-28T18:47:51.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.270+0000 7fad43400640 1 -- 172.21.15.119:0/2777197348 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad3c005500 con 0x7fad4c107eb0 2024-03-28T18:47:51.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.270+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c1a3c20 con 0x7fad4c107eb0 2024-03-28T18:47:51.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.270+0000 7fad43400640 1 -- 172.21.15.119:0/2777197348 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c010690 con 0x7fad4c107eb0 2024-03-28T18:47:51.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.270+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c1ab390 con 0x7fad4c107eb0 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad43400640 1 -- 172.21.15.119:0/2777197348 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fad3c0046c0 con 0x7fad4c107eb0 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 Processor -- start 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 -- start start 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd90630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd93050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bceb2c0 con 0x563b8bd90c00 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bc93c90 con 0x563b8bd88080 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad545cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bce7360 con 0x563b8bd876d0 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.273+0000 7fad4b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd90630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46606/0 (socket says 172.21.15.119:46606) 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 -- 172.21.15.119:0/2201222214 learned_addr learned my addr 172.21.15.119:0/2201222214 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad43e00640 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd93050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 -- 172.21.15.119:0/2201222214 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 msgr2=0x563b8bd90630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd90630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 -- 172.21.15.119:0/2201222214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 msgr2=0x563b8bd93050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd93050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 -- 172.21.15.119:0/2201222214 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b8bcb8080 con 0x563b8bd876d0 2024-03-28T18:47:51.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd90630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad4b400640 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fad44009f20 tx=0x7fad4400f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a65244962ce7679 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.274+0000 7fad43400640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 0x7fad300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad3be00640 1 -- 172.21.15.119:0/2201222214 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44015070 con 0x563b8bd876d0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad43400640 1 -- 172.21.15.119:0/2777197348 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fad3c0cf3d0 con 0x7fad4c107eb0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad3be00640 1 -- 172.21.15.119:0/2201222214 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad4401ab40 con 0x563b8bd876d0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad3be00640 1 -- 172.21.15.119:0/2201222214 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44011ca0 con 0x563b8bd876d0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 0x7fad300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad545cb740 1 -- 172.21.15.119:0/2201222214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 msgr2=0x563b8bd87ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad4aa00640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 0x7fad300a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x563b8bd871a0 tx=0x7fad3c0d1820 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.275+0000 7fad545cb740 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fad44009f20 tx=0x7fad4400f7f0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 -- 172.21.15.119:0/2201222214 shutdown_connections 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd93050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd90630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 --2- 172.21.15.119:0/2201222214 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd87ab0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 -- 172.21.15.119:0/2201222214 >> 172.21.15.119:0/2201222214 conn(0x563b8bd81da0 msgr2=0x563b8bd82180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 -- 172.21.15.119:0/2201222214 shutdown_connections 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 -- 172.21.15.119:0/2201222214 wait complete. 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 -- start start 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.276+0000 7fad545cb740 1 Processor -- start 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bd93b70 con 0x563b8bd90c00 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bca70d0 con 0x563b8bd88080 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8bd93cf0 con 0x563b8bd876d0 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad545cb740 10 client.0 _ll_register_callbacks cb 0x563b8bcabe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45982/0 (socket says 172.21.15.119:45982) 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 -- 172.21.15.119:0/1474194568 learned_addr learned my addr 172.21.15.119:0/1474194568 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad4b400640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad4b400640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 -- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 msgr2=0x563b8bd904f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd904f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 -- 172.21.15.119:0/1474194568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 msgr2=0x563b8bd8de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad43e00640 1 -- 172.21.15.119:0/1474194568 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b8b9ff3f0 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.277+0000 7fad4b400640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad43e00640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad34005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9f04334f1fee33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad38c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3402cb10 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad4401ab40 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fad3402ccb0 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad340046c0 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad545cb740 10 client.26420 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563b8bac97d0 con 0x563b8bd88080 2024-03-28T18:47:51.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.278+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad4401ab40 con 0x563b8bd88080 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fad340035f0 con 0x563b8bd88080 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 10 client.26420 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 10 client.26420 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 20 client.26420 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bda0a00 0x563b8bda2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 4 client.26420 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b8bcf8800 con 0x563b8bda0a00 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.279+0000 7fad4b400640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bda0a00 0x563b8bda2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.280+0000 7fad38c00640 10 client.26420 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.280+0000 7fad4b400640 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bda0a00 0x563b8bda2df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c874f69fcc0918e server_cookie=f32af82a476d475a in_seq=0 out_seq=0 2024-03-28T18:47:51.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.280+0000 7fad38c00640 1 -- 172.21.15.119:0/1474194568 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563b8bcf8800 con 0x563b8bda0a00 2024-03-28T18:47:51.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.280+0000 7fad38c00640 10 client.26420 handle_command_reply: tid=0 2024-03-28T18:47:51.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.280+0000 7fad545cb740 1 client.26420 shutdown 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 msgr2=0x563b8bd8d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad34005e70 comp rx=0 tx=0).stop 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 shutdown_connections 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bda0a00 0x563b8bda2df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd8d880 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad38c00640 0 client.26420 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 --2- 172.21.15.119:0/1474194568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd904f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 >> 172.21.15.119:0/1474194568 conn(0x563b8bd81da0 msgr2=0x563b8bd83af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 shutdown_connections 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 1 -- 172.21.15.119:0/1474194568 wait complete. 2024-03-28T18:47:51.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.281+0000 7fad545cb740 20 client.26420 trim_cache size 0 max 16384 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 Processor -- start 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 -- start start 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd876d0 0x563b8bd8ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b3960 con 0x563b8bd90c00 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b52e0 con 0x563b8bd876d0 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad545cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b3fc0 con 0x563b8bd88080 2024-03-28T18:47:51.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad43e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad4b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd876d0 0x563b8bd8ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad43e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46634/0 (socket says 172.21.15.119:46634) 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad43e00640 1 -- 172.21.15.119:0/2143566396 learned_addr learned my addr 172.21.15.119:0/2143566396 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.295+0000 7fad4b400640 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 -- 172.21.15.119:0/2143566396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd876d0 msgr2=0x563b8bd8ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd876d0 0x563b8bd8ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 -- 172.21.15.119:0/2143566396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 msgr2=0x563b8bd8ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 -- 172.21.15.119:0/2143566396 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b8bda3160 con 0x563b8bd88080 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad4b400640 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad43e00640 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fad3402c170 tx=0x7fad34005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1e7f6808035818 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad3b400640 1 -- 172.21.15.119:0/2143566396 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3402cb10 con 0x563b8bd88080 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad3b400640 1 -- 172.21.15.119:0/2143566396 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad3402ccb0 con 0x563b8bd88080 2024-03-28T18:47:51.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.296+0000 7fad3b400640 1 -- 172.21.15.119:0/2143566396 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34005120 con 0x563b8bd88080 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 -- 172.21.15.119:0/2143566396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 msgr2=0x563b8bd904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fad3402c170 tx=0x7fad34005c00 comp rx=0 tx=0).stop 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 -- 172.21.15.119:0/2143566396 shutdown_connections 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd8ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd88080 0x563b8bd904f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 --2- 172.21.15.119:0/2143566396 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd876d0 0x563b8bd8ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 -- 172.21.15.119:0/2143566396 >> 172.21.15.119:0/2143566396 conn(0x563b8bd81da0 msgr2=0x563b8bd80eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 -- 172.21.15.119:0/2143566396 shutdown_connections 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.297+0000 7fad545cb740 1 -- 172.21.15.119:0/2143566396 wait complete. 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 -- start start 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 Processor -- start 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd96ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b4c80 con 0x563b8bd90c00 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b4620 con 0x563b8bd88080 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c1b1b20 con 0x563b8bd876d0 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad545cb740 10 client.0 _ll_register_callbacks cb 0x7fad34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.298+0000 7fad43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:45998/0 (socket says 172.21.15.119:45998) 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 -- 172.21.15.119:0/1685378552 learned_addr learned my addr 172.21.15.119:0/1685378552 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad4b400640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad4b400640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd96ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 -- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 msgr2=0x563b8bd988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 -- 172.21.15.119:0/1685378552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 msgr2=0x563b8bd96ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd96ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 -- 172.21.15.119:0/1685378552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b8bd976e0 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad4b400640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad43e00640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fad3402c820 tx=0x7fad34003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7162f1dc688c1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad38c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34004c70 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.299+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fad34004ee0 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad3402ccb0 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad340050e0 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 10 client.26426 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563b8bdfcf30 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad3402ccb0 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fad34059070 con 0x563b8bd88080 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 10 client.26426 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 10 client.26426 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:51.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 20 client.26426 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bdfd4c0 0x563b8bdff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 4 client.26426 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563b8bcf2b50 con 0x563b8bdfd4c0 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.300+0000 7fad4b400640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bdfd4c0 0x563b8bdff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad4b400640 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bdfd4c0 0x563b8bdff8e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c14dcadaaf4e0b2 server_cookie=f5f99f4456abe5bf in_seq=0 out_seq=0 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad38c00640 10 client.26426 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad38c00640 1 -- 172.21.15.119:0/1685378552 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563b8bcf2b50 con 0x563b8bdfd4c0 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad38c00640 10 client.26426 handle_command_reply: tid=0 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 client.26426 shutdown 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 msgr2=0x563b8bd98e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fad3402c820 tx=0x7fad34003e30 comp rx=0 tx=0).stop 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 shutdown_connections 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x563b8bdfd4c0 0x563b8bdff8e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563b8bd90c00 0x563b8bd96ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563b8bd88080 0x563b8bd98e30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 --2- 172.21.15.119:0/1685378552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563b8bd876d0 0x563b8bd988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad38c00640 0 client.26426 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.301+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 >> 172.21.15.119:0/1685378552 conn(0x563b8bd81da0 msgr2=0x563b8bd83030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 shutdown_connections 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad545cb740 1 -- 172.21.15.119:0/1685378552 wait complete. 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad545cb740 20 client.26426 trim_cache size 0 max 16384 2024-03-28T18:47:51.306 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 msgr2=0x7fad300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 0x7fad300a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x563b8bd871a0 tx=0x7fad3c0d1820 comp rx=0 tx=0).stop 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 msgr2=0x7fad4c19e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00e9f0 tx=0x7fad3c00edf0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 shutdown_connections 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fad300a2970 0x7fad300a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4c1a2e40 0x7fad4c1a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fad4c10cf80 0x7fad4c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 --2- 172.21.15.119:0/2777197348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad4c107eb0 0x7fad4c19e580 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.302+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 >> 172.21.15.119:0/2777197348 conn(0x7fad4c075be0 msgr2=0x7fad4c077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.303+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 shutdown_connections 2024-03-28T18:47:51.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.303+0000 7fad51800640 1 -- 172.21.15.119:0/2777197348 wait complete. 2024-03-28T18:47:51.319 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:51.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:47:51.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 -- 172.21.15.119:0/1266707813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 msgr2=0x7f6b70103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1266707813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b70103cc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009900 tx=0x7f6b5802bc50 comp rx=0 tx=0).stop 2024-03-28T18:47:51.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 -- 172.21.15.119:0/1266707813 shutdown_connections 2024-03-28T18:47:51.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1266707813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b70110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1266707813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b70103cc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1266707813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 0x7f6b70109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 -- 172.21.15.119:0/1266707813 >> 172.21.15.119:0/1266707813 conn(0x7f6b700ff2f0 msgr2=0x7f6b70101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 -- 172.21.15.119:0/1266707813 shutdown_connections 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.532+0000 7f6b75a00640 1 -- 172.21.15.119:0/1266707813 wait complete. 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.533+0000 7f6b75a00640 1 Processor -- start 2024-03-28T18:47:51.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 -- start start 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b7019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b7019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 0x7f6b7019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70113cb0 con 0x7f6b70109850 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70113b30 con 0x7f6b70103840 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b75a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70113e30 con 0x7f6b70104200 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b7019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b6ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b7019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b7019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46668/0 (socket says 172.21.15.119:46668) 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b67e00640 1 -- 172.21.15.119:0/1463895223 learned_addr learned my addr 172.21.15.119:0/1463895223 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.534+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 0x7f6b7019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 -- 172.21.15.119:0/1463895223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 msgr2=0x7f6b7019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b7019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 -- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 msgr2=0x7f6b7019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 0x7f6b7019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 -- 172.21.15.119:0/1463895223 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7019f170 con 0x7f6b70103840 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b7019a1d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000e9f0 tx=0x7f6b6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b67400640 1 -- 172.21.15.119:0/1463895223 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6000ccc0 con 0x7f6b70103840 2024-03-28T18:47:51.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7019f460 con 0x7f6b70103840 2024-03-28T18:47:51.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b67400640 1 -- 172.21.15.119:0/1463895223 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b60010910 con 0x7f6b70103840 2024-03-28T18:47:51.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.535+0000 7f6b67400640 1 -- 172.21.15.119:0/1463895223 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60018960 con 0x7f6b70103840 2024-03-28T18:47:51.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.536+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b701a6cf0 con 0x7f6b70103840 2024-03-28T18:47:51.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.538+0000 7f6b67400640 1 -- 172.21.15.119:0/1463895223 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6b60010ac0 con 0x7f6b70103840 2024-03-28T18:47:51.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 Processor -- start 2024-03-28T18:47:51.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 -- start start 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b440477050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4403cf2c0 con 0x55b440474c00 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b440377c90 con 0x55b44046b6d0 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b78829740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4403cb360 con 0x55b44046c080 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b6f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b6f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46030/0 (socket says 172.21.15.119:46030) 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b6f400640 1 -- 172.21.15.119:0/194224411 learned_addr learned my addr 172.21.15.119:0/194224411 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.539+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b67e00640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b440477050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 -- 172.21.15.119:0/194224411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 msgr2=0x55b440474630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 -- 172.21.15.119:0/194224411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 msgr2=0x55b440477050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b440477050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 -- 172.21.15.119:0/194224411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b44039c080 con 0x55b44046b6d0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b67e00640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b440477050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b6f400640 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009fd0 tx=0x7f6b680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df46b84a3ac2da8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b67400640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 0x7f6b540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b5fe00640 1 -- 172.21.15.119:0/194224411 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68004d20 con 0x55b44046b6d0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b5fe00640 1 -- 172.21.15.119:0/194224411 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b68004050 con 0x55b44046b6d0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.540+0000 7f6b5fe00640 1 -- 172.21.15.119:0/194224411 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68005260 con 0x55b44046b6d0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 0x7f6b540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b67400640 1 -- 172.21.15.119:0/1463895223 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b600cf240 con 0x7f6b70103840 2024-03-28T18:47:51.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 -- 172.21.15.119:0/194224411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 msgr2=0x55b44046bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009fd0 tx=0x7f6b680118d0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 -- 172.21.15.119:0/194224411 shutdown_connections 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b440477050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 --2- 172.21.15.119:0/194224411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b44046bab0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 -- 172.21.15.119:0/194224411 >> 172.21.15.119:0/194224411 conn(0x55b440465da0 msgr2=0x55b440466180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b6ea00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 0x7f6b540a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55b44046b1a0 tx=0x7f6b600d2cf0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 -- 172.21.15.119:0/194224411 shutdown_connections 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.541+0000 7f6b78829740 1 -- 172.21.15.119:0/194224411 wait complete. 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.542+0000 7f6b78829740 1 -- start start 2024-03-28T18:47:51.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.542+0000 7f6b78829740 1 Processor -- start 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046b6d0 0x55b440474560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b440474c00 0x55b440471f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b440477b70 con 0x55b44046b6d0 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44038b0d0 con 0x55b44046c080 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b440477cf0 con 0x55b440474c00 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b78829740 10 client.0 _ll_register_callbacks cb 0x55b44038fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b6f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046b6d0 0x55b440474560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46038/0 (socket says 172.21.15.119:46038) 2024-03-28T18:47:51.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 -- 172.21.15.119:0/1481489511 learned_addr learned my addr 172.21.15.119:0/1481489511 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b440474c00 0x55b440471f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 -- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b440474c00 msgr2=0x55b440471f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b440474c00 0x55b440471f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 -- 172.21.15.119:0/1481489511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046b6d0 msgr2=0x55b440474560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046b6d0 0x55b440474560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.543+0000 7f6b67e00640 1 -- 172.21.15.119:0/1481489511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4400e33f0 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b5802c260 tx=0x7f6b58005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aee851fd7123887 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5802cad0 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b5802cc70 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b68004050 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b580046c0 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b78829740 10 client.26438 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.544+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b4401ad7d0 con 0x55b44046c080 2024-03-28T18:47:51.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b68004050 con 0x55b44046c080 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6b580035f0 con 0x55b44046c080 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 10 client.26438 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 10 client.26438 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 20 client.26438 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b440484ac0 0x55b440486eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 4 client.26438 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4403dc800 con 0x55b440484ac0 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.545+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b440484ac0 0x55b440486eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.546+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b440484ac0 0x55b440486eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1cf41e5eca1bc3d0 server_cookie=107ea74814368562 in_seq=0 out_seq=0 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.546+0000 7f6b5cc00640 10 client.26438 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.546+0000 7f6b5cc00640 1 -- 172.21.15.119:0/1481489511 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4403dc800 con 0x55b440484ac0 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.546+0000 7f6b5cc00640 10 client.26438 handle_command_reply: tid=0 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.546+0000 7f6b78829740 1 client.26438 shutdown 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 msgr2=0x55b4404718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b5802c260 tx=0x7f6b58005b50 comp rx=0 tx=0).stop 2024-03-28T18:47:51.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 shutdown_connections 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b440484ac0 0x55b440486eb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b440474c00 0x55b440471f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046c080 0x55b4404718f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b5cc00640 0 client.26438 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 --2- 172.21.15.119:0/1481489511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046b6d0 0x55b440474560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 >> 172.21.15.119:0/1481489511 conn(0x55b440465da0 msgr2=0x55b440467af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 shutdown_connections 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 1 -- 172.21.15.119:0/1481489511 wait complete. 2024-03-28T18:47:51.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.547+0000 7f6b78829740 20 client.26438 trim_cache size 0 max 16384 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.560+0000 7f6b78829740 1 Processor -- start 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 -- start start 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b440474020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b44046eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701af5f0 con 0x55b440474c00 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701b0f70 con 0x55b44046b6d0 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b78829740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701afc50 con 0x55b44046c080 2024-03-28T18:47:51.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46704/0 (socket says 172.21.15.119:46704) 2024-03-28T18:47:51.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b67e00640 1 -- 172.21.15.119:0/1523087139 learned_addr learned my addr 172.21.15.119:0/1523087139 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b440474020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.561+0000 7f6b6f400640 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b44046eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 -- 172.21.15.119:0/1523087139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 msgr2=0x55b440474020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b440474020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 -- 172.21.15.119:0/1523087139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 msgr2=0x55b44046eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b44046eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 -- 172.21.15.119:0/1523087139 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b440477a50 con 0x55b44046c080 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b67e00640 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000c00 tx=0x7f6b58009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb12d002940c261 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b5f400640 1 -- 172.21.15.119:0/1523087139 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58005850 con 0x55b44046c080 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b5f400640 1 -- 172.21.15.119:0/1523087139 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b58003bb0 con 0x55b44046c080 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.562+0000 7f6b5f400640 1 -- 172.21.15.119:0/1523087139 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b580509b0 con 0x55b44046c080 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- 172.21.15.119:0/1523087139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 msgr2=0x55b440474560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000c00 tx=0x7f6b58009800 comp rx=0 tx=0).stop 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- 172.21.15.119:0/1523087139 shutdown_connections 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b440474c00 0x55b44046eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046c080 0x55b440474560 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 --2- 172.21.15.119:0/1523087139 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b44046b6d0 0x55b440474020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- 172.21.15.119:0/1523087139 >> 172.21.15.119:0/1523087139 conn(0x55b440465da0 msgr2=0x55b440464eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- 172.21.15.119:0/1523087139 shutdown_connections 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- 172.21.15.119:0/1523087139 wait complete. 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 -- start start 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.563+0000 7f6b78829740 1 Processor -- start 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046b6d0 0x55b44047aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b440474c00 0x55b4404e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701b0910 con 0x55b44046c080 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701b02b0 con 0x55b440474c00 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b701ad7b0 con 0x55b44046b6d0 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b78829740 10 client.0 _ll_register_callbacks cb 0x7f6b680093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b6f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046b6d0 0x55b44047aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36812/0 (socket says 172.21.15.119:36812) 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 -- 172.21.15.119:0/3901401940 learned_addr learned my addr 172.21.15.119:0/3901401940 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b6f400640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b440474c00 0x55b4404e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 -- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046b6d0 msgr2=0x55b44047aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046b6d0 0x55b44047aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 -- 172.21.15.119:0/3901401940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b440474c00 msgr2=0x55b4404e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b440474c00 0x55b4404e1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.564+0000 7f6b67e00640 1 -- 172.21.15.119:0/3901401940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4404e2120 con 0x55b44046c080 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b67e00640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b580039c0 tx=0x7f6b58005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff18afd65f216f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5802cce0 con 0x55b44046c080 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b58004860 con 0x55b44046c080 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b58003bb0 con 0x55b44046c080 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b58004a60 con 0x55b44046c080 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b78829740 10 client.16224 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.565+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b4404e94a0 con 0x55b44046c080 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b58003bb0 con 0x55b44046c080 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6b58005040 con 0x55b44046c080 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 10 client.16224 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 10 client.16224 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 20 client.16224 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b4404e9b20 0x55b4404ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 4 client.16224 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4403d6b50 con 0x55b4404e9b20 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.566+0000 7f6b6f400640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b4404e9b20 0x55b4404ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b6f400640 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b4404e9b20 0x55b4404ebf40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=842ca19c8b9c1fdc server_cookie=a50aab3f2ab973b1 in_seq=0 out_seq=0 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b5cc00640 10 client.16224 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b5cc00640 1 -- 172.21.15.119:0/3901401940 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4403d6b50 con 0x55b4404e9b20 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b5cc00640 10 client.16224 handle_command_reply: tid=0 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 client.16224 shutdown 2024-03-28T18:47:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 msgr2=0x55b44047b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b580039c0 tx=0x7f6b58005b50 comp rx=0 tx=0).stop 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 shutdown_connections 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55b4404e9b20 0x55b4404ebf40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b440474c00 0x55b4404e1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b44046c080 0x55b44047b0e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b78829740 1 --2- 172.21.15.119:0/3901401940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b44046b6d0 0x55b44047aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.567+0000 7f6b5cc00640 0 client.16224 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 >> 172.21.15.119:0/3901401940 conn(0x55b440465da0 msgr2=0x55b4404670a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 shutdown_connections 2024-03-28T18:47:51.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b78829740 1 -- 172.21.15.119:0/3901401940 wait complete. 2024-03-28T18:47:51.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:51 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4002136911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:51.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b78829740 20 client.16224 trim_cache size 0 max 16384 2024-03-28T18:47:51.572 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.568+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 msgr2=0x7f6b540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 0x7f6b540a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55b44046b1a0 tx=0x7f6b600d2cf0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 msgr2=0x7f6b7019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b7019a1d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b6000e9f0 tx=0x7f6b6000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 shutdown_connections 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b540a2970 0x7f6b540a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70109850 0x7f6b7019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70104200 0x7f6b7019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 --2- 172.21.15.119:0/1463895223 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70103840 0x7f6b7019a1d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 >> 172.21.15.119:0/1463895223 conn(0x7f6b700ff2f0 msgr2=0x7f6b700ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 shutdown_connections 2024-03-28T18:47:51.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.569+0000 7f6b75a00640 1 -- 172.21.15.119:0/1463895223 wait complete. 2024-03-28T18:47:51.585 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:51.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:47:51.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 -- 172.21.15.119:0/163044028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0104200 msgr2=0x7ff3d0110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 --2- 172.21.15.119:0/163044028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0104200 0x7ff3d0110710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009ca0 tx=0x7ff3c802c100 comp rx=0 tx=0).stop 2024-03-28T18:47:51.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 -- 172.21.15.119:0/163044028 shutdown_connections 2024-03-28T18:47:51.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 --2- 172.21.15.119:0/163044028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0104200 0x7ff3d0110710 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 --2- 172.21.15.119:0/163044028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0103840 0x7ff3d0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 --2- 172.21.15.119:0/163044028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 -- 172.21.15.119:0/163044028 >> 172.21.15.119:0/163044028 conn(0x7ff3d00ff2f0 msgr2=0x7ff3d0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 -- 172.21.15.119:0/163044028 shutdown_connections 2024-03-28T18:47:51.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.776+0000 7ff3d5400640 1 -- 172.21.15.119:0/163044028 wait complete. 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.777+0000 7ff3d5400640 1 Processor -- start 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.777+0000 7ff3d5400640 1 -- start start 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0103840 0x7ff3d019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0104200 0x7ff3d019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0113c20 con 0x7ff3d0104200 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0113aa0 con 0x7ff3d0103840 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3d5400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0113da0 con 0x7ff3d0109850 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0103840 0x7ff3d019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46728/0 (socket says 172.21.15.119:46728) 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0104200 0x7ff3d019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 -- 172.21.15.119:0/1188225686 learned_addr learned my addr 172.21.15.119:0/1188225686 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 -- 172.21.15.119:0/1188225686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0103840 msgr2=0x7ff3d019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0103840 0x7ff3d019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 -- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0104200 msgr2=0x7ff3d019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0104200 0x7ff3d019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.778+0000 7ff3cf200640 1 -- 172.21.15.119:0/1188225686 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d019f100 con 0x7ff3d0109850 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3cf200640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8037040 tx=0x7ff3c8034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3c7400640 1 -- 172.21.15.119:0/1188225686 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8039050 con 0x7ff3d0109850 2024-03-28T18:47:51.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d019f390 con 0x7ff3d0109850 2024-03-28T18:47:51.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3c7400640 1 -- 172.21.15.119:0/1188225686 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c8045070 con 0x7ff3d0109850 2024-03-28T18:47:51.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3c7400640 1 -- 172.21.15.119:0/1188225686 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80045a0 con 0x7ff3d0109850 2024-03-28T18:47:51.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.779+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d01a6c80 con 0x7ff3d0109850 2024-03-28T18:47:51.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.783+0000 7ff3d81ce740 1 Processor -- start 2024-03-28T18:47:51.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.783+0000 7ff3c7400640 1 -- 172.21.15.119:0/1188225686 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ff3c8003970 con 0x7ff3d0109850 2024-03-28T18:47:51.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 -- start start 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af957342c0 con 0x55af957d06d0 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af956dcc90 con 0x55af957d1080 2024-03-28T18:47:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af95730360 con 0x55af957d9c00 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46076/0 (socket says 172.21.15.119:46076) 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3ce800640 1 -- 172.21.15.119:0/815741315 learned_addr learned my addr 172.21.15.119:0/815741315 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3cde00640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.784+0000 7ff3cde00640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 -- 172.21.15.119:0/815741315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 msgr2=0x55af957dc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957dc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 -- 172.21.15.119:0/815741315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 msgr2=0x55af957d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 -- 172.21.15.119:0/815741315 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af95701080 con 0x55af957d1080 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3cde00640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3ce800640 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000bc90 tx=0x7ff3c00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22e1a0490a3e572e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3bb400640 1 -- 172.21.15.119:0/815741315 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c001c480 con 0x55af957d1080 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3c7400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 0x7ff3b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3bb400640 1 -- 172.21.15.119:0/815741315 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c001cb00 con 0x55af957d1080 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3bb400640 1 -- 172.21.15.119:0/815741315 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0011c60 con 0x55af957d1080 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3c7400640 1 -- 172.21.15.119:0/1188225686 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff3c80f13f0 con 0x7ff3d0109850 2024-03-28T18:47:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.785+0000 7ff3cde00640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 0x7ff3b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- 172.21.15.119:0/815741315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 msgr2=0x55af957d9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000bc90 tx=0x7ff3c00107a0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- 172.21.15.119:0/815741315 shutdown_connections 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957dc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d9630 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/815741315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- 172.21.15.119:0/815741315 >> 172.21.15.119:0/815741315 conn(0x55af957cada0 msgr2=0x55af957cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- 172.21.15.119:0/815741315 shutdown_connections 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- 172.21.15.119:0/815741315 wait complete. 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3cde00640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 0x7ff3b40a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0058c0 tx=0x7ff3bc015040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.786+0000 7ff3d81ce740 1 -- start start 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 Processor -- start 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af957dcb70 con 0x55af957d06d0 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af956f00d0 con 0x55af957d1080 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af957dccf0 con 0x55af957d9c00 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3d81ce740 10 client.0 _ll_register_callbacks cb 0x55af956f4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36854/0 (socket says 172.21.15.119:36854) 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3ce800640 1 -- 172.21.15.119:0/3944318759 learned_addr learned my addr 172.21.15.119:0/3944318759 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.787+0000 7ff3cf200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 -- 172.21.15.119:0/3944318759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 msgr2=0x55af957d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957d6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 -- 172.21.15.119:0/3944318759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 msgr2=0x55af957d9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 -- 172.21.15.119:0/3944318759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af954483f0 con 0x55af957d1080 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cf200640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957d6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3ce800640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3cde00640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d6920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc021b70 tx=0x7ff3bc005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60b494c0dc532c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3afe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc019030 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.788+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff3bc011bd0 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3c001cb00 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3bc011dd0 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 10 client.26456 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55af955127d0 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c001cb00 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff3bc02a030 con 0x55af957d1080 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 10 client.26456 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 10 client.26456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 20 client.26456 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af957e9be0 0x55af957ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 4 client.26456 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.789+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af95741800 con 0x55af957e9be0 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.790+0000 7ff3ce800640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af957e9be0 0x55af957ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.790+0000 7ff3ce800640 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af957e9be0 0x55af957ebfd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c423770c582ee725 server_cookie=6ee62f4358baee03 in_seq=0 out_seq=0 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.790+0000 7ff3afe00640 10 client.26456 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3afe00640 1 -- 172.21.15.119:0/3944318759 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55af95741800 con 0x55af957e9be0 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3afe00640 10 client.26456 handle_command_reply: tid=0 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 client.26456 shutdown 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 msgr2=0x55af957d6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d6920 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc021b70 tx=0x7ff3bc005850 comp rx=0 tx=0).stop 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 shutdown_connections 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af957e9be0 0x55af957ebfd0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d9c00 0x55af957d6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3afe00640 0 client.26456 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d1080 0x55af957d6920 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/3944318759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d06d0 0x55af957d9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 >> 172.21.15.119:0/3944318759 conn(0x55af957cada0 msgr2=0x55af957ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 shutdown_connections 2024-03-28T18:47:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.791+0000 7ff3d81ce740 1 -- 172.21.15.119:0/3944318759 wait complete. 2024-03-28T18:47:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.792+0000 7ff3d81ce740 20 client.26456 trim_cache size 0 max 16384 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 Processor -- start 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 -- start start 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 0x55af957d9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01af520 con 0x55af957d9c00 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01b0ea0 con 0x55af957d06d0 2024-03-28T18:47:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.805+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01afb80 con 0x55af957d1080 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46088/0 (socket says 172.21.15.119:46088) 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3cf200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 0x55af957d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 -- 172.21.15.119:0/2529854702 learned_addr learned my addr 172.21.15.119:0/2529854702 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46748/0 (socket says 172.21.15.119:46748) 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 -- 172.21.15.119:0/2529854702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 msgr2=0x55af957c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 -- 172.21.15.119:0/2529854702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 msgr2=0x55af957d9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 0x55af957d9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 -- 172.21.15.119:0/2529854702 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af957d72a0 con 0x55af957d06d0 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3cde00640 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3cf200640 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 0x55af957d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3ce800640 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0010ca0 tx=0x7ff3c0005cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c071c23657471b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.806+0000 7ff3baa00640 1 -- 172.21.15.119:0/2529854702 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0031070 con 0x55af957d06d0 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3baa00640 1 -- 172.21.15.119:0/2529854702 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c000bd80 con 0x55af957d06d0 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3baa00640 1 -- 172.21.15.119:0/2529854702 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0011ac0 con 0x55af957d06d0 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 -- 172.21.15.119:0/2529854702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 msgr2=0x55af957cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0010ca0 tx=0x7ff3c0005cd0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 -- 172.21.15.119:0/2529854702 shutdown_connections 2024-03-28T18:47:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d9c00 0x55af957d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d1080 0x55af957c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/2529854702 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d06d0 0x55af957cca80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 -- 172.21.15.119:0/2529854702 >> 172.21.15.119:0/2529854702 conn(0x55af957cada0 msgr2=0x55af957c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 -- 172.21.15.119:0/2529854702 shutdown_connections 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.807+0000 7ff3d81ce740 1 -- 172.21.15.119:0/2529854702 wait complete. 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 -- start start 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 Processor -- start 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d1080 0x55af957e0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 0x55af9583e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01b0840 con 0x55af957d1080 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01b01e0 con 0x55af957d9c00 2024-03-28T18:47:51.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01ad6e0 con 0x55af957d06d0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3d81ce740 10 client.0 _ll_register_callbacks cb 0x7ff3bc021820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46762/0 (socket says 172.21.15.119:46762) 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.808+0000 7ff3ce800640 1 -- 172.21.15.119:0/1932029257 learned_addr learned my addr 172.21.15.119:0/1932029257 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3cf200640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 0x55af9583e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3cde00640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d1080 0x55af957e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 -- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 msgr2=0x55af9583e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 0x55af9583e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 -- 172.21.15.119:0/1932029257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d1080 msgr2=0x55af957e0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d1080 0x55af957e0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 -- 172.21.15.119:0/1932029257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af9583ea30 con 0x55af957d06d0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3cf200640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 0x55af9583e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3ce800640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff3c002b480 tx=0x7ff3c002b170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23cb2cd7d0ade4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3afe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:51.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0031040 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ff3c0005040 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.809+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3c000bd80 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3c0005260 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 10 client.26446 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55af95845db0 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c000bd80 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff3c002b880 con 0x55af957d06d0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 10 client.26446 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 10 client.26446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 20 client.26446 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af958461e0 0x55af958485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 4 client.26446 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.810+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55af9573bb50 con 0x55af958461e0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3cde00640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af958461e0 0x55af958485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3cde00640 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af958461e0 0x55af958485d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af5cca7eac740522 server_cookie=7b6e0a3cfd0e353 in_seq=0 out_seq=0 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3afe00640 10 client.26446 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3afe00640 1 -- 172.21.15.119:0/1932029257 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55af9573bb50 con 0x55af958461e0 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3afe00640 10 client.26446 handle_command_reply: tid=0 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 client.26446 shutdown 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 msgr2=0x55af957dfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff3c002b480 tx=0x7ff3c002b170 comp rx=0 tx=0).stop 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 shutdown_connections 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55af958461e0 0x55af958485d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55af957d9c00 0x55af9583e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55af957d1080 0x55af957e0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3afe00640 0 client.26446 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 --2- 172.21.15.119:0/1932029257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55af957d06d0 0x55af957dfbd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.811+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 >> 172.21.15.119:0/1932029257 conn(0x55af957cada0 msgr2=0x55af957c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 shutdown_connections 2024-03-28T18:47:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d81ce740 1 -- 172.21.15.119:0/1932029257 wait complete. 2024-03-28T18:47:51.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d81ce740 20 client.26446 trim_cache size 0 max 16384 2024-03-28T18:47:51.817 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 msgr2=0x7ff3b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 0x7ff3b40a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0058c0 tx=0x7ff3bc015040 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 msgr2=0x7ff3d019e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.812+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8037040 tx=0x7ff3c8034670 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 shutdown_connections 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff3b40a2970 0x7ff3b40a4e30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0109850 0x7ff3d019e9d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d0104200 0x7ff3d019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 --2- 172.21.15.119:0/1188225686 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff3d0103840 0x7ff3d019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:51.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 >> 172.21.15.119:0/1188225686 conn(0x7ff3d00ff2f0 msgr2=0x7ff3d00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:51.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 shutdown_connections 2024-03-28T18:47:51.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:51.813+0000 7ff3d5400640 1 -- 172.21.15.119:0/1188225686 wait complete. 2024-03-28T18:47:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:51 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4002136911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:51.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:51 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4002136911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:52 smithi119 ceph-mon[31813]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:52.829 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:47:52.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:47:52.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:47:52.845 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:52 smithi131 ceph-mon[36065]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:52 smithi196 ceph-mon[36300]: pgmap v117: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 -- 172.21.15.119:0/1776749220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 msgr2=0x7f1448103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 --2- 172.21.15.119:0/1776749220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448103cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1434009930 tx=0x7f143402bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 -- 172.21.15.119:0/1776749220 shutdown_connections 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 --2- 172.21.15.119:0/1776749220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 0x7f1448110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 --2- 172.21.15.119:0/1776749220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448103cc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 --2- 172.21.15.119:0/1776749220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 0x7f1448109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 -- 172.21.15.119:0/1776749220 >> 172.21.15.119:0/1776749220 conn(0x7f14480ff2f0 msgr2=0x7f1448101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 -- 172.21.15.119:0/1776749220 shutdown_connections 2024-03-28T18:47:53.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.054+0000 7f1450400640 1 -- 172.21.15.119:0/1776749220 wait complete. 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.055+0000 7f1450400640 1 Processor -- start 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.055+0000 7f1450400640 1 -- start start 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 0x7f144810d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 0x7f144810dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448113cb0 con 0x7f1448104200 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448113b30 con 0x7f1448109850 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f1450400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448113e30 con 0x7f1448103840 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46778/0 (socket says 172.21.15.119:46778) 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 -- 172.21.15.119:0/962270270 learned_addr learned my addr 172.21.15.119:0/962270270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144e400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 0x7f144810dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144d000640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 0x7f144810d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 -- 172.21.15.119:0/962270270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 msgr2=0x7f144810dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 0x7f144810dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 -- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 msgr2=0x7f144810d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 0x7f144810d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.056+0000 7f144da00640 1 -- 172.21.15.119:0/962270270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14481aa990 con 0x7f1448103840 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f144da00640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e9f0 tx=0x7f143c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f1446000640 1 -- 172.21.15.119:0/962270270 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c00cd80 con 0x7f1448103840 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f1446000640 1 -- 172.21.15.119:0/962270270 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f143c005500 con 0x7f1448103840 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f1446000640 1 -- 172.21.15.119:0/962270270 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143c010690 con 0x7f1448103840 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14481aab60 con 0x7f1448103840 2024-03-28T18:47:53.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.057+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14481ab030 con 0x7f1448103840 2024-03-28T18:47:53.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.060+0000 7f14531d7740 1 Processor -- start 2024-03-28T18:47:53.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.060+0000 7f14531d7740 1 -- start start 2024-03-28T18:47:53.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.060+0000 7f1446000640 1 -- 172.21.15.119:0/962270270 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f143c0046c0 con 0x7f1448103840 2024-03-28T18:47:53.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d97040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298d99970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298cf2eb0 con 0x55e298d97580 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298c9ba60 con 0x55e298d8e080 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f14531d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298ceef50 con 0x55e298d8e9a0 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46118/0 (socket says 172.21.15.119:46118) 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f144e400640 1 -- 172.21.15.119:0/1347859759 learned_addr learned my addr 172.21.15.119:0/1347859759 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f144d000640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298d99970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.061+0000 7f144da00640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d97040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 -- 172.21.15.119:0/1347859759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 msgr2=0x55e298d97040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d97040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 -- 172.21.15.119:0/1347859759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 msgr2=0x55e298d99970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298d99970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 -- 172.21.15.119:0/1347859759 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e298acf7e0 con 0x55e298d8e080 2024-03-28T18:47:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144d000640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298d99970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144e400640 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1440010370 tx=0x7f1440010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f594af04128d7132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f1446000640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 0x7f14300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f143aa00640 1 -- 172.21.15.119:0/1347859759 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440016070 con 0x55e298d8e080 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f143aa00640 1 -- 172.21.15.119:0/1347859759 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1440004d20 con 0x55e298d8e080 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f143aa00640 1 -- 172.21.15.119:0/1347859759 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14400053e0 con 0x55e298d8e080 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f1446000640 1 -- 172.21.15.119:0/962270270 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f143c0cf310 con 0x7f1448103840 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.062+0000 7f144da00640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 0x7f14300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- 172.21.15.119:0/1347859759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 msgr2=0x55e298d8e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1440010370 tx=0x7f1440010830 comp rx=0 tx=0).stop 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- 172.21.15.119:0/1347859759 shutdown_connections 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298d99970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d97040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 --2- 172.21.15.119:0/1347859759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d8e460 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- 172.21.15.119:0/1347859759 >> 172.21.15.119:0/1347859759 conn(0x55e298d887b0 msgr2=0x55e298d88b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f144da00640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 0x7f14300a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55e298d8db50 tx=0x7f143c004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- 172.21.15.119:0/1347859759 shutdown_connections 2024-03-28T18:47:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- 172.21.15.119:0/1347859759 wait complete. 2024-03-28T18:47:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.063+0000 7f14531d7740 1 -- start start 2024-03-28T18:47:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 Processor -- start 2024-03-28T18:47:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d94290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d97580 0x55e298d94810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298d9a370 con 0x55e298d8e9a0 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298caeeb0 con 0x55e298d8e080 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e298d9a4f0 con 0x55e298d97580 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f14531d7740 10 client.0 _ll_register_callbacks cb 0x55e298ca3150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f144d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d94290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46132/0 (socket says 172.21.15.119:46132) 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.064+0000 7f144e400640 1 -- 172.21.15.119:0/1968945338 learned_addr learned my addr 172.21.15.119:0/1968945338 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144e400640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d97580 0x55e298d94810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 -- 172.21.15.119:0/1968945338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d97580 msgr2=0x55e298d94810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d97580 0x55e298d94810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 -- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 msgr2=0x55e298d96f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 -- 172.21.15.119:0/1968945338 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e298b2ff70 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144e400640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f144d000640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d94290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1434036040 tx=0x7f1434004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1711a28c31473b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.065+0000 7f142f400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143402cb10 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f143402ccb0 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1440004d20 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14340046c0 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f14531d7740 10 client.16254 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e2989c7170 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1440004d20 con 0x55e298d8e9a0 2024-03-28T18:47:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.066+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f14340035f0 con 0x55e298d8e9a0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 10 client.16254 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 10 client.16254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 20 client.16254 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298da75f0 0x55e298da99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 4 client.16254 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e298d003f0 con 0x55e298da75f0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f144e400640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298da75f0 0x55e298da99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f144e400640 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298da75f0 0x55e298da99e0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fe38687059d5091 server_cookie=78f375ad0b994f51 in_seq=0 out_seq=0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.067+0000 7f142f400640 10 client.16254 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f142f400640 1 -- 172.21.15.119:0/1968945338 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e298d003f0 con 0x55e298da75f0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f142f400640 10 client.16254 handle_command_reply: tid=0 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 client.16254 shutdown 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 msgr2=0x55e298d94290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d94290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1434036040 tx=0x7f1434004610 comp rx=0 tx=0).stop 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 shutdown_connections 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298da75f0 0x55e298da99e0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d97580 0x55e298d94810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d94290 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 --2- 172.21.15.119:0/1968945338 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f142f400640 0 client.16254 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 >> 172.21.15.119:0/1968945338 conn(0x55e298d887b0 msgr2=0x55e298d8a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 shutdown_connections 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.068+0000 7f14531d7740 1 -- 172.21.15.119:0/1968945338 wait complete. 2024-03-28T18:47:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.069+0000 7f14531d7740 20 client.16254 trim_cache size 0 max 16384 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.086+0000 7f14531d7740 1 Processor -- start 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.086+0000 7f14531d7740 1 -- start start 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d969c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d97580 0x55e298d96f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b3990 con 0x55e298d8e9a0 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b5310 con 0x55e298d97580 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f14531d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b3ff0 con 0x55e298d8e080 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46814/0 (socket says 172.21.15.119:46814) 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 -- 172.21.15.119:0/4134816747 learned_addr learned my addr 172.21.15.119:0/4134816747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144d000640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d97580 0x55e298d96f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 -- 172.21.15.119:0/4134816747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d97580 msgr2=0x55e298d96f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d97580 0x55e298d96f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 -- 172.21.15.119:0/4134816747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 msgr2=0x55e298d969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144e400640 1 -- 172.21.15.119:0/4134816747 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e298da9d50 con 0x55e298d8e080 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.087+0000 7f144d000640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f144e400640 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1440010d30 tx=0x7f1440012f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8f02ad83b6f63e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f143a000640 1 -- 172.21.15.119:0/4134816747 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440016040 con 0x55e298d8e080 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f143a000640 1 -- 172.21.15.119:0/4134816747 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1440025040 con 0x55e298d8e080 2024-03-28T18:47:53.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f143a000640 1 -- 172.21.15.119:0/4134816747 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440024040 con 0x55e298d8e080 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f14531d7740 1 -- 172.21.15.119:0/4134816747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 msgr2=0x55e298d8a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.088+0000 7f14531d7740 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1440010d30 tx=0x7f1440012f60 comp rx=0 tx=0).stop 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 -- 172.21.15.119:0/4134816747 shutdown_connections 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d97580 0x55e298d96f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d8e9a0 0x55e298d969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 --2- 172.21.15.119:0/4134816747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e080 0x55e298d8a2d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 -- 172.21.15.119:0/4134816747 >> 172.21.15.119:0/4134816747 conn(0x55e298d887b0 msgr2=0x55e298d89bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 -- 172.21.15.119:0/4134816747 shutdown_connections 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 -- 172.21.15.119:0/4134816747 wait complete. 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 -- start start 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.089+0000 7f14531d7740 1 Processor -- start 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d91170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298e01e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b4cb0 con 0x55e298d97580 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b4650 con 0x55e298d8e080 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14481b1b50 con 0x55e298d8e9a0 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f14531d7740 10 client.0 _ll_register_callbacks cb 0x7f1434009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d91170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144e400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d91170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46160/0 (socket says 172.21.15.119:46160) 2024-03-28T18:47:53.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46816/0 (socket says 172.21.15.119:46816) 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144e400640 1 -- 172.21.15.119:0/2292520887 learned_addr learned my addr 172.21.15.119:0/2292520887 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144e400640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298e01e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 -- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 msgr2=0x55e298d91170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d91170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 -- 172.21.15.119:0/2292520887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 msgr2=0x55e298e01e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298e01e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 -- 172.21.15.119:0/2292520887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e298e02430 con 0x55e298d8e9a0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f144d000640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1434003a40 tx=0x7f143402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ff81b197397ad61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.090+0000 7f142f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1434000e60 con 0x55e298d8e9a0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f1434004860 con 0x55e298d8e9a0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1440025040 con 0x55e298d8e9a0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f143402cd10 con 0x55e298d8e9a0 2024-03-28T18:47:53.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 10 client.26464 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e298e037a0 con 0x55e298d8e9a0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1440025040 con 0x55e298d8e9a0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1434059020 con 0x55e298d8e9a0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 10 client.26464 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 10 client.26464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 20 client.26464 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298e03ea0 0x55e298e062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 4 client.26464 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.091+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e298cfa740 con 0x55e298e03ea0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.092+0000 7f144e400640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298e03ea0 0x55e298e062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.092+0000 7f144e400640 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298e03ea0 0x55e298e062c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=455a3c87b9102367 server_cookie=e54953674ce38e19 in_seq=0 out_seq=0 2024-03-28T18:47:53.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.092+0000 7f142f400640 10 client.26464 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f142f400640 1 -- 172.21.15.119:0/2292520887 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e298cfa740 con 0x55e298e03ea0 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f142f400640 10 client.26464 handle_command_reply: tid=0 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 client.26464 shutdown 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 msgr2=0x55e298d916b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1434003a40 tx=0x7f143402bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:53.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 shutdown_connections 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e298e03ea0 0x55e298e062c0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e298d97580 0x55e298e01e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e298d8e9a0 0x55e298d916b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f142f400640 0 client.26464 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 --2- 172.21.15.119:0/2292520887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e298d8e080 0x55e298d91170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.093+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 >> 172.21.15.119:0/2292520887 conn(0x55e298d887b0 msgr2=0x55e298d8f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 shutdown_connections 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f14531d7740 1 -- 172.21.15.119:0/2292520887 wait complete. 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f14531d7740 20 client.26464 trim_cache size 0 max 16384 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:47:53.097 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "95c0b83c-4232-42f4-b9be-ffc59f539e37", 2024-03-28T18:47:53.098 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:47:53.098 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:53.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 msgr2=0x7f14300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 0x7f14300a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x55e298d8db50 tx=0x7f143c004a40 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 msgr2=0x7f1448110680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e9f0 tx=0x7f143c00edf0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 shutdown_connections 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f14300a2970 0x7f14300a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1448109850 0x7f144810dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1448104200 0x7f144810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 --2- 172.21.15.119:0/962270270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448103840 0x7f1448110680 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 >> 172.21.15.119:0/962270270 conn(0x7f14480ff2f0 msgr2=0x7f14480ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.094+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 shutdown_connections 2024-03-28T18:47:53.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.095+0000 7f1450400640 1 -- 172.21.15.119:0/962270270 wait complete. 2024-03-28T18:47:53.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:47:53.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 -- 172.21.15.119:0/3901564354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 msgr2=0x7fafe8108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 --2- 172.21.15.119:0/3901564354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe8108230 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009900 tx=0x7fafd802bc40 comp rx=0 tx=0).stop 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 -- 172.21.15.119:0/3901564354 shutdown_connections 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 --2- 172.21.15.119:0/3901564354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe810cea0 0x7fafe8114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 --2- 172.21.15.119:0/3901564354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe8108800 0x7fafe810c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 --2- 172.21.15.119:0/3901564354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe8108230 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 -- 172.21.15.119:0/3901564354 >> 172.21.15.119:0/3901564354 conn(0x7fafe80774c0 msgr2=0x7fafe81064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 -- 172.21.15.119:0/3901564354 shutdown_connections 2024-03-28T18:47:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.329+0000 7fafeea00640 1 -- 172.21.15.119:0/3901564354 wait complete. 2024-03-28T18:47:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.330+0000 7fafeea00640 1 Processor -- start 2024-03-28T18:47:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.330+0000 7fafeea00640 1 -- start start 2024-03-28T18:47:53.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe810cea0 0x7fafe819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe81a2cb0 0x7fafe81a3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8117fc0 con 0x7fafe810cea0 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8117e40 con 0x7fafe8107e50 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafeea00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8118140 con 0x7fafe81a2cb0 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46174/0 (socket says 172.21.15.119:46174) 2024-03-28T18:47:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 -- 172.21.15.119:0/3081095392 learned_addr learned my addr 172.21.15.119:0/3081095392 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafee000640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe81a2cb0 0x7fafe81a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafecc00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe810cea0 0x7fafe819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 -- 172.21.15.119:0/3081095392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe81a2cb0 msgr2=0x7fafe81a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe81a2cb0 0x7fafe81a3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 -- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe810cea0 msgr2=0x7fafe819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe810cea0 0x7fafe819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.331+0000 7fafed600640 1 -- 172.21.15.119:0/3081095392 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe81a37a0 con 0x7fafe8107e50 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafed600640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005ab0 tx=0x7fafd80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafe6000640 1 -- 172.21.15.119:0/3081095392 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8041070 con 0x7fafe8107e50 2024-03-28T18:47:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafe6000640 1 -- 172.21.15.119:0/3081095392 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd802cca0 con 0x7fafe8107e50 2024-03-28T18:47:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafe6000640 1 -- 172.21.15.119:0/3081095392 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8003d90 con 0x7fafe8107e50 2024-03-28T18:47:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe81a3a30 con 0x7fafe8107e50 2024-03-28T18:47:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.332+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe81ab1a0 con 0x7fafe8107e50 2024-03-28T18:47:53.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.335+0000 7fafe6000640 1 -- 172.21.15.119:0/3081095392 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fafd8044070 con 0x7fafe8107e50 2024-03-28T18:47:53.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.335+0000 7faff2c60740 1 Processor -- start 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.335+0000 7faff2c60740 1 -- start start 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fe0f0 0x5619071fe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 0x561907209850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561907162250 con 0x5619071fedd0 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56190710aa30 con 0x561907207460 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7faff2c60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56190715e2f0 con 0x5619071fe0f0 2024-03-28T18:47:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafee000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafee000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36954/0 (socket says 172.21.15.119:36954) 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafee000640 1 -- 172.21.15.119:0/2365002061 learned_addr learned my addr 172.21.15.119:0/2365002061 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafecc00640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 0x561907209850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafed600640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fe0f0 0x5619071fe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.336+0000 7fafee000640 1 -- 172.21.15.119:0/2365002061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fe0f0 msgr2=0x5619071fe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafee000640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fe0f0 0x5619071fe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafee000640 1 -- 172.21.15.119:0/2365002061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 msgr2=0x561907209850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafee000640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 0x561907209850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafee000640 1 -- 172.21.15.119:0/2365002061 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619070f5690 con 0x5619071fedd0 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafecc00640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 0x561907209850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafee000640 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009f40 tx=0x7fafe00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a566a302d5181d12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafde600640 1 -- 172.21.15.119:0/2365002061 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0004d20 con 0x5619071fedd0 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafde600640 1 -- 172.21.15.119:0/2365002061 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafe0004050 con 0x5619071fedd0 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafde600640 1 -- 172.21.15.119:0/2365002061 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0005220 con 0x5619071fedd0 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.337+0000 7fafe6000640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 0x7fafcc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7fafed600640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 0x7fafcc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7fafe6000640 1 -- 172.21.15.119:0/3081095392 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fafd80f0750 con 0x7fafe8107e50 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 -- 172.21.15.119:0/2365002061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 msgr2=0x5619071fdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009f40 tx=0x7fafe00118d0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 -- 172.21.15.119:0/2365002061 shutdown_connections 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561907207460 0x561907209850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fe0f0 0x5619071fe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 --2- 172.21.15.119:0/2365002061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619071fdb20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 -- 172.21.15.119:0/2365002061 >> 172.21.15.119:0/2365002061 conn(0x5619071c5200 msgr2=0x5619071f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 -- 172.21.15.119:0/2365002061 shutdown_connections 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7faff2c60740 1 -- 172.21.15.119:0/2365002061 wait complete. 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.338+0000 7fafed600640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 0x7fafcc0a4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5619071ff320 tx=0x7fafd802cf90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 -- start start 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 Processor -- start 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907206b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x5619072043f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.339+0000 7faff2c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56190720a2d0 con 0x561907207460 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7faff2c60740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56190711de80 con 0x5619071fe0f0 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7faff2c60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56190720a450 con 0x5619071fedd0 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7faff2c60740 10 client.0 _ll_register_callbacks cb 0x561907130130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46862/0 (socket says 172.21.15.119:46862) 2024-03-28T18:47:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 -- 172.21.15.119:0/1680646444 learned_addr learned my addr 172.21.15.119:0/1680646444 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafee000640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907206b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafee000640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x5619072043f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 -- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 msgr2=0x561907206b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907206b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 -- 172.21.15.119:0/1680646444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 msgr2=0x5619072043f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x5619072043f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 -- 172.21.15.119:0/1680646444 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561906f407d0 con 0x5619071fedd0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafee000640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907206b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafecc00640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fafd4012730 tx=0x7fafd4012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f08f56e2e2e90f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.340+0000 7fafd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd400f920 con 0x5619071fedd0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafe0004050 con 0x5619071fedd0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fafd400fac0 con 0x5619071fedd0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd400fcc0 con 0x5619071fedd0 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7faff2c60740 10 client.26476 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561906e763f0 con 0x5619071fedd0 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.341+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafe0004050 con 0x5619071fedd0 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fafd401c420 con 0x5619071fedd0 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 10 client.26476 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 10 client.26476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 20 client.26476 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5619072173e0 0x561907219800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 4 client.26476 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56190716f790 con 0x5619072173e0 2024-03-28T18:47:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7fafee000640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5619072173e0 0x561907219800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7fafd2a00640 10 client.26476 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.342+0000 7fafee000640 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5619072173e0 0x561907219800 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aefa6b6eb96332e server_cookie=9f041ffa93911e36 in_seq=0 out_seq=0 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7fafd2a00640 1 -- 172.21.15.119:0/1680646444 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56190716f790 con 0x5619072173e0 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7fafd2a00640 10 client.26476 handle_command_reply: tid=0 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 client.26476 shutdown 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 msgr2=0x561907203eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fafd4012730 tx=0x7fafd4012bf0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 shutdown_connections 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5619072173e0 0x561907219800 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x5619072043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907203eb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7fafd2a00640 0 client.26476 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 --2- 172.21.15.119:0/1680646444 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907206b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.343+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 >> 172.21.15.119:0/1680646444 conn(0x5619071c5200 msgr2=0x5619071fa890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.344+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 shutdown_connections 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.344+0000 7faff2c60740 1 -- 172.21.15.119:0/1680646444 wait complete. 2024-03-28T18:47:53.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.344+0000 7faff2c60740 20 client.26476 trim_cache size 0 max 16384 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 Processor -- start 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 -- start start 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x5619072065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x561907208850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b37a0 con 0x561907207460 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b5120 con 0x5619071fe0f0 2024-03-28T18:47:53.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.361+0000 7faff2c60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b3e00 con 0x5619071fedd0 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46876/0 (socket says 172.21.15.119:46876) 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 -- 172.21.15.119:0/1079965936 learned_addr learned my addr 172.21.15.119:0/1079965936 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafee000640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x5619072065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafee000640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x561907208850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 -- 172.21.15.119:0/1079965936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 msgr2=0x5619072065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x5619072065e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 -- 172.21.15.119:0/1079965936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 msgr2=0x561907208850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x561907208850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 -- 172.21.15.119:0/1079965936 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561906e360f0 con 0x5619071fedd0 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafee000640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x5619072065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafecc00640 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fafd401c890 tx=0x7fafd400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c4487723d2d799 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.362+0000 7fafddc00640 1 -- 172.21.15.119:0/1079965936 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd401c420 con 0x5619071fedd0 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7fafddc00640 1 -- 172.21.15.119:0/1079965936 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd4005520 con 0x5619071fedd0 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7fafddc00640 1 -- 172.21.15.119:0/1079965936 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40048a0 con 0x5619071fedd0 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 -- 172.21.15.119:0/1079965936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 msgr2=0x561907206b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fafd401c890 tx=0x7fafd400f790 comp rx=0 tx=0).stop 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 -- 172.21.15.119:0/1079965936 shutdown_connections 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561907207460 0x561907208850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5619071fedd0 0x561907206b20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 --2- 172.21.15.119:0/1079965936 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x5619072065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 -- 172.21.15.119:0/1079965936 >> 172.21.15.119:0/1079965936 conn(0x5619071c5200 msgr2=0x5619072027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 -- 172.21.15.119:0/1079965936 shutdown_connections 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.363+0000 7faff2c60740 1 -- 172.21.15.119:0/1079965936 wait complete. 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 -- start start 2024-03-28T18:47:53.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 Processor -- start 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907200c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561907207460 0x561907271c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b4ac0 con 0x5619071fedd0 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b4460 con 0x5619071fe0f0 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81b1960 con 0x561907207460 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.364+0000 7faff2c60740 10 client.0 _ll_register_callbacks cb 0x7fafd4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:53.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafee000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907200c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36986/0 (socket says 172.21.15.119:36986) 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 -- 172.21.15.119:0/4067967924 learned_addr learned my addr 172.21.15.119:0/4067967924 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafee000640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561907207460 0x561907271c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 -- 172.21.15.119:0/4067967924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561907207460 msgr2=0x561907271c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561907207460 0x561907271c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 -- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 msgr2=0x561907200c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907200c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 -- 172.21.15.119:0/4067967924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619072721b0 con 0x5619071fedd0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafecc00640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fafd401c810 tx=0x7fafd401c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4de266f360e5bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.365+0000 7fafd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd402d690 con 0x5619071fedd0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fafd402d830 con 0x5619071fedd0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafd4005520 con 0x5619071fedd0 2024-03-28T18:47:53.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd403e050 con 0x5619071fedd0 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 10 client.16278 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561907273430 con 0x5619071fedd0 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafd4005520 con 0x5619071fedd0 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fafd402b070 con 0x5619071fedd0 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 10 client.16278 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 20 client.16278 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:53.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561907273a40 0x561907275e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 4 client.16278 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.366+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561907169ae0 con 0x561907273a40 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7fafee000640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561907273a40 0x561907275e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7fafee000640 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561907273a40 0x561907275e60 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d86d9130a101a14 server_cookie=9712e35655cd6103 in_seq=0 out_seq=0 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7fafd2a00640 10 client.16278 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7fafd2a00640 1 -- 172.21.15.119:0/4067967924 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561907169ae0 con 0x561907273a40 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7fafd2a00640 10 client.16278 handle_command_reply: tid=0 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.367+0000 7faff2c60740 1 client.16278 shutdown 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 msgr2=0x5619072011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fafd401c810 tx=0x7fafd401c650 comp rx=0 tx=0).stop 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 shutdown_connections 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561907273a40 0x561907275e60 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561907207460 0x561907271c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5619071fedd0 0x5619072011a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 --2- 172.21.15.119:0/4067967924 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5619071fe0f0 0x561907200c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7fafd2a00640 0 client.16278 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 >> 172.21.15.119:0/4067967924 conn(0x5619071c5200 msgr2=0x5619071f7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 shutdown_connections 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 1 -- 172.21.15.119:0/4067967924 wait complete. 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7faff2c60740 20 client.16278 trim_cache size 0 max 16384 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:53.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 msgr2=0x7fafcc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 0x7fafcc0a4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5619071ff320 tx=0x7fafd802cf90 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 msgr2=0x7fafe819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.368+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005ab0 tx=0x7fafd80042f0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 shutdown_connections 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fafcc0a29f0 0x7fafcc0a4eb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe81a2cb0 0x7fafe81a3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafe810cea0 0x7fafe819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 --2- 172.21.15.119:0/3081095392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fafe8107e50 0x7fafe819e440 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 >> 172.21.15.119:0/3081095392 conn(0x7fafe80774c0 msgr2=0x7fafe8077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 shutdown_connections 2024-03-28T18:47:53.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.369+0000 7fafeea00640 1 -- 172.21.15.119:0/3081095392 wait complete. 2024-03-28T18:47:53.386 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:47:53.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 -- 172.21.15.119:0/4026082176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 msgr2=0x7feea0075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 --2- 172.21.15.119:0/4026082176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea0075fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fee94009930 tx=0x7fee9402bc20 comp rx=0 tx=0).stop 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 -- 172.21.15.119:0/4026082176 shutdown_connections 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 --2- 172.21.15.119:0/4026082176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 --2- 172.21.15.119:0/4026082176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea0111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 --2- 172.21.15.119:0/4026082176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea0075fb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 -- 172.21.15.119:0/4026082176 >> 172.21.15.119:0/4026082176 conn(0x7feea00ff310 msgr2=0x7feea0101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.580+0000 7feea8800640 1 -- 172.21.15.119:0/4026082176 shutdown_connections 2024-03-28T18:47:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.581+0000 7feea8800640 1 -- 172.21.15.119:0/4026082176 wait complete. 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 Processor -- start 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 -- start start 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0118090 con 0x7feea00764f0 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0117f10 con 0x7feea01124e0 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea8800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0118210 con 0x7feea0075bd0 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46228/0 (socket says 172.21.15.119:46228) 2024-03-28T18:47:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.582+0000 7feea6800640 1 -- 172.21.15.119:0/3249882587 learned_addr learned my addr 172.21.15.119:0/3249882587 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5400640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 -- 172.21.15.119:0/3249882587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 msgr2=0x7feea01a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 -- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 msgr2=0x7feea019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 -- 172.21.15.119:0/3249882587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea01a3580 con 0x7feea0075bd0 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5400640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea6800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:53.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea5e00640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea019e590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fee94036040 tx=0x7fee94004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94043020 con 0x7feea0075bd0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee94005000 con 0x7feea0075bd0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94003a80 con 0x7feea0075bd0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.583+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea01a3810 con 0x7feea0075bd0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.584+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea01ab100 con 0x7feea0075bd0 2024-03-28T18:47:53.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.585+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea00688e0 con 0x7feea0075bd0 2024-03-28T18:47:53.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.586+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fee94046020 con 0x7feea0075bd0 2024-03-28T18:47:53.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.587+0000 7fee9aa00640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 0x7fee880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:53.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.588+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fee940f0cd0 con 0x7feea0075bd0 2024-03-28T18:47:53.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.588+0000 7feea5400640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 0x7fee880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:53.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.588+0000 7feea5400640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 0x7fee880a4eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feea00788b0 tx=0x7fee90005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:53.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.595+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee9404d020 con 0x7feea0075bd0 2024-03-28T18:47:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:53 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:47:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:53 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:53 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:47:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:53 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:53 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:47:53.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.833+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feea010d670 con 0x7feea0075bd0 2024-03-28T18:47:53.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.834+0000 7fee9aa00640 1 -- 172.21.15.119:0/3249882587 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fee940bda80 con 0x7feea0075bd0 2024-03-28T18:47:53.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:47:53.835 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:47:53.835 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 msgr2=0x7fee880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 0x7fee880a4eb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feea00788b0 tx=0x7fee90005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 msgr2=0x7feea019e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea019e590 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fee94036040 tx=0x7fee94004250 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 shutdown_connections 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.837+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fee880a29f0 0x7fee880a4eb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7feea01124e0 0x7feea01a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feea00764f0 0x7feea019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 --2- 172.21.15.119:0/3249882587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feea0075bd0 0x7feea019e590 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 >> 172.21.15.119:0/3249882587 conn(0x7feea00ff310 msgr2=0x7feea0101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:53.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 shutdown_connections 2024-03-28T18:47:53.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:53.838+0000 7feea8800640 1 -- 172.21.15.119:0/3249882587 wait complete. 2024-03-28T18:47:53.856 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:53.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:47:53.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:53 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:47:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:53 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:53 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:47:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:53 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:53 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:47:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:53 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:47:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:53 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:53 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:47:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:53 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:53.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:53 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 -- 172.21.15.119:0/1907461353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb374103720 msgr2=0x7fb374103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 --2- 172.21.15.119:0/1907461353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb374103720 0x7fb374103ba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009930 tx=0x7fb35c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 -- 172.21.15.119:0/1907461353 shutdown_connections 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 --2- 172.21.15.119:0/1907461353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3741040e0 0x7fb3741105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 --2- 172.21.15.119:0/1907461353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb374103720 0x7fb374103ba0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 --2- 172.21.15.119:0/1907461353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374109730 0x7fb374109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 -- 172.21.15.119:0/1907461353 >> 172.21.15.119:0/1907461353 conn(0x7fb3740ff1d0 msgr2=0x7fb3741015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 -- 172.21.15.119:0/1907461353 shutdown_connections 2024-03-28T18:47:54.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.066+0000 7fb373e00640 1 -- 172.21.15.119:0/1907461353 wait complete. 2024-03-28T18:47:54.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.067+0000 7fb373e00640 1 Processor -- start 2024-03-28T18:47:54.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.067+0000 7fb373e00640 1 -- start start 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb374109730 0x7fb37419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374113b90 con 0x7fb374109730 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374113a10 con 0x7fb374103720 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374113d10 con 0x7fb3741040e0 2024-03-28T18:47:54.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46934/0 (socket says 172.21.15.119:46934) 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46242/0 (socket says 172.21.15.119:46242) 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372000640 1 -- 172.21.15.119:0/3499677872 learned_addr learned my addr 172.21.15.119:0/3499677872 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb373400640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb374109730 0x7fb37419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372000640 1 -- 172.21.15.119:0/3499677872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 msgr2=0x7fb37419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.068+0000 7fb372000640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb372000640 1 -- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb374109730 msgr2=0x7fb37419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb372000640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb374109730 0x7fb37419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb372000640 1 -- 172.21.15.119:0/3499677872 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37419f090 con 0x7fb3741040e0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb372a00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb372000640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009900 tx=0x7fb35c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb378a00640 1 -- 172.21.15.119:0/3499677872 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c043020 con 0x7fb3741040e0 2024-03-28T18:47:54.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb378a00640 1 -- 172.21.15.119:0/3499677872 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb35c0054e0 con 0x7fb3741040e0 2024-03-28T18:47:54.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37419f320 con 0x7fb3741040e0 2024-03-28T18:47:54.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb378a00640 1 -- 172.21.15.119:0/3499677872 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c003b80 con 0x7fb3741040e0 2024-03-28T18:47:54.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.069+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3741a6c10 con 0x7fb3741040e0 2024-03-28T18:47:54.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.072+0000 7fb37ca1f740 1 Processor -- start 2024-03-28T18:47:54.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.072+0000 7fb378a00640 1 -- 172.21.15.119:0/3499677872 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fb35c046020 con 0x7fb3741040e0 2024-03-28T18:47:54.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 -- start start 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334dfa4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334defc2c0 con 0x56334dfa1de0 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334dea4c90 con 0x56334df99260 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334def8360 con 0x56334df988b0 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb372000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334dfa4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46942/0 (socket says 172.21.15.119:46942) 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 -- 172.21.15.119:0/4198807863 learned_addr learned my addr 172.21.15.119:0/4198807863 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb372a00640 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 -- 172.21.15.119:0/4198807863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 msgr2=0x56334dfa1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.073+0000 7fb373400640 1 -- 172.21.15.119:0/4198807863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 msgr2=0x56334dfa4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb373400640 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334dfa4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb373400640 1 -- 172.21.15.119:0/4198807863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334dec9080 con 0x56334df988b0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb373400640 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb36c010370 tx=0x7fb36c010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f8094fb476a21f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb378a00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 0x7fb3540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb363e00640 1 -- 172.21.15.119:0/4198807863 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c016070 con 0x56334df988b0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb363e00640 1 -- 172.21.15.119:0/4198807863 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb36c004d20 con 0x56334df988b0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb363e00640 1 -- 172.21.15.119:0/4198807863 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c005420 con 0x56334df988b0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb378a00640 1 -- 172.21.15.119:0/3499677872 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb35c0f0ed0 con 0x7fb3741040e0 2024-03-28T18:47:54.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.074+0000 7fb372a00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 0x7fb3540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- 172.21.15.119:0/4198807863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 msgr2=0x56334df98c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb36c010370 tx=0x7fb36c010830 comp rx=0 tx=0).stop 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- 172.21.15.119:0/4198807863 shutdown_connections 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334dfa4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/4198807863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df98c90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- 172.21.15.119:0/4198807863 >> 172.21.15.119:0/4198807863 conn(0x56334df92f80 msgr2=0x56334df93360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- 172.21.15.119:0/4198807863 shutdown_connections 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb372a00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 0x7fb3540a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56334df93ff0 tx=0x7fb364005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- 172.21.15.119:0/4198807863 wait complete. 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.075+0000 7fb37ca1f740 1 -- start start 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 Processor -- start 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 0x56334dfa17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334dfa1de0 0x56334df9f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334dfa4d50 con 0x56334df988b0 2024-03-28T18:47:54.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334deb80d0 con 0x56334df99260 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334dfa4ed0 con 0x56334dfa1de0 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb37ca1f740 10 client.0 _ll_register_callbacks cb 0x56334debce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb372000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb372000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46262/0 (socket says 172.21.15.119:46262) 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb372000640 1 -- 172.21.15.119:0/2456483629 learned_addr learned my addr 172.21.15.119:0/2456483629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.076+0000 7fb372a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 0x56334dfa17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb373400640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334dfa1de0 0x56334df9f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 -- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334dfa1de0 msgr2=0x56334df9f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334dfa1de0 0x56334df9f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 -- 172.21.15.119:0/2456483629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 msgr2=0x56334dfa17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 0x56334dfa17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 -- 172.21.15.119:0/2456483629 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56334dc103f0 con 0x56334df99260 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372a00640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 0x56334dfa17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb372000640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b9b00 tx=0x7fb35c0f4570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53182e7ae4c7bc63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb360c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0b9940 con 0x56334df99260 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb36c004d20 con 0x56334df99260 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb35c0ee7e0 con 0x56334df99260 2024-03-28T18:47:54.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb35c0bd000 con 0x56334df99260 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb37ca1f740 10 client.26516 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.077+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56334dcda7d0 con 0x56334df99260 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb36c004d20 con 0x56334df99260 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb35c0f4020 con 0x56334df99260 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 10 client.26516 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 10 client.26516 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 20 client.26516 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334dfb4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 4 client.26516 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56334df09800 con 0x56334dfb1c20 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.078+0000 7fb372a00640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334dfb4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.079+0000 7fb360c00640 10 client.26516 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.079+0000 7fb372a00640 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334dfb4080 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7f27e9e12e39ca5c server_cookie=86df772e5697532f in_seq=0 out_seq=0 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb360c00640 1 -- 172.21.15.119:0/2456483629 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56334df09800 con 0x56334dfb1c20 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb360c00640 10 client.26516 handle_command_reply: tid=0 2024-03-28T18:47:54.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 client.26516 shutdown 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 msgr2=0x56334df9eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b9b00 tx=0x7fb35c0f4570 comp rx=0 tx=0).stop 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 shutdown_connections 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334dfb4080 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334dfa1de0 0x56334df9f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334df9eb50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb360c00640 0 client.26516 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:54.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/2456483629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df988b0 0x56334dfa17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 >> 172.21.15.119:0/2456483629 conn(0x56334df92f80 msgr2=0x56334df94cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 shutdown_connections 2024-03-28T18:47:54.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.080+0000 7fb37ca1f740 1 -- 172.21.15.119:0/2456483629 wait complete. 2024-03-28T18:47:54.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.081+0000 7fb37ca1f740 20 client.26516 trim_cache size 0 max 16384 2024-03-28T18:47:54.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 Processor -- start 2024-03-28T18:47:54.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 -- start start 2024-03-28T18:47:54.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 0x56334dfa1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 0x56334dfa17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741af3e0 con 0x56334df99260 2024-03-28T18:47:54.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741b0d60 con 0x56334dfa1de0 2024-03-28T18:47:54.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.094+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741afa40 con 0x56334df988b0 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 0x56334dfa1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb373400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 0x56334dfa17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46958/0 (socket says 172.21.15.119:46958) 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 0x56334dfa1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37046/0 (socket says 172.21.15.119:37046) 2024-03-28T18:47:54.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb373400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 0x56334dfa17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46268/0 (socket says 172.21.15.119:46268) 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 -- 172.21.15.119:0/3057453079 learned_addr learned my addr 172.21.15.119:0/3057453079 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 -- 172.21.15.119:0/3057453079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 msgr2=0x56334dfa17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 0x56334dfa17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 -- 172.21.15.119:0/3057453079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 msgr2=0x56334dfa1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 0x56334dfa1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 -- 172.21.15.119:0/3057453079 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334df92720 con 0x56334df988b0 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.095+0000 7fb372a00640 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x56334df98380 tx=0x7fb364017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db8a6c78cb88b7a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb363400640 1 -- 172.21.15.119:0/3057453079 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36402b680 con 0x56334df988b0 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb363400640 1 -- 172.21.15.119:0/3057453079 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb364030070 con 0x56334df988b0 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb363400640 1 -- 172.21.15.119:0/3057453079 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36401fb00 con 0x56334df988b0 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 -- 172.21.15.119:0/3057453079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 msgr2=0x56334df9dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x56334df98380 tx=0x7fb364017000 comp rx=0 tx=0).stop 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 -- 172.21.15.119:0/3057453079 shutdown_connections 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334dfa1de0 0x56334dfa17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334df99260 0x56334dfa1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/3057453079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334df9dbc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.096+0000 7fb37ca1f740 1 -- 172.21.15.119:0/3057453079 >> 172.21.15.119:0/3057453079 conn(0x56334df92f80 msgr2=0x56334df99640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.097+0000 7fb37ca1f740 1 -- 172.21.15.119:0/3057453079 shutdown_connections 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.097+0000 7fb37ca1f740 1 -- 172.21.15.119:0/3057453079 wait complete. 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.097+0000 7fb37ca1f740 1 -- start start 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.097+0000 7fb37ca1f740 1 Processor -- start 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334dfa7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334e00c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741b0700 con 0x56334dfa1de0 2024-03-28T18:47:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741b00a0 con 0x56334df99260 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741ad5a0 con 0x56334df988b0 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb37ca1f740 10 client.0 _ll_register_callbacks cb 0x7fb35c02ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb373400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334e00c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334dfa7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46282/0 (socket says 172.21.15.119:46282) 2024-03-28T18:47:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334dfa7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46962/0 (socket says 172.21.15.119:46962) 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb373400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334e00c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37050/0 (socket says 172.21.15.119:37050) 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 -- 172.21.15.119:0/687966685 learned_addr learned my addr 172.21.15.119:0/687966685 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 -- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 msgr2=0x56334dfa7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334dfa7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 -- 172.21.15.119:0/687966685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 msgr2=0x56334e00c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334e00c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.098+0000 7fb372000640 1 -- 172.21.15.119:0/687966685 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56334e00caa0 con 0x56334df99260 2024-03-28T18:47:54.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb372000640 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb35c02cb40 tx=0x7fb35c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46f63b8783a4db0e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0bd1e0 con 0x56334df99260 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb35c0bcaf0 con 0x56334df99260 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb364030070 con 0x56334df99260 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb35c0bd470 con 0x56334df99260 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 10 client.26522 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56334e00caa0 con 0x56334df99260 2024-03-28T18:47:54.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb364030070 con 0x56334df99260 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb35c0b93f0 con 0x56334df99260 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 10 client.26522 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 10 client.26522 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 20 client.26522 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.099+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334e010080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb37ca1f740 4 client.26522 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56334df03b50 con 0x56334dfb1c20 2024-03-28T18:47:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb372a00640 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334e010080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb372a00640 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334e010080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6e7eaadec2398b1 server_cookie=e1ccedf1a303160f in_seq=0 out_seq=0 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb360c00640 10 client.26522 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb360c00640 1 -- 172.21.15.119:0/687966685 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56334df03b50 con 0x56334dfb1c20 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.100+0000 7fb360c00640 10 client.26522 handle_command_reply: tid=0 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 client.26522 shutdown 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 msgr2=0x56334dfa8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb35c02cb40 tx=0x7fb35c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 shutdown_connections 2024-03-28T18:47:54.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56334dfb1c20 0x56334e010080 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56334dfa1de0 0x56334e00c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56334df99260 0x56334dfa8160 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb360c00640 0 client.26522 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 --2- 172.21.15.119:0/687966685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56334df988b0 0x56334dfa7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 >> 172.21.15.119:0/687966685 conn(0x56334df92f80 msgr2=0x56334df9da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 shutdown_connections 2024-03-28T18:47:54.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 1 -- 172.21.15.119:0/687966685 wait complete. 2024-03-28T18:47:54.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.101+0000 7fb37ca1f740 20 client.26522 trim_cache size 0 max 16384 2024-03-28T18:47:54.113 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:54.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 msgr2=0x7fb3540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 0x7fb3540a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x56334df93ff0 tx=0x7fb364005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 msgr2=0x7fb37419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009900 tx=0x7fb35c004360 comp rx=0 tx=0).stop 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 shutdown_connections 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb3540a2970 0x7fb3540a4e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb374109730 0x7fb37419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3741040e0 0x7fb37419a630 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 --2- 172.21.15.119:0/3499677872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb374103720 0x7fb37419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 >> 172.21.15.119:0/3499677872 conn(0x7fb3740ff1d0 msgr2=0x7fb3740ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 shutdown_connections 2024-03-28T18:47:54.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.102+0000 7fb373e00640 1 -- 172.21.15.119:0/3499677872 wait complete. 2024-03-28T18:47:54.119 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:54.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.323+0000 7f3ed7a00640 1 -- 172.21.15.119:0/1902545018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed00764f0 msgr2=0x7f3ed0111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.323+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/1902545018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed00764f0 0x7f3ed0111fa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8009930 tx=0x7f3eb802bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 -- 172.21.15.119:0/1902545018 shutdown_connections 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/1902545018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed01124e0 0x7f3ed01148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/1902545018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed00764f0 0x7f3ed0111fa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/1902545018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed0075bd0 0x7f3ed0075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 -- 172.21.15.119:0/1902545018 >> 172.21.15.119:0/1902545018 conn(0x7f3ed00ff2f0 msgr2=0x7f3ed0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 -- 172.21.15.119:0/1902545018 shutdown_connections 2024-03-28T18:47:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.324+0000 7f3ed7a00640 1 -- 172.21.15.119:0/1902545018 wait complete. 2024-03-28T18:47:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 Processor -- start 2024-03-28T18:47:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 -- start start 2024-03-28T18:47:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 0x7f3ed019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed01124e0 0x7f3ed01a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.325+0000 7f3ed7a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed0118090 con 0x7f3ed00764f0 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed7a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed0117f10 con 0x7f3ed0075bd0 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed7a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed0118210 con 0x7f3ed01124e0 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46294/0 (socket says 172.21.15.119:46294) 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 -- 172.21.15.119:0/3876711885 learned_addr learned my addr 172.21.15.119:0/3876711885 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed01124e0 0x7f3ed01a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 0x7f3ed019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 -- 172.21.15.119:0/3876711885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed01124e0 msgr2=0x7f3ed01a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed01124e0 0x7f3ed01a2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 -- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 msgr2=0x7f3ed019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 0x7f3ed019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 -- 172.21.15.119:0/3876711885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed01a3620 con 0x7f3ed0075bd0 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 0x7f3ed019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.326+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000e9f0 tx=0x7f3ec000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.327+0000 7f3ece000640 1 -- 172.21.15.119:0/3876711885 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000cd80 con 0x7f3ed0075bd0 2024-03-28T18:47:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.327+0000 7f3ece000640 1 -- 172.21.15.119:0/3876711885 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ec0010910 con 0x7f3ed0075bd0 2024-03-28T18:47:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.327+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed01a3910 con 0x7f3ed0075bd0 2024-03-28T18:47:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.327+0000 7f3ece000640 1 -- 172.21.15.119:0/3876711885 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00188f0 con 0x7f3ed0075bd0 2024-03-28T18:47:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.327+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed01ab1a0 con 0x7f3ed0075bd0 2024-03-28T18:47:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.330+0000 7f3eda7ac740 1 Processor -- start 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.330+0000 7f3ece000640 1 -- 172.21.15.119:0/3876711885 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3ec0010ac0 con 0x7f3ed0075bd0 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.330+0000 7f3eda7ac740 1 -- start start 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030e8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030f4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e30304c2c0 con 0x55e3030f1c00 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e302ff4c90 con 0x55e3030e86d0 2024-03-28T18:47:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e303048360 con 0x55e3030e9080 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030e8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030e8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46306/0 (socket says 172.21.15.119:46306) 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5a00640 1 -- 172.21.15.119:0/3114915830 learned_addr learned my addr 172.21.15.119:0/3114915830 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 -- 172.21.15.119:0/3114915830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 msgr2=0x55e3030e8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030e8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 -- 172.21.15.119:0/3114915830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 msgr2=0x55e3030f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030f4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.331+0000 7f3ed5000640 1 -- 172.21.15.119:0/3114915830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e303019080 con 0x55e3030e9080 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.332+0000 7f3ed5000640 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f1630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00cc970 tx=0x7f3ec00d7930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3876b1ed0f5d796 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.332+0000 7f3ec6000640 1 -- 172.21.15.119:0/3114915830 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00e2070 con 0x55e3030e9080 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.332+0000 7f3ec6000640 1 -- 172.21.15.119:0/3114915830 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ec00d1480 con 0x55e3030e9080 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.332+0000 7f3ec6000640 1 -- 172.21.15.119:0/3114915830 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00ddae0 con 0x55e3030e9080 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.332+0000 7f3ece000640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 0x7f3eb40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3ece000640 1 -- 172.21.15.119:0/3876711885 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3ec00cf2e0 con 0x7f3ed0075bd0 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- 172.21.15.119:0/3114915830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 msgr2=0x55e3030f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f1630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00cc970 tx=0x7f3ec00d7930 comp rx=0 tx=0).stop 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 0x7f3eb40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- 172.21.15.119:0/3114915830 shutdown_connections 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f1630 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/3114915830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030e8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- 172.21.15.119:0/3114915830 >> 172.21.15.119:0/3114915830 conn(0x55e3030e2da0 msgr2=0x55e3030e3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- 172.21.15.119:0/3114915830 shutdown_connections 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- 172.21.15.119:0/3114915830 wait complete. 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3eda7ac740 1 -- start start 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.333+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 0x7f3eb40a4eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005fd0 tx=0x7f3ec80096a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 Processor -- start 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030ee880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3030f4b70 con 0x55e3030f1c00 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3030080d0 con 0x55e3030e86d0 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3030f4cf0 con 0x55e3030e9080 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3eda7ac740 10 client.0 _ll_register_callbacks cb 0x55e30300ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3ec7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3ec7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030ee880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:46994/0 (socket says 172.21.15.119:46994) 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3ec7e00640 1 -- 172.21.15.119:0/532305778 learned_addr learned my addr 172.21.15.119:0/532305778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.334+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 -- 172.21.15.119:0/532305778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 msgr2=0x55e3030ee880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030ee880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 -- 172.21.15.119:0/532305778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 msgr2=0x55e3030eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030eee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 -- 172.21.15.119:0/532305778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e302d603f0 con 0x55e3030e86d0 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ed5000640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030f14f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55e3030e3e10 tx=0x7f3ec0009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c240ac40beda23e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ebea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0097b20 con 0x55e3030e86d0 2024-03-28T18:47:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3ec00860d0 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ec00d1480 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec009b1e0 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3eda7ac740 10 client.26534 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.335+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e302e2a7d0 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ec00d1480 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3ec00ccbf0 con 0x55e3030e86d0 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 10 client.26534 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 10 client.26534 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 20 client.26534 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e303101950 0x55e303103d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 4 client.26534 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e303059800 con 0x55e303101950 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.336+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e303101950 0x55e303103d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.337+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e303101950 0x55e303103d40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=33fe3a7a1f6c9e7 server_cookie=76890e5fe5fb0995 in_seq=0 out_seq=0 2024-03-28T18:47:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.337+0000 7f3ebea00640 10 client.26534 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.337+0000 7f3ebea00640 1 -- 172.21.15.119:0/532305778 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e303059800 con 0x55e303101950 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.337+0000 7f3ebea00640 10 client.26534 handle_command_reply: tid=0 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.337+0000 7f3eda7ac740 1 client.26534 shutdown 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 msgr2=0x55e3030f14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030f14f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55e3030e3e10 tx=0x7f3ec0009770 comp rx=0 tx=0).stop 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 shutdown_connections 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e303101950 0x55e303103d40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e3030eee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030ee880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/532305778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e86d0 0x55e3030f14f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3ebea00640 0 client.26534 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 >> 172.21.15.119:0/532305778 conn(0x55e3030e2da0 msgr2=0x55e3030e4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 shutdown_connections 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 1 -- 172.21.15.119:0/532305778 wait complete. 2024-03-28T18:47:54.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.338+0000 7f3eda7ac740 20 client.26534 trim_cache size 0 max 16384 2024-03-28T18:47:54.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.351+0000 7f3eda7ac740 1 Processor -- start 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.351+0000 7f3eda7ac740 1 -- start start 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030f1c00 0x55e3030f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b3ad0 con 0x55e3030e86d0 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b5450 con 0x55e3030f1c00 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b4130 con 0x55e3030e9080 2024-03-28T18:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47000/0 (socket says 172.21.15.119:47000) 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 -- 172.21.15.119:0/2392170561 learned_addr learned my addr 172.21.15.119:0/2392170561 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ed5000640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37110/0 (socket says 172.21.15.119:37110) 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030f1c00 0x55e3030f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 -- 172.21.15.119:0/2392170561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030f1c00 msgr2=0x55e3030f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030f1c00 0x55e3030f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 -- 172.21.15.119:0/2392170561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 msgr2=0x55e3030fdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ec7e00640 1 -- 172.21.15.119:0/2392170561 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3030ef170 con 0x55e3030e9080 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.352+0000 7f3ed5000640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.353+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3eb802c240 tx=0x7f3eb802c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=286d1197ff888ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.353+0000 7f3ec5600640 1 -- 172.21.15.119:0/2392170561 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb804f070 con 0x55e3030e9080 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.353+0000 7f3ec5600640 1 -- 172.21.15.119:0/2392170561 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8004320 con 0x55e3030e9080 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.353+0000 7f3ec5600640 1 -- 172.21.15.119:0/2392170561 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb80046c0 con 0x55e3030e9080 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2392170561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 msgr2=0x55e3030f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3eb802c240 tx=0x7f3eb802c330 comp rx=0 tx=0).stop 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2392170561 shutdown_connections 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030f1c00 0x55e3030f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e9080 0x55e3030f0fb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2392170561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030e86d0 0x55e3030fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2392170561 >> 172.21.15.119:0/2392170561 conn(0x55e3030e2da0 msgr2=0x55e3030e1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2392170561 shutdown_connections 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2392170561 wait complete. 2024-03-28T18:47:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 -- start start 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.354+0000 7f3eda7ac740 1 Processor -- start 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 0x55e303106c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e303104d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b4df0 con 0x55e3030f1c00 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b4790 con 0x55e3030e9080 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed01b1c90 con 0x55e3030e86d0 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3eda7ac740 10 client.0 _ll_register_callbacks cb 0x7f3ec00e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47002/0 (socket says 172.21.15.119:47002) 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 -- 172.21.15.119:0/2005374285 learned_addr learned my addr 172.21.15.119:0/2005374285 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 0x55e303106c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e303104d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 -- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 msgr2=0x55e303106c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 0x55e303106c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 -- 172.21.15.119:0/2005374285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 msgr2=0x55e303104d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e303104d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5000640 1 -- 172.21.15.119:0/2005374285 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e303105570 con 0x55e3030e86d0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 0x55e303106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.355+0000 7f3ed5a00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e303104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ed5000640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0097ce0 tx=0x7f3ec00d2d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a799e5483458a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec009b3c0 con 0x55e3030e86d0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3ec00a3d90 con 0x55e3030e86d0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3eb8004320 con 0x55e3030e86d0 2024-03-28T18:47:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec009b650 con 0x55e3030e86d0 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3eda7ac740 10 client.26524 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e303105570 con 0x55e3030e86d0 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8004320 con 0x55e3030e86d0 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3ec009bae0 con 0x55e3030e86d0 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3eda7ac740 10 client.26524 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.356+0000 7f3eda7ac740 10 client.26524 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3eda7ac740 20 client.26524 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e30315dd50 0x55e303160140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3eda7ac740 4 client.26524 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e303053b50 con 0x55e30315dd50 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e30315dd50 0x55e303160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3ec7e00640 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e30315dd50 0x55e303160140 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64a20f9404d2ec40 server_cookie=a1d1c2479d0f7258 in_seq=0 out_seq=0 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3ebea00640 10 client.26524 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3ebea00640 1 -- 172.21.15.119:0/2005374285 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e303053b50 con 0x55e30315dd50 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3ebea00640 10 client.26524 handle_command_reply: tid=0 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.357+0000 7f3eda7ac740 1 client.26524 shutdown 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 msgr2=0x55e303106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0097ce0 tx=0x7f3ec00d2d90 comp rx=0 tx=0).stop 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 shutdown_connections 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55e30315dd50 0x55e303160140 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e3030f1c00 0x55e303104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e3030e9080 0x55e303106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 --2- 172.21.15.119:0/2005374285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e3030e86d0 0x55e303106730 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3ebea00640 0 client.26524 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 >> 172.21.15.119:0/2005374285 conn(0x55e3030e2da0 msgr2=0x55e3030e40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 shutdown_connections 2024-03-28T18:47:54.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 1 -- 172.21.15.119:0/2005374285 wait complete. 2024-03-28T18:47:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3eda7ac740 20 client.26524 trim_cache size 0 max 16384 2024-03-28T18:47:54.373 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:54.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 msgr2=0x7f3eb40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.358+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 0x7f3eb40a4eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8005fd0 tx=0x7f3ec80096a0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 msgr2=0x7f3ed019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000e9f0 tx=0x7f3ec000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 shutdown_connections 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3eb40a29f0 0x7f3eb40a4eb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed01124e0 0x7f3ed01a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ed00764f0 0x7f3ed019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 --2- 172.21.15.119:0/3876711885 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3ed0075bd0 0x7f3ed019e630 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 >> 172.21.15.119:0/3876711885 conn(0x7f3ed00ff2f0 msgr2=0x7f3ed01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 shutdown_connections 2024-03-28T18:47:54.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.359+0000 7f3ed7a00640 1 -- 172.21.15.119:0/3876711885 wait complete. 2024-03-28T18:47:54.377 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:54.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 -- 172.21.15.119:0/3810458264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 msgr2=0x7f2a1010d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 --2- 172.21.15.119:0/3810458264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1010d910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009900 tx=0x7f29fc02bcc0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 -- 172.21.15.119:0/3810458264 shutdown_connections 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 --2- 172.21.15.119:0/3810458264 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a1010de50 0x7f2a101102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 --2- 172.21.15.119:0/3810458264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1010d910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 --2- 172.21.15.119:0/3810458264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a10073ad0 0x7f2a10073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 -- 172.21.15.119:0/3810458264 >> 172.21.15.119:0/3810458264 conn(0x7f2a100fd100 msgr2=0x7f2a100ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 -- 172.21.15.119:0/3810458264 shutdown_connections 2024-03-28T18:47:54.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.595+0000 7f2a16a00640 1 -- 172.21.15.119:0/3810458264 wait complete. 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.596+0000 7f2a16a00640 1 Processor -- start 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.596+0000 7f2a16a00640 1 -- start start 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.596+0000 7f2a16a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a10073ad0 0x7f2a1019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a16a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a16a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a10113c40 con 0x7f2a10073ad0 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a16a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a10113ac0 con 0x7f2a1010de50 2024-03-28T18:47:54.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a16a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a10113dc0 con 0x7f2a100743f0 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a10073ad0 0x7f2a1019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46364/0 (socket says 172.21.15.119:46364) 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a0f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 -- 172.21.15.119:0/2128606677 learned_addr learned my addr 172.21.15.119:0/2128606677 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 -- 172.21.15.119:0/2128606677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 msgr2=0x7f2a1019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 -- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a10073ad0 msgr2=0x7f2a1019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a10073ad0 0x7f2a1019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a14a00640 1 -- 172.21.15.119:0/2128606677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1019f1a0 con 0x7f2a1010de50 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.597+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a14a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800eb10 tx=0x7f2a0800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a0cc00640 1 -- 172.21.15.119:0/2128606677 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08004b20 con 0x7f2a1010de50 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a0cc00640 1 -- 172.21.15.119:0/2128606677 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a08004cc0 con 0x7f2a1010de50 2024-03-28T18:47:54.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a0cc00640 1 -- 172.21.15.119:0/2128606677 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a08010760 con 0x7f2a1010de50 2024-03-28T18:47:54.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1019f490 con 0x7f2a1010de50 2024-03-28T18:47:54.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.598+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a101a6d20 con 0x7f2a1010de50 2024-03-28T18:47:54.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.601+0000 7f2a0cc00640 1 -- 172.21.15.119:0/2128606677 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2a08005020 con 0x7f2a1010de50 2024-03-28T18:47:54.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.601+0000 7f2a197d3740 1 Processor -- start 2024-03-28T18:47:54.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.601+0000 7f2a197d3740 1 -- start start 2024-03-28T18:47:54.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabbbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2fab1f2c0 con 0x55a2fabc4c00 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2faac7c90 con 0x55a2fabbb6d0 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a197d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2fab1b360 con 0x55a2fabbc080 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a0f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabbbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47026/0 (socket says 172.21.15.119:47026) 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a0fe00640 1 -- 172.21.15.119:0/40778813 learned_addr learned my addr 172.21.15.119:0/40778813 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.602+0000 7f2a0f400640 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 -- 172.21.15.119:0/40778813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 msgr2=0x55a2fabbbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabbbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 -- 172.21.15.119:0/40778813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 msgr2=0x55a2fabc7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 -- 172.21.15.119:0/40778813 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2faaec080 con 0x55a2fabbc080 2024-03-28T18:47:54.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0f400640 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d20 tx=0x7f2a00011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac920d5e1f11b560 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0cc00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 0x7f29f40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a05200640 1 -- 172.21.15.119:0/40778813 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0001e070 con 0x55a2fabbc080 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a05200640 1 -- 172.21.15.119:0/40778813 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00019b10 con 0x55a2fabbc080 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a05200640 1 -- 172.21.15.119:0/40778813 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0000ebf0 con 0x55a2fabbc080 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0cc00640 1 -- 172.21.15.119:0/2128606677 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2a080cf3d0 con 0x7f2a1010de50 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.603+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 0x7f29f40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 -- 172.21.15.119:0/40778813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 msgr2=0x55a2fabc4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d20 tx=0x7f2a00011810 comp rx=0 tx=0).stop 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 -- 172.21.15.119:0/40778813 shutdown_connections 2024-03-28T18:47:54.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc4630 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 --2- 172.21.15.119:0/40778813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabbbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 0x7f29f40a4eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29fc005ab0 tx=0x7f29fc005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 -- 172.21.15.119:0/40778813 >> 172.21.15.119:0/40778813 conn(0x55a2fabb5da0 msgr2=0x55a2fabb6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 -- 172.21.15.119:0/40778813 shutdown_connections 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.604+0000 7f2a197d3740 1 -- 172.21.15.119:0/40778813 wait complete. 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 -- start start 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 Processor -- start 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fabc1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2fabc7b70 con 0x55a2fabbc080 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2faadb0d0 con 0x55a2fabbb6d0 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.605+0000 7f2a197d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2fabc7cf0 con 0x55a2fabc4c00 2024-03-28T18:47:54.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a197d3740 10 client.0 _ll_register_callbacks cb 0x55a2faadfe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a14a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fabc1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37160/0 (socket says 172.21.15.119:37160) 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0f400640 1 -- 172.21.15.119:0/1292708024 learned_addr learned my addr 172.21.15.119:0/1292708024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 -- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 msgr2=0x55a2fabc1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fabc1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 -- 172.21.15.119:0/1292708024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 msgr2=0x55a2fabc1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 -- 172.21.15.119:0/1292708024 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2fa8333f0 con 0x55a2fabbb6d0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a14a00640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fabc1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc44e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55a2fabb6e10 tx=0x7f2a0000e5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=723877d669a724e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.606+0000 7f29f9600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00026070 con 0x55a2fabbb6d0 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2a00019840 con 0x55a2fabbb6d0 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a00019b10 con 0x55a2fabbb6d0 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a0002f050 con 0x55a2fabbb6d0 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 10 client.26552 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a2fa8fd7d0 con 0x55a2fabbb6d0 2024-03-28T18:47:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00019b10 con 0x55a2fabbb6d0 2024-03-28T18:47:54.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2a00029020 con 0x55a2fabbb6d0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 10 client.26552 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 10 client.26552 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 20 client.26552 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fabd49b0 0x55a2fabd6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.607+0000 7f2a197d3740 4 client.26552 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.608+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2fab2c800 con 0x55a2fabd49b0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.608+0000 7f2a0f400640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fabd49b0 0x55a2fabd6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.608+0000 7f2a0f400640 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fabd49b0 0x55a2fabd6da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f9aae874a073255 server_cookie=f5fec3c879d9069 in_seq=0 out_seq=0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.608+0000 7f29f9600640 10 client.26552 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f29f9600640 1 -- 172.21.15.119:0/1292708024 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a2fab2c800 con 0x55a2fabd49b0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f29f9600640 10 client.26552 handle_command_reply: tid=0 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 client.26552 shutdown 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 msgr2=0x55a2fabc44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc44e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55a2fabb6e10 tx=0x7f2a0000e5e0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 shutdown_connections 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fabd49b0 0x55a2fabd6da0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fabc1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 --2- 172.21.15.119:0/1292708024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc44e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 >> 172.21.15.119:0/1292708024 conn(0x55a2fabb5da0 msgr2=0x55a2fabb7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f29f9600640 0 client.26552 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.609+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 shutdown_connections 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.610+0000 7f2a197d3740 1 -- 172.21.15.119:0/1292708024 wait complete. 2024-03-28T18:47:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.610+0000 7f2a197d3740 20 client.26552 trim_cache size 0 max 16384 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 Processor -- start 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 -- start start 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101af610 con 0x55a2fabc4c00 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101b0f90 con 0x55a2fabbb6d0 2024-03-28T18:47:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.623+0000 7f2a197d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101afc70 con 0x55a2fabbc080 2024-03-28T18:47:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a14a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46396/0 (socket says 172.21.15.119:46396) 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 -- 172.21.15.119:0/859316101 learned_addr learned my addr 172.21.15.119:0/859316101 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0f400640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 -- 172.21.15.119:0/859316101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 msgr2=0x55a2fabc3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 -- 172.21.15.119:0/859316101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 msgr2=0x55a2fabc44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 -- 172.21.15.119:0/859316101 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2fabbbe20 con 0x55a2fabbb6d0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0f400640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a14a00640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.624+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d20 tx=0x7f2a0000e500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f644e3987064a254 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f29fbe00640 1 -- 172.21.15.119:0/859316101 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00015070 con 0x55a2fabbb6d0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f29fbe00640 1 -- 172.21.15.119:0/859316101 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00004520 con 0x55a2fabbb6d0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f29fbe00640 1 -- 172.21.15.119:0/859316101 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00026070 con 0x55a2fabbb6d0 2024-03-28T18:47:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 -- 172.21.15.119:0/859316101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 msgr2=0x55a2fabb4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a00009d20 tx=0x7f2a0000e500 comp rx=0 tx=0).stop 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 -- 172.21.15.119:0/859316101 shutdown_connections 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabc4c00 0x55a2fabc44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabbc080 0x55a2fabc3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 --2- 172.21.15.119:0/859316101 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabb4eb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 -- 172.21.15.119:0/859316101 >> 172.21.15.119:0/859316101 conn(0x55a2fabb5da0 msgr2=0x55a2fabbc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 -- 172.21.15.119:0/859316101 shutdown_connections 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.625+0000 7f2a197d3740 1 -- 172.21.15.119:0/859316101 wait complete. 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 -- start start 2024-03-28T18:47:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 Processor -- start 2024-03-28T18:47:54.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fac2f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101b0930 con 0x55a2fabbc080 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101b02d0 con 0x55a2fabbb6d0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a101ad7d0 con 0x55a2fabc4c00 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a197d3740 10 client.0 _ll_register_callbacks cb 0x7f2a0001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a0f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.626+0000 7f2a0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46410/0 (socket says 172.21.15.119:46410) 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 -- 172.21.15.119:0/2210630159 learned_addr learned my addr 172.21.15.119:0/2210630159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a14a00640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fac2f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 -- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 msgr2=0x55a2fac2f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fac2f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 -- 172.21.15.119:0/2210630159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 msgr2=0x55a2fabc8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 -- 172.21.15.119:0/2210630159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2fac2f560 con 0x55a2fabbb6d0 2024-03-28T18:47:54.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a0fe00640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2a00005c20 tx=0x7f2a00009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbdf582558ae1d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f29f9600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00041060 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a00004520 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2a0000e8a0 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.627+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a00015070 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 10 client.26558 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a2fac30790 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a00004520 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2a00029040 con 0x55a2fabbb6d0 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 10 client.26558 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 10 client.26558 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 20 client.26558 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fac30ca0 0x55a2fac33090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 4 client.26558 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a2fab26b50 con 0x55a2fac30ca0 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.628+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fac30ca0 0x55a2fac33090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a0f400640 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fac30ca0 0x55a2fac33090 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fa167423c7f42d2 server_cookie=6413cc930f6b52b in_seq=0 out_seq=0 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f29f9600640 10 client.26558 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f29f9600640 1 -- 172.21.15.119:0/2210630159 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a2fab26b50 con 0x55a2fac30ca0 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f29f9600640 10 client.26558 handle_command_reply: tid=0 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 client.26558 shutdown 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 msgr2=0x55a2fabc89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2a00005c20 tx=0x7f2a00009620 comp rx=0 tx=0).stop 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 shutdown_connections 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55a2fac30ca0 0x55a2fac33090 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2fabc4c00 0x55a2fac2f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a2fabbc080 0x55a2fabc8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 --2- 172.21.15.119:0/2210630159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a2fabbb6d0 0x55a2fabc89c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f29f9600640 0 client.26558 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 >> 172.21.15.119:0/2210630159 conn(0x55a2fabb5da0 msgr2=0x55a2fabb4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 shutdown_connections 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 1 -- 172.21.15.119:0/2210630159 wait complete. 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.629+0000 7f2a197d3740 20 client.26558 trim_cache size 0 max 16384 2024-03-28T18:47:54.634 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 msgr2=0x7f29f40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 0x7f29f40a4eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f29fc005ab0 tx=0x7f29fc005a20 comp rx=0 tx=0).stop 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 msgr2=0x7f2a1019ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800eb10 tx=0x7f2a0800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 shutdown_connections 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29f40a29f0 0x7f29f40a4eb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2a1010de50 0x7f2a1019ea70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a100743f0 0x7f2a1019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 --2- 172.21.15.119:0/2128606677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a10073ad0 0x7f2a1019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 >> 172.21.15.119:0/2128606677 conn(0x7f2a100fd100 msgr2=0x7f2a100fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 shutdown_connections 2024-03-28T18:47:54.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.630+0000 7f2a16a00640 1 -- 172.21.15.119:0/2128606677 wait complete. 2024-03-28T18:47:54.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:54 smithi119 ceph-mon[31813]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:54.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:54 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3249882587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:54.648 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:54.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 -- 172.21.15.119:0/3517706637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101ef0 msgr2=0x7fa05810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 --2- 172.21.15.119:0/3517706637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101ef0 0x7fa05810e3b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa050009930 tx=0x7fa05002bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 -- 172.21.15.119:0/3517706637 shutdown_connections 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 --2- 172.21.15.119:0/3517706637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101ef0 0x7fa05810e3b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 --2- 172.21.15.119:0/3517706637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101530 0x7fa0581019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 --2- 172.21.15.119:0/3517706637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa0581079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 -- 172.21.15.119:0/3517706637 >> 172.21.15.119:0/3517706637 conn(0x7fa0580fd000 msgr2=0x7fa0580ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 -- 172.21.15.119:0/3517706637 shutdown_connections 2024-03-28T18:47:54.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.854+0000 7fa05e800640 1 -- 172.21.15.119:0/3517706637 wait complete. 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 Processor -- start 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 -- start start 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101ef0 0x7fa05819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa05819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa058111900 con 0x7fa058101ef0 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa058111780 con 0x7fa0581075f0 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.855+0000 7fa05e800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa058111a80 con 0x7fa058101530 2024-03-28T18:47:54.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa05819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47074/0 (socket says 172.21.15.119:47074) 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 -- 172.21.15.119:0/605458481 learned_addr learned my addr 172.21.15.119:0/605458481 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05ca00640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101ef0 0x7fa05819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 -- 172.21.15.119:0/605458481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 msgr2=0x7fa05819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa05819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 -- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101ef0 msgr2=0x7fa05819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101ef0 0x7fa05819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 -- 172.21.15.119:0/605458481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05819eee0 con 0x7fa058101530 2024-03-28T18:47:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05de00640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa05819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.856+0000 7fa05d400640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00ebc0 tx=0x7fa04c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.857+0000 7fa056000640 1 -- 172.21.15.119:0/605458481 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c010630 con 0x7fa058101530 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.857+0000 7fa056000640 1 -- 172.21.15.119:0/605458481 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa04c010cb0 con 0x7fa058101530 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.857+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05819f1d0 con 0x7fa058101530 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.857+0000 7fa056000640 1 -- 172.21.15.119:0/605458481 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c018bd0 con 0x7fa058101530 2024-03-28T18:47:54.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.857+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0581a6a60 con 0x7fa058101530 2024-03-28T18:47:54.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.860+0000 7fa056000640 1 -- 172.21.15.119:0/605458481 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa04c0107d0 con 0x7fa058101530 2024-03-28T18:47:54.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.860+0000 7fa062a30740 1 Processor -- start 2024-03-28T18:47:54.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.860+0000 7fa062a30740 1 -- start start 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281501c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28150d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2814652c0 con 0x55d2815018b0 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d28140dc90 con 0x55d281502260 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa062a30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d281461360 con 0x55d28150ade0 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa05de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281501c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa05ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28150d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46442/0 (socket says 172.21.15.119:46442) 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.861+0000 7fa05d400640 1 -- 172.21.15.119:0/717209488 learned_addr learned my addr 172.21.15.119:0/717209488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 -- 172.21.15.119:0/717209488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 msgr2=0x55d28150d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28150d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 -- 172.21.15.119:0/717209488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 msgr2=0x55d281501c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281501c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 -- 172.21.15.119:0/717209488 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d281432080 con 0x55d281502260 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa056000640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 0x7fa0400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05d400640 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0cca90 tx=0x7fa04c0d5a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80166cb6804a342 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa056000640 1 -- 172.21.15.119:0/605458481 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa04c0cf490 con 0x7fa058101530 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa05de00640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 0x7fa0400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa04a600640 1 -- 172.21.15.119:0/717209488 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c010e60 con 0x55d281502260 2024-03-28T18:47:54.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa04a600640 1 -- 172.21.15.119:0/717209488 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa04c09bae0 con 0x55d281502260 2024-03-28T18:47:54.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.862+0000 7fa04a600640 1 -- 172.21.15.119:0/717209488 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c09b620 con 0x55d281502260 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 -- 172.21.15.119:0/717209488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 msgr2=0x55d28150a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa05de00640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 0x7fa0400a4eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa050036040 tx=0x7fa05002df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0cca90 tx=0x7fa04c0d5a70 comp rx=0 tx=0).stop 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 -- 172.21.15.119:0/717209488 shutdown_connections 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28150d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d28150a810 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 --2- 172.21.15.119:0/717209488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281501c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 -- 172.21.15.119:0/717209488 >> 172.21.15.119:0/717209488 conn(0x55d2814fbf80 msgr2=0x55d2814fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 -- 172.21.15.119:0/717209488 shutdown_connections 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.863+0000 7fa062a30740 1 -- 172.21.15.119:0/717209488 wait complete. 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 -- start start 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 Processor -- start 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d281502260 0x55d281507af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d281508100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d28150dd50 con 0x55d281502260 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2814210d0 con 0x55d2815018b0 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d28150ded0 con 0x55d28150ade0 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa062a30740 10 client.0 _ll_register_callbacks cb 0x55d281425e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.864+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46446/0 (socket says 172.21.15.119:46446) 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 -- 172.21.15.119:0/942113164 learned_addr learned my addr 172.21.15.119:0/942113164 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05de00640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d281508100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05ca00640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d281502260 0x55d281507af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 -- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 msgr2=0x55d281508100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d281508100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 -- 172.21.15.119:0/942113164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d281502260 msgr2=0x55d281507af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d281502260 0x55d281507af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 -- 172.21.15.119:0/942113164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2811793f0 con 0x55d2815018b0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05de00640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d281508100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa05d400640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa04c09b7e0 tx=0x7fa04c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5cc2e2708a613f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa03ea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c010e60 con 0x55d2815018b0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa04c0ccda0 con 0x55d2815018b0 2024-03-28T18:47:54.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.865+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa04c09bae0 con 0x55d2815018b0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa04c09b440 con 0x55d2815018b0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 10 client.26570 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d2812437d0 con 0x55d2815018b0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa04c09bae0 con 0x55d2815018b0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa04c09bd40 con 0x55d2815018b0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 10 client.26570 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 10 client.26570 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 20 client.26570 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28151ab70 0x55d28151cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 4 client.26570 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.866+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d281472800 con 0x55d28151ab70 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.867+0000 7fa05ca00640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28151ab70 0x55d28151cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.867+0000 7fa05ca00640 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28151ab70 0x55d28151cf60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b5feacf9ee2be45 server_cookie=9b98a2af95618206 in_seq=0 out_seq=0 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.867+0000 7fa03ea00640 10 client.26570 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:54.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa03ea00640 1 -- 172.21.15.119:0/942113164 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d281472800 con 0x55d28151ab70 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa03ea00640 10 client.26570 handle_command_reply: tid=0 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 client.26570 shutdown 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 msgr2=0x55d28150a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa04c09b7e0 tx=0x7fa04c000f40 comp rx=0 tx=0).stop 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 shutdown_connections 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28151ab70 0x55d28151cf60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d281508100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d281502260 0x55d281507af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa03ea00640 0 client.26570 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 --2- 172.21.15.119:0/942113164 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d28150a760 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 >> 172.21.15.119:0/942113164 conn(0x55d2814fbf80 msgr2=0x55d2814fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 shutdown_connections 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 1 -- 172.21.15.119:0/942113164 wait complete. 2024-03-28T18:47:54.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.868+0000 7fa062a30740 20 client.26570 trim_cache size 0 max 16384 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 Processor -- start 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 -- start start 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d281502260 0x55d281506b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d28150ade0 0x55d28150a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581af310 con 0x55d28150ade0 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581b0c90 con 0x55d2815018b0 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa062a30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581af970 con 0x55d281502260 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa05d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46452/0 (socket says 172.21.15.119:46452) 2024-03-28T18:47:54.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa05d400640 1 -- 172.21.15.119:0/156330060 learned_addr learned my addr 172.21.15.119:0/156330060 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.882+0000 7fa05ca00640 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d281502260 0x55d281506b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05de00640 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d28150ade0 0x55d28150a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 -- 172.21.15.119:0/156330060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d281502260 msgr2=0x55d281506b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d281502260 0x55d281506b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 -- 172.21.15.119:0/156330060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d28150ade0 msgr2=0x55d28150a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d28150ade0 0x55d28150a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 -- 172.21.15.119:0/156330060 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d28151d2d0 con 0x55d2815018b0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa05d400640 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0984b0 tx=0x7fa04c099ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a515f8cffa32aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa049c00640 1 -- 172.21.15.119:0/156330060 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c010e60 con 0x55d2815018b0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa049c00640 1 -- 172.21.15.119:0/156330060 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa04c0ba2e0 con 0x55d2815018b0 2024-03-28T18:47:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.883+0000 7fa049c00640 1 -- 172.21.15.119:0/156330060 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c010ac0 con 0x55d2815018b0 2024-03-28T18:47:54.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- 172.21.15.119:0/156330060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 msgr2=0x55d2814f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0984b0 tx=0x7fa04c099ff0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- 172.21.15.119:0/156330060 shutdown_connections 2024-03-28T18:47:54.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d28150ade0 0x55d28150a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d281502260 0x55d281506b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 --2- 172.21.15.119:0/156330060 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d2815018b0 0x55d2814f70a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- 172.21.15.119:0/156330060 >> 172.21.15.119:0/156330060 conn(0x55d2814fbf80 msgr2=0x55d2814fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- 172.21.15.119:0/156330060 shutdown_connections 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- 172.21.15.119:0/156330060 wait complete. 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 -- start start 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.884+0000 7fa062a30740 1 Processor -- start 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28157d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581b0630 con 0x55d2815018b0 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581affd0 con 0x55d281502260 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0581ad4d0 con 0x55d28150ade0 2024-03-28T18:47:54.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa062a30740 10 client.0 _ll_register_callbacks cb 0x7fa04c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37244/0 (socket says 172.21.15.119:37244) 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46466/0 (socket says 172.21.15.119:46466) 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 -- 172.21.15.119:0/3434821842 learned_addr learned my addr 172.21.15.119:0/3434821842 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05de00640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28157d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 -- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 msgr2=0x55d28157d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28157d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 -- 172.21.15.119:0/3434821842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 msgr2=0x55d281510d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05d400640 1 -- 172.21.15.119:0/3434821842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d281511600 con 0x55d2815018b0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.885+0000 7fa05ca00640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa05d400640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa04c09fba0 tx=0x7fa04c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88df238ebfb48aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:54.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa03ea00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c080060 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa04c0a3f40 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa04c0ba2e0 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa04c010e60 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa062a30740 10 client.16353 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d281511600 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.886+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa04c0ba2e0 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa04c097830 con 0x55d2815018b0 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 10 client.16353 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 10 client.16353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 20 client.16353 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28157f170 0x55d281581560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 4 client.16353 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d28146cb50 con 0x55d28157f170 2024-03-28T18:47:54.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa05ca00640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28157f170 0x55d281581560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa03ea00640 10 client.16353 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.887+0000 7fa05ca00640 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28157f170 0x55d281581560 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=829715ed0fe6ef01 server_cookie=8594c4803d90a190 in_seq=0 out_seq=0 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa03ea00640 1 -- 172.21.15.119:0/3434821842 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d28146cb50 con 0x55d28157f170 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa03ea00640 10 client.16353 handle_command_reply: tid=0 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 client.16353 shutdown 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 msgr2=0x55d281510830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa04c09fba0 tx=0x7fa04c00cf90 comp rx=0 tx=0).stop 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 shutdown_connections 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55d28157f170 0x55d281581560 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d28150ade0 0x55d28157d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa03ea00640 0 client.16353 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d281502260 0x55d281510d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 --2- 172.21.15.119:0/3434821842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d2815018b0 0x55d281510830 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 >> 172.21.15.119:0/3434821842 conn(0x55d2814fbf80 msgr2=0x55d281506290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 shutdown_connections 2024-03-28T18:47:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 1 -- 172.21.15.119:0/3434821842 wait complete. 2024-03-28T18:47:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.888+0000 7fa062a30740 20 client.16353 trim_cache size 0 max 16384 2024-03-28T18:47:54.893 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 msgr2=0x7fa0400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 0x7fa0400a4eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa050036040 tx=0x7fa05002df90 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 msgr2=0x7fa058199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00ebc0 tx=0x7fa04c00c6a0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 shutdown_connections 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0400a29f0 0x7fa0400a4eb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0581075f0 0x7fa05819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa058101ef0 0x7fa05819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 --2- 172.21.15.119:0/605458481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa058101530 0x7fa058199f90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 >> 172.21.15.119:0/605458481 conn(0x7fa0580fd000 msgr2=0x7fa0580fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 shutdown_connections 2024-03-28T18:47:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:54.889+0000 7fa05e800640 1 -- 172.21.15.119:0/605458481 wait complete. 2024-03-28T18:47:54.908 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:47:54.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:47:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:54 smithi196 ceph-mon[36300]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:54 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3249882587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:54 smithi131 ceph-mon[36065]: pgmap v118: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-03-28T18:47:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:54 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3249882587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.121+0000 7fe15f200640 1 -- 172.21.15.119:0/1596832271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe158073ad0 msgr2=0x7fe158073eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.121+0000 7fe15f200640 1 --2- 172.21.15.119:0/1596832271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe158073ad0 0x7fe158073eb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe148009860 tx=0x7fe14802bc00 comp rx=0 tx=0).stop 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 -- 172.21.15.119:0/1596832271 shutdown_connections 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 --2- 172.21.15.119:0/1596832271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe15810de70 0x7fe1581102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 --2- 172.21.15.119:0/1596832271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1580743f0 0x7fe15810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 --2- 172.21.15.119:0/1596832271 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe158073ad0 0x7fe158073eb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 -- 172.21.15.119:0/1596832271 >> 172.21.15.119:0/1596832271 conn(0x7fe158078aa0 msgr2=0x7fe158078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 -- 172.21.15.119:0/1596832271 shutdown_connections 2024-03-28T18:47:55.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 -- 172.21.15.119:0/1596832271 wait complete. 2024-03-28T18:47:55.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.122+0000 7fe15f200640 1 Processor -- start 2024-03-28T18:47:55.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 -- start start 2024-03-28T18:47:55.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe158073ad0 0x7fe15819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1580743f0 0x7fe15819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158113c10 con 0x7fe158073ad0 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158113a90 con 0x7fe1580743f0 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15f200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe158113d90 con 0x7fe15810de70 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe157400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1580743f0 0x7fe15819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47144/0 (socket says 172.21.15.119:47144) 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe157e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe158073ad0 0x7fe15819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.123+0000 7fe15d200640 1 -- 172.21.15.119:0/2237478466 learned_addr learned my addr 172.21.15.119:0/2237478466 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 -- 172.21.15.119:0/2237478466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1580743f0 msgr2=0x7fe15819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1580743f0 0x7fe15819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 -- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe158073ad0 msgr2=0x7fe15819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe158073ad0 0x7fe15819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 -- 172.21.15.119:0/2237478466 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15819f120 con 0x7fe15810de70 2024-03-28T18:47:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15d200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe15000eb10 tx=0x7fe15000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe154c00640 1 -- 172.21.15.119:0/2237478466 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150004b20 con 0x7fe15810de70 2024-03-28T18:47:55.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15819f410 con 0x7fe15810de70 2024-03-28T18:47:55.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe154c00640 1 -- 172.21.15.119:0/2237478466 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe150004cc0 con 0x7fe15810de70 2024-03-28T18:47:55.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe154c00640 1 -- 172.21.15.119:0/2237478466 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150010760 con 0x7fe15810de70 2024-03-28T18:47:55.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.124+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1581a6ca0 con 0x7fe15810de70 2024-03-28T18:47:55.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.127+0000 7fe154c00640 1 -- 172.21.15.119:0/2237478466 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe150005020 con 0x7fe15810de70 2024-03-28T18:47:55.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 Processor -- start 2024-03-28T18:47:55.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 -- start start 2024-03-28T18:47:55.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c47ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c53050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13bab2c0 con 0x557c13c476d0 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13b53c90 con 0x557c13c48080 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.128+0000 7fe161f81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13ba7360 con 0x557c13c50c00 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c47ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46482/0 (socket says 172.21.15.119:46482) 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 -- 172.21.15.119:0/1698362398 learned_addr learned my addr 172.21.15.119:0/1698362398 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe154c00640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 0x7fe13c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 -- 172.21.15.119:0/1698362398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 msgr2=0x557c13c53050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157400640 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c53050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c53050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 -- 172.21.15.119:0/1698362398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 msgr2=0x557c13c47ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe154c00640 1 -- 172.21.15.119:0/2237478466 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe1500cf430 con 0x7fe15810de70 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c47ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 -- 172.21.15.119:0/1698362398 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c13b78080 con 0x557c13c48080 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157400640 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c53050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe15d200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 0x7fe13c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.129+0000 7fe157e00640 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe148009780 tx=0x7fe148009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc5eead7a5c2298 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe14d600640 1 -- 172.21.15.119:0/1698362398 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148043020 con 0x557c13c48080 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe14d600640 1 -- 172.21.15.119:0/1698362398 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe14d600640 1 -- 172.21.15.119:0/1698362398 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148039040 con 0x557c13c48080 2024-03-28T18:47:55.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe161f81740 1 -- 172.21.15.119:0/1698362398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 msgr2=0x557c13c50630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe15d200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 0x7fe13c0a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1500ccb20 tx=0x7fe150005b40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.130+0000 7fe161f81740 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe148009780 tx=0x7fe148009510 comp rx=0 tx=0).stop 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 -- 172.21.15.119:0/1698362398 shutdown_connections 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c53050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50630 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 --2- 172.21.15.119:0/1698362398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 -- 172.21.15.119:0/1698362398 >> 172.21.15.119:0/1698362398 conn(0x557c13c41da0 msgr2=0x557c13c42180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 -- 172.21.15.119:0/1698362398 shutdown_connections 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 -- 172.21.15.119:0/1698362398 wait complete. 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 -- start start 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.131+0000 7fe161f81740 1 Processor -- start 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c50560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c4df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13c53b10 con 0x557c13c48080 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13b670d0 con 0x557c13c50c00 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c13c53c90 con 0x557c13c476d0 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe161f81740 10 client.0 _ll_register_callbacks cb 0x557c13b6be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c50560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37286/0 (socket says 172.21.15.119:37286) 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 -- 172.21.15.119:0/2933988250 learned_addr learned my addr 172.21.15.119:0/2933988250 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157400640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c4df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 -- 172.21.15.119:0/2933988250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 msgr2=0x557c13c50560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c50560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 -- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 msgr2=0x557c13c4df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c4df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157e00640 1 -- 172.21.15.119:0/2933988250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c138bf3f0 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.132+0000 7fe157400640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c4df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe157e00640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe148037040 tx=0x7fe148034980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbacee103dd31690 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe142000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148043030 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe148004240 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe14805a050 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe161f81740 10 client.16365 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.133+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c139897d0 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe148051070 con 0x557c13c48080 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 10 client.16365 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:55.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 20 client.16365 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13c608c0 0x557c13c62cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 4 client.16365 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c13bb8800 con 0x557c13c608c0 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.134+0000 7fe157400640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13c608c0 0x557c13c62cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.135+0000 7fe157400640 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13c608c0 0x557c13c62cb0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331b860b5b8d598c server_cookie=374022cf4a5afa74 in_seq=0 out_seq=0 2024-03-28T18:47:55.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.135+0000 7fe142000640 10 client.16365 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:55.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.135+0000 7fe142000640 1 -- 172.21.15.119:0/2933988250 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c13bb8800 con 0x557c13c608c0 2024-03-28T18:47:55.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.135+0000 7fe142000640 10 client.16365 handle_command_reply: tid=0 2024-03-28T18:47:55.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.135+0000 7fe161f81740 1 client.16365 shutdown 2024-03-28T18:47:55.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 msgr2=0x557c13c4d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe148037040 tx=0x7fe148034980 comp rx=0 tx=0).stop 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 shutdown_connections 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13c608c0 0x557c13c62cb0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c4df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c4d8f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe142000640 0 client.16365 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 --2- 172.21.15.119:0/2933988250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c50560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 >> 172.21.15.119:0/2933988250 conn(0x557c13c41da0 msgr2=0x557c13c43af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 shutdown_connections 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 1 -- 172.21.15.119:0/2933988250 wait complete. 2024-03-28T18:47:55.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.136+0000 7fe161f81740 20 client.16365 trim_cache size 0 max 16384 2024-03-28T18:47:55.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.149+0000 7fe161f81740 1 Processor -- start 2024-03-28T18:47:55.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.149+0000 7fe161f81740 1 -- start start 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c5cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c50560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581af530 con 0x557c13c476d0 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581b0eb0 con 0x557c13c48080 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe161f81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581afb90 con 0x557c13c50c00 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c5cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c50560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46498/0 (socket says 172.21.15.119:46498) 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157e00640 1 -- 172.21.15.119:0/1483644340 learned_addr learned my addr 172.21.15.119:0/1483644340 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.150+0000 7fe157400640 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c50560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47174/0 (socket says 172.21.15.119:47174) 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 -- 172.21.15.119:0/1483644340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 msgr2=0x557c13c50560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c50560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 -- 172.21.15.119:0/1483644340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 msgr2=0x557c13c5cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c5cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 -- 172.21.15.119:0/1483644340 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c13c4e270 con 0x557c13c48080 2024-03-28T18:47:55.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe157e00640 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe14802c140 tx=0x7fe148003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95780b87a29c1087 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe14cc00640 1 -- 172.21.15.119:0/1483644340 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148063070 con 0x557c13c48080 2024-03-28T18:47:55.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe14cc00640 1 -- 172.21.15.119:0/1483644340 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.151+0000 7fe14cc00640 1 -- 172.21.15.119:0/1483644340 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148043040 con 0x557c13c48080 2024-03-28T18:47:55.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- 172.21.15.119:0/1483644340 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 msgr2=0x557c13c50020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe14802c140 tx=0x7fe148003e10 comp rx=0 tx=0).stop 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- 172.21.15.119:0/1483644340 shutdown_connections 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c50c00 0x557c13c50560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c48080 0x557c13c50020 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 --2- 172.21.15.119:0/1483644340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c476d0 0x557c13c5cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- 172.21.15.119:0/1483644340 >> 172.21.15.119:0/1483644340 conn(0x557c13c41da0 msgr2=0x557c13c40eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- 172.21.15.119:0/1483644340 shutdown_connections 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- 172.21.15.119:0/1483644340 wait complete. 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 -- start start 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.152+0000 7fe161f81740 1 Processor -- start 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c65930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c63920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581b0850 con 0x557c13c48080 2024-03-28T18:47:55.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581b01f0 con 0x557c13c50c00 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1581ad6f0 con 0x557c13c476d0 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe161f81740 10 client.0 _ll_register_callbacks cb 0x7fe144009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe157e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c65930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe157400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe157400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47188/0 (socket says 172.21.15.119:47188) 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe157400640 1 -- 172.21.15.119:0/2704326525 learned_addr learned my addr 172.21.15.119:0/2704326525 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.153+0000 7fe157400640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c63920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 -- 172.21.15.119:0/2704326525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 msgr2=0x557c13c653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 -- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 msgr2=0x557c13c63920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c63920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 -- 172.21.15.119:0/2704326525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c13c63fc0 con 0x557c13c48080 2024-03-28T18:47:55.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157400640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe157e00640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c65930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x557c13c5d860 tx=0x7fe148003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d999a632867951ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe142000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148063030 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe148003800 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe148043040 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe161f81740 10 client.16371 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.154+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c13c63fc0 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe148034440 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe148051070 con 0x557c13c48080 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 10 client.16371 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 20 client.16371 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:55.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13cbcb70 0x557c13cbef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 4 client.16371 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c13bb2b50 con 0x557c13cbcb70 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.155+0000 7fe157400640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13cbcb70 0x557c13cbef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe157400640 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13cbcb70 0x557c13cbef90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c72d34ebb5449bc server_cookie=10ccc1a2a246735b in_seq=0 out_seq=0 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe142000640 10 client.16371 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe142000640 1 -- 172.21.15.119:0/2704326525 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557c13bb2b50 con 0x557c13cbcb70 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe142000640 10 client.16371 handle_command_reply: tid=0 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 client.16371 shutdown 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 msgr2=0x557c13c65930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c65930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x557c13c5d860 tx=0x7fe148003e80 comp rx=0 tx=0).stop 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 shutdown_connections 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557c13cbcb70 0x557c13cbef90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557c13c50c00 0x557c13c63920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c13c48080 0x557c13c65930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 --2- 172.21.15.119:0/2704326525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c13c476d0 0x557c13c653f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe142000640 0 client.16371 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 >> 172.21.15.119:0/2704326525 conn(0x557c13c41da0 msgr2=0x557c13c48460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 shutdown_connections 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 1 -- 172.21.15.119:0/2704326525 wait complete. 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.156+0000 7fe161f81740 20 client.16371 trim_cache size 0 max 16384 2024-03-28T18:47:55.161 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 msgr2=0x7fe13c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 0x7fe13c0a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe1500ccb20 tx=0x7fe150005b40 comp rx=0 tx=0).stop 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 msgr2=0x7fe15819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe15000eb10 tx=0x7fe15000c2d0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 shutdown_connections 2024-03-28T18:47:55.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe13c0a2970 0x7fe13c0a4e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe15810de70 0x7fe15819e9f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1580743f0 0x7fe15819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 --2- 172.21.15.119:0/2237478466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe158073ad0 0x7fe15819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 >> 172.21.15.119:0/2237478466 conn(0x7fe158078aa0 msgr2=0x7fe1580fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 shutdown_connections 2024-03-28T18:47:55.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:55.157+0000 7fe15f200640 1 -- 172.21.15.119:0/2237478466 wait complete. 2024-03-28T18:47:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:55 smithi119 ceph-mon[31813]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-03-28T18:47:55.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:55 smithi196 ceph-mon[36300]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-03-28T18:47:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:55 smithi131 ceph-mon[36065]: pgmap v119: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-03-28T18:47:56.175 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:47:56.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:47:56.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:47:56.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 -- 172.21.15.119:0/3871747973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c073a70 msgr2=0x7fa41c073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 --2- 172.21.15.119:0/3871747973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c073a70 0x7fa41c073e50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009ca0 tx=0x7fa40c02c180 comp rx=0 tx=0).stop 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 -- 172.21.15.119:0/3871747973 shutdown_connections 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 --2- 172.21.15.119:0/3871747973 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c10ded0 0x7fa41c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 --2- 172.21.15.119:0/3871747973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 0x7fa41c10d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 --2- 172.21.15.119:0/3871747973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c073a70 0x7fa41c073e50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 -- 172.21.15.119:0/3871747973 >> 172.21.15.119:0/3871747973 conn(0x7fa41c078aa0 msgr2=0x7fa41c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 -- 172.21.15.119:0/3871747973 shutdown_connections 2024-03-28T18:47:56.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.399+0000 7fa422a00640 1 -- 172.21.15.119:0/3871747973 wait complete. 2024-03-28T18:47:56.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.400+0000 7fa422a00640 1 Processor -- start 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.400+0000 7fa422a00640 1 -- start start 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.400+0000 7fa422a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.400+0000 7fa422a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 0x7fa41c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.400+0000 7fa422a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 0x7fa41c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa422a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c113c50 con 0x7fa41c074390 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa422a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c113ad0 con 0x7fa41c073a70 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa422a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c113dd0 con 0x7fa41c10ded0 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa420a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 0x7fa41c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46516/0 (socket says 172.21.15.119:46516) 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 -- 172.21.15.119:0/1184958364 learned_addr learned my addr 172.21.15.119:0/1184958364 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41b400640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 0x7fa41c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 -- 172.21.15.119:0/1184958364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 msgr2=0x7fa41c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 0x7fa41c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 -- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 msgr2=0x7fa41c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 0x7fa41c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 -- 172.21.15.119:0/1184958364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c19f100 con 0x7fa41c073a70 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa420a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 0x7fa41c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.401+0000 7fa41be00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005e30 tx=0x7fa40c035d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.402+0000 7fa418c00640 1 -- 172.21.15.119:0/1184958364 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c039040 con 0x7fa41c073a70 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.402+0000 7fa418c00640 1 -- 172.21.15.119:0/1184958364 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa40c044070 con 0x7fa41c073a70 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.402+0000 7fa418c00640 1 -- 172.21.15.119:0/1184958364 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0328d0 con 0x7fa41c073a70 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.402+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c19f390 con 0x7fa41c073a70 2024-03-28T18:47:56.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.402+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c1a6c80 con 0x7fa41c073a70 2024-03-28T18:47:56.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa418c00640 1 -- 172.21.15.119:0/1184958364 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa40c032a70 con 0x7fa41c073a70 2024-03-28T18:47:56.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 Processor -- start 2024-03-28T18:47:56.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 -- start start 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a775e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a7767040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a76c2eb0 con 0x5574a775e080 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a766ba60 con 0x5574a775e9a0 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.405+0000 7fa4257e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a76bef50 con 0x5574a7767580 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa420a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a775e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a7767040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47210/0 (socket says 172.21.15.119:47210) 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa420a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a775e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37330/0 (socket says 172.21.15.119:37330) 2024-03-28T18:47:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 -- 172.21.15.119:0/3830533581 learned_addr learned my addr 172.21.15.119:0/3830533581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 -- 172.21.15.119:0/3830533581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 msgr2=0x5574a7767040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a7767040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 -- 172.21.15.119:0/3830533581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 msgr2=0x5574a775e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a775e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 -- 172.21.15.119:0/3830533581 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574a749f7e0 con 0x5574a7767580 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa41b400640 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa40400bc90 tx=0x7fa4040107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4114df5c7e8e58a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.406+0000 7fa411200640 1 -- 172.21.15.119:0/3830533581 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404005220 con 0x5574a7767580 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa411200640 1 -- 172.21.15.119:0/3830533581 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa40400be40 con 0x5574a7767580 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa411200640 1 -- 172.21.15.119:0/3830533581 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404011770 con 0x5574a7767580 2024-03-28T18:47:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa418c00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 0x7fa4000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa418c00640 1 -- 172.21.15.119:0/1184958364 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa40c0f0f90 con 0x7fa41c073a70 2024-03-28T18:47:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa420a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 0x7fa4000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 -- 172.21.15.119:0/3830533581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 msgr2=0x5574a7769970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa40400bc90 tx=0x7fa4040107a0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 -- 172.21.15.119:0/3830533581 shutdown_connections 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a7769970 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a7767040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 --2- 172.21.15.119:0/3830533581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a775e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.407+0000 7fa4257e7740 1 -- 172.21.15.119:0/3830533581 >> 172.21.15.119:0/3830533581 conn(0x5574a77587b0 msgr2=0x5574a7758b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.408+0000 7fa4257e7740 1 -- 172.21.15.119:0/3830533581 shutdown_connections 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.408+0000 7fa4257e7740 1 -- 172.21.15.119:0/3830533581 wait complete. 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.408+0000 7fa420a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 0x7fa4000a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa41c06af40 tx=0x7fa414005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.408+0000 7fa4257e7740 1 -- start start 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.408+0000 7fa4257e7740 1 Processor -- start 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 0x5574a7764180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a7767580 0x5574a77646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a776a370 con 0x5574a775e9a0 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a767eeb0 con 0x5574a7767580 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574a776a4f0 con 0x5574a775e080 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa4257e7740 10 client.0 _ll_register_callbacks cb 0x5574a7673150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 0x5574a7764180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47218/0 (socket says 172.21.15.119:47218) 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 -- 172.21.15.119:0/775589830 learned_addr learned my addr 172.21.15.119:0/775589830 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa420a00640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a7767580 0x5574a77646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 -- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a7767580 msgr2=0x5574a77646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a7767580 0x5574a77646c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 -- 172.21.15.119:0/775589830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 msgr2=0x5574a7764180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 0x5574a7764180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41be00640 1 -- 172.21.15.119:0/775589830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574a74fff70 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.409+0000 7fa41b400640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 0x5574a7764180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa41be00640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0ee7f0 tx=0x7fa40c0b9340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9128c1d550faa8a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa409600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0bd1e0 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa40c0f4c90 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574a76d03f0 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa40c0bd000 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa4257e7740 10 client.26587 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5574a7397170 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.410+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5574a76d03f0 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa40c03c030 con 0x5574a775e080 2024-03-28T18:47:56.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 10 client.26587 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 20 client.26587 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77774b0 0x5574a77798f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 4 client.26587 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574a76ca740 con 0x5574a77774b0 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa41b400640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77774b0 0x5574a77798f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa41b400640 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77774b0 0x5574a77798f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23bea9c7e5fed221 server_cookie=27e053d0d1bc97f8 in_seq=0 out_seq=0 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.411+0000 7fa409600640 10 client.26587 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa409600640 1 -- 172.21.15.119:0/775589830 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5574a76ca740 con 0x5574a77774b0 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa409600640 10 client.26587 handle_command_reply: tid=0 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 client.26587 shutdown 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 msgr2=0x5574a7766df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0ee7f0 tx=0x7fa40c0b9340 comp rx=0 tx=0).stop 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 shutdown_connections 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77774b0 0x5574a77798f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a7767580 0x5574a77646c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e9a0 0x5574a7764180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa409600640 0 client.26587 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 --2- 172.21.15.119:0/775589830 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e080 0x5574a7766df0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.412+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 >> 172.21.15.119:0/775589830 conn(0x5574a77587b0 msgr2=0x5574a775a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.413+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 shutdown_connections 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.413+0000 7fa4257e7740 1 -- 172.21.15.119:0/775589830 wait complete. 2024-03-28T18:47:56.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.413+0000 7fa4257e7740 20 client.26587 trim_cache size 0 max 16384 2024-03-28T18:47:56.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 Processor -- start 2024-03-28T18:47:56.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 -- start start 2024-03-28T18:47:56.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e080 0x5574a77576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 0x5574a7766df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1af4e0 con 0x5574a7767580 2024-03-28T18:47:56.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1b0e60 con 0x5574a775e080 2024-03-28T18:47:56.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.430+0000 7fa4257e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1afb40 con 0x5574a775e9a0 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47224/0 (socket says 172.21.15.119:47224) 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 -- 172.21.15.119:0/1214341579 learned_addr learned my addr 172.21.15.119:0/1214341579 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41be00640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e080 0x5574a77576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa420a00640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 0x5574a7766df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 -- 172.21.15.119:0/1214341579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e080 msgr2=0x5574a77576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e080 0x5574a77576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 -- 172.21.15.119:0/1214341579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 msgr2=0x5574a7766df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 0x5574a7766df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 -- 172.21.15.119:0/1214341579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574a7753c20 con 0x5574a775e9a0 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa420a00640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 0x5574a7766df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.431+0000 7fa41b400640 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5574a7759940 tx=0x7fa404005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a1000cef90ae7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa40be00640 1 -- 172.21.15.119:0/1214341579 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404004020 con 0x5574a775e9a0 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa40be00640 1 -- 172.21.15.119:0/1214341579 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4040041c0 con 0x5574a775e9a0 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa40be00640 1 -- 172.21.15.119:0/1214341579 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40402ab50 con 0x5574a775e9a0 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 -- 172.21.15.119:0/1214341579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 msgr2=0x5574a77668b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5574a7759940 tx=0x7fa404005780 comp rx=0 tx=0).stop 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 -- 172.21.15.119:0/1214341579 shutdown_connections 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a7767580 0x5574a7766df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a775e9a0 0x5574a77668b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 --2- 172.21.15.119:0/1214341579 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e080 0x5574a77576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 -- 172.21.15.119:0/1214341579 >> 172.21.15.119:0/1214341579 conn(0x5574a77587b0 msgr2=0x5574a7756fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 -- 172.21.15.119:0/1214341579 shutdown_connections 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.432+0000 7fa4257e7740 1 -- 172.21.15.119:0/1214341579 wait complete. 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.433+0000 7fa4257e7740 1 -- start start 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.433+0000 7fa4257e7740 1 Processor -- start 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.433+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a7761e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.433+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa4257e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a776ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa4257e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1b0800 con 0x5574a775e080 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa4257e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1b01a0 con 0x5574a775e9a0 2024-03-28T18:47:56.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa4257e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1ad6a0 con 0x5574a7767580 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa4257e7740 10 client.0 _ll_register_callbacks cb 0x7fa40c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46546/0 (socket says 172.21.15.119:46546) 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a7761e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 -- 172.21.15.119:0/546905021 learned_addr learned my addr 172.21.15.119:0/546905021 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa420a00640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a776ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 -- 172.21.15.119:0/546905021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 msgr2=0x5574a776ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a776ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 -- 172.21.15.119:0/546905021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 msgr2=0x5574a7761e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a7761e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 -- 172.21.15.119:0/546905021 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574a776d3c0 con 0x5574a775e9a0 2024-03-28T18:47:56.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41be00640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a7761e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa41b400640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa40400bc60 tx=0x7fa404005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d62c414a8e3dd47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.434+0000 7fa409600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404038020 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa40402a170 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4040041c0 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa404043050 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 10 client.26612 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5574a77d3d10 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4040041c0 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa404004660 con 0x5574a775e9a0 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 10 client.26612 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 10 client.26612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:56.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 20 client.26612 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77d42a0 0x5574a77d66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 4 client.26612 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.435+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5574a7648a40 con 0x5574a77d42a0 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.436+0000 7fa41be00640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77d42a0 0x5574a77d66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.436+0000 7fa41be00640 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77d42a0 0x5574a77d66c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d7bdfe3898372d server_cookie=1e5bc602bcdb26a9 in_seq=0 out_seq=0 2024-03-28T18:47:56.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.436+0000 7fa409600640 10 client.26612 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.437+0000 7fa409600640 1 -- 172.21.15.119:0/546905021 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5574a7648a40 con 0x5574a77d42a0 2024-03-28T18:47:56.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.437+0000 7fa409600640 10 client.26612 handle_command_reply: tid=0 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.437+0000 7fa4257e7740 1 client.26612 shutdown 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 msgr2=0x5574a776c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa40400bc60 tx=0x7fa404005ed0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 shutdown_connections 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5574a77d42a0 0x5574a77d66c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5574a7767580 0x5574a776ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5574a775e9a0 0x5574a776c770 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 --2- 172.21.15.119:0/546905021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574a775e080 0x5574a7761e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 >> 172.21.15.119:0/546905021 conn(0x5574a77587b0 msgr2=0x5574a7757830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa409600640 0 client.26612 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 shutdown_connections 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 1 -- 172.21.15.119:0/546905021 wait complete. 2024-03-28T18:47:56.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.438+0000 7fa4257e7740 20 client.26612 trim_cache size 0 max 16384 2024-03-28T18:47:56.444 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:47:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "4bc3b72d-b6a5-4df7-89f1-db3a76c08562", 2024-03-28T18:47:56.444 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:47:56.444 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 msgr2=0x7fa4000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 0x7fa4000a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa41c06af40 tx=0x7fa414005f90 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 msgr2=0x7fa41c19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa40c005e30 tx=0x7fa40c035d70 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 shutdown_connections 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa4000a2970 0x7fa4000a4e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa41c10ded0 0x7fa41c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa41c074390 0x7fa41c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 --2- 172.21.15.119:0/1184958364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa41c073a70 0x7fa41c19a110 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 >> 172.21.15.119:0/1184958364 conn(0x7fa41c078aa0 msgr2=0x7fa41c0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 shutdown_connections 2024-03-28T18:47:56.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.439+0000 7fa422a00640 1 -- 172.21.15.119:0/1184958364 wait complete. 2024-03-28T18:47:56.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:47:56.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 -- 172.21.15.119:0/2045789938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101530 msgr2=0x7f8ac01019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 --2- 172.21.15.119:0/2045789938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101530 0x7f8ac01019b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009930 tx=0x7f8aac02bc30 comp rx=0 tx=0).stop 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 -- 172.21.15.119:0/2045789938 shutdown_connections 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 --2- 172.21.15.119:0/2045789938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac0101ef0 0x7f8ac010e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 --2- 172.21.15.119:0/2045789938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101530 0x7f8ac01019b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 --2- 172.21.15.119:0/2045789938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac01075f0 0x7f8ac01079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 -- 172.21.15.119:0/2045789938 >> 172.21.15.119:0/2045789938 conn(0x7f8ac00fd000 msgr2=0x7f8ac00ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 -- 172.21.15.119:0/2045789938 shutdown_connections 2024-03-28T18:47:56.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.657+0000 7f8ac6200640 1 -- 172.21.15.119:0/2045789938 wait complete. 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.658+0000 7f8ac6200640 1 Processor -- start 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.658+0000 7f8ac6200640 1 -- start start 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0101530 0x7f8ac00739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac01075f0 0x7f8ac0074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac0111900 con 0x7f8ac0101530 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac0111780 con 0x7f8ac01075f0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac6200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac0111a80 con 0x7f8ac0101ef0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47258/0 (socket says 172.21.15.119:47258) 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0101530 0x7f8ac00739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 -- 172.21.15.119:0/3929986866 learned_addr learned my addr 172.21.15.119:0/3929986866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8ac5800640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac01075f0 0x7f8ac0074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 -- 172.21.15.119:0/3929986866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac01075f0 msgr2=0x7f8ac0074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac01075f0 0x7f8ac0074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 -- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0101530 msgr2=0x7f8ac00739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0101530 0x7f8ac00739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.659+0000 7f8abfe00640 1 -- 172.21.15.119:0/3929986866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac01a6280 con 0x7f8ac0101ef0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8abfe00640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009900 tx=0x7f8aac004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8abd600640 1 -- 172.21.15.119:0/3929986866 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac043020 con 0x7f8ac0101ef0 2024-03-28T18:47:56.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8abd600640 1 -- 172.21.15.119:0/3929986866 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8aac0054e0 con 0x7f8ac0101ef0 2024-03-28T18:47:56.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8abd600640 1 -- 172.21.15.119:0/3929986866 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac003d00 con 0x7f8ac0101ef0 2024-03-28T18:47:56.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac01a6510 con 0x7f8ac0101ef0 2024-03-28T18:47:56.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.660+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac01a6a10 con 0x7f8ac0101ef0 2024-03-28T18:47:56.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.663+0000 7f8aca5b0740 1 Processor -- start 2024-03-28T18:47:56.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.663+0000 7f8abd600640 1 -- 172.21.15.119:0/3929986866 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8aac046020 con 0x7f8ac0101ef0 2024-03-28T18:47:56.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.663+0000 7f8aca5b0740 1 -- start start 2024-03-28T18:47:56.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be11460 0x555b5be13850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5bd6c250 con 0x555b5be08dd0 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5bd14a30 con 0x555b5be080f0 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5bd682f0 con 0x555b5be11460 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8ac5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be11460 0x555b5be13850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8ac5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37378/0 (socket says 172.21.15.119:37378) 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:46574/0 (socket says 172.21.15.119:46574) 2024-03-28T18:47:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.664+0000 7f8ac5800640 1 -- 172.21.15.119:0/1743238818 learned_addr learned my addr 172.21.15.119:0/1743238818 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 -- 172.21.15.119:0/1743238818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be11460 msgr2=0x555b5be13850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be11460 0x555b5be13850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 -- 172.21.15.119:0/1743238818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 msgr2=0x555b5be07b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 -- 172.21.15.119:0/1743238818 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b5bcff690 con 0x555b5be080f0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac5800640 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400bb70 tx=0x7f8ab40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ea25e2651432f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8abd600640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 0x7f8aa80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ab2000640 1 -- 172.21.15.119:0/1743238818 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4004030 con 0x555b5be080f0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ab2000640 1 -- 172.21.15.119:0/1743238818 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab40041d0 con 0x555b5be080f0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ab2000640 1 -- 172.21.15.119:0/1743238818 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4011680 con 0x555b5be080f0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8abd600640 1 -- 172.21.15.119:0/3929986866 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8aac0f0f70 con 0x7f8ac0101ef0 2024-03-28T18:47:56.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.665+0000 7f8ac5800640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 0x7f8aa80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 -- 172.21.15.119:0/1743238818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 msgr2=0x555b5be08570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400bb70 tx=0x7f8ab40107a0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8ac5800640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 0x7f8aa80a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005f40 tx=0x7f8ab8009600 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 -- 172.21.15.119:0/1743238818 shutdown_connections 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be11460 0x555b5be13850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be080f0 0x555b5be08570 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/1743238818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 -- 172.21.15.119:0/1743238818 >> 172.21.15.119:0/1743238818 conn(0x555b5bdcf200 msgr2=0x555b5be02b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 -- 172.21.15.119:0/1743238818 shutdown_connections 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.666+0000 7f8aca5b0740 1 -- 172.21.15.119:0/1743238818 wait complete. 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.667+0000 7f8aca5b0740 1 -- start start 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.667+0000 7f8aca5b0740 1 Processor -- start 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.667+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.667+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be0df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5be142d0 con 0x555b5be08dd0 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5bd27e80 con 0x555b5be11460 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b5be14450 con 0x555b5be080f0 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aca5b0740 10 client.0 _ll_register_callbacks cb 0x555b5bd3a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47270/0 (socket says 172.21.15.119:47270) 2024-03-28T18:47:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 -- 172.21.15.119:0/342103551 learned_addr learned my addr 172.21.15.119:0/342103551 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be0df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac5800640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 -- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 msgr2=0x555b5be0e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 -- 172.21.15.119:0/342103551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 msgr2=0x555b5be0df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be0df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 -- 172.21.15.119:0/342103551 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b5bb4a7d0 con 0x555b5be080f0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac5800640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4011010 tx=0x7f8ab400b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49204f88ac0dfac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.668+0000 7f8aa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab402a060 con 0x555b5be080f0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ab40041d0 con 0x555b5be080f0 2024-03-28T18:47:56.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8ab400bd20 con 0x555b5be080f0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ab40035f0 con 0x555b5be080f0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aca5b0740 10 client.26605 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555b5ba803f0 con 0x555b5be080f0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab40041d0 con 0x555b5be080f0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ab40047e0 con 0x555b5be080f0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aca5b0740 10 client.26605 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.669+0000 7f8aca5b0740 10 client.26605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8aca5b0740 20 client.26605 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be212e0 0x555b5be236d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8aca5b0740 4 client.26605 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b5bd79790 con 0x555b5be212e0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8abfe00640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be212e0 0x555b5be236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8abfe00640 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be212e0 0x555b5be236d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecc1587a602e07e6 server_cookie=984617a96260fb3c in_seq=0 out_seq=0 2024-03-28T18:47:56.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.670+0000 7f8aa6a00640 10 client.26605 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aa6a00640 1 -- 172.21.15.119:0/342103551 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b5bd79790 con 0x555b5be212e0 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aa6a00640 10 client.26605 handle_command_reply: tid=0 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 client.26605 shutdown 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 msgr2=0x555b5be10bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4011010 tx=0x7f8ab400b740 comp rx=0 tx=0).stop 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 shutdown_connections 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be212e0 0x555b5be236d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be08dd0 0x555b5be0df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aa6a00640 0 client.26605 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/342103551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10bd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 >> 172.21.15.119:0/342103551 conn(0x555b5bdcf200 msgr2=0x555b5be04890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 shutdown_connections 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 1 -- 172.21.15.119:0/342103551 wait complete. 2024-03-28T18:47:56.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.671+0000 7f8aca5b0740 20 client.26605 trim_cache size 0 max 16384 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 Processor -- start 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 -- start start 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 0x555b5be10bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be11460 0x555b5be12850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01af340 con 0x555b5be11460 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01b0cc0 con 0x555b5be08dd0 2024-03-28T18:47:56.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01af9a0 con 0x555b5be080f0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:47282/0 (socket says 172.21.15.119:47282) 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 0x555b5be10bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8ac4e00640 1 -- 172.21.15.119:0/213445648 learned_addr learned my addr 172.21.15.119:0/213445648 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.689+0000 7f8ac5800640 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be11460 0x555b5be12850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 -- 172.21.15.119:0/213445648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 msgr2=0x555b5be10bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 0x555b5be10bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 -- 172.21.15.119:0/213445648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be11460 msgr2=0x555b5be12850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be11460 0x555b5be12850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 -- 172.21.15.119:0/213445648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b5ba400f0 con 0x555b5be080f0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8abfe00640 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 0x555b5be10bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40107e0 tx=0x7f8ab400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45261c43fd7c452f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ab1600640 1 -- 172.21.15.119:0/213445648 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab402a040 con 0x555b5be080f0 2024-03-28T18:47:56.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ab1600640 1 -- 172.21.15.119:0/213445648 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4003c50 con 0x555b5be080f0 2024-03-28T18:47:56.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.690+0000 7f8ab1600640 1 -- 172.21.15.119:0/213445648 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4004f40 con 0x555b5be080f0 2024-03-28T18:47:56.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- 172.21.15.119:0/213445648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 msgr2=0x555b5be10690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40107e0 tx=0x7f8ab400b480 comp rx=0 tx=0).stop 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- 172.21.15.119:0/213445648 shutdown_connections 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be11460 0x555b5be12850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be08dd0 0x555b5be10bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/213445648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be080f0 0x555b5be10690 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- 172.21.15.119:0/213445648 >> 172.21.15.119:0/213445648 conn(0x555b5bdcf200 msgr2=0x555b5be02670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- 172.21.15.119:0/213445648 shutdown_connections 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- 172.21.15.119:0/213445648 wait complete. 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 -- start start 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.691+0000 7f8aca5b0740 1 Processor -- start 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be08dd0 0x555b5be0a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01b0660 con 0x555b5be080f0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01b0000 con 0x555b5be11460 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac01ad500 con 0x555b5be08dd0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8aca5b0740 10 client.0 _ll_register_callbacks cb 0x7f8ab8009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37400/0 (socket says 172.21.15.119:37400) 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 -- 172.21.15.119:0/2302348204 learned_addr learned my addr 172.21.15.119:0/2302348204 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac5800640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8abfe00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be08dd0 0x555b5be0a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 -- 172.21.15.119:0/2302348204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be08dd0 msgr2=0x555b5be0a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be08dd0 0x555b5be0a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 -- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 msgr2=0x555b5be0a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.692+0000 7f8ac4e00640 1 -- 172.21.15.119:0/2302348204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b5be0aff0 con 0x555b5be080f0 2024-03-28T18:47:56.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8ac4e00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4010e80 tx=0x7f8ab40042c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa1bcd8f663feb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab402a070 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8ab4004590 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ab4003c50 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ab4044050 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aca5b0740 10 client.16407 fetch_fsmap learned FSMap version 28 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555b5be7c7a0 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.693+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4003c50 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8ab4034020 con 0x555b5be080f0 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 10 client.16407 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 20 client.16407 populate_metadata read hostname 'smithi119' 2024-03-28T18:47:56.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be7cda0 0x555b5be7f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 4 client.16407 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555b5bd73ae0 con 0x555b5be7cda0 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.694+0000 7f8abfe00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be7cda0 0x555b5be7f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8abfe00640 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be7cda0 0x555b5be7f1c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15245b5ab74250a5 server_cookie=eb0c6136c756c07d in_seq=0 out_seq=0 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aa6a00640 10 client.16407 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aa6a00640 1 -- 172.21.15.119:0/2302348204 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555b5bd73ae0 con 0x555b5be7cda0 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aa6a00640 10 client.16407 handle_command_reply: tid=0 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 client.16407 shutdown 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 msgr2=0x555b5be0bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4010e80 tx=0x7f8ab40042c0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 shutdown_connections 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555b5be7cda0 0x555b5be7f1c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555b5be11460 0x555b5be0a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555b5be08dd0 0x555b5be0a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 --2- 172.21.15.119:0/2302348204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555b5be080f0 0x555b5be0bc50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aa6a00640 0 client.16407 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 >> 172.21.15.119:0/2302348204 conn(0x555b5bdcf200 msgr2=0x555b5be1c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 shutdown_connections 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 1 -- 172.21.15.119:0/2302348204 wait complete. 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.695+0000 7f8aca5b0740 20 client.16407 trim_cache size 0 max 16384 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:47:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: "4bc3b72d-b6a5-4df7-89f1-db3a76c08562": { 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "4bc3b72d-b6a5-4df7-89f1-db3a76c08562", 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:47:56.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 msgr2=0x7f8aa80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 0x7f8aa80a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005f40 tx=0x7f8ab8009600 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 msgr2=0x7f8ac0073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009900 tx=0x7f8aac004430 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 shutdown_connections 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8aa80a2970 0x7f8aa80a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8ac01075f0 0x7f8ac0074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac0101ef0 0x7f8ac0073ee0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 --2- 172.21.15.119:0/3929986866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0101530 0x7f8ac00739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 >> 172.21.15.119:0/3929986866 conn(0x7f8ac00fd000 msgr2=0x7f8ac00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 shutdown_connections 2024-03-28T18:47:56.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:47:56.696+0000 7f8ac6200640 1 -- 172.21.15.119:0/3929986866 wait complete. 2024-03-28T18:47:56.713 INFO:tasks.cephfs.filesystem:scrub status for tag:4bc3b72d-b6a5-4df7-89f1-db3a76c08562 - {'path': '/', 'tag': '4bc3b72d-b6a5-4df7-89f1-db3a76c08562', 'options': 'recursive,force'} 2024-03-28T18:47:57.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:57 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:47:57.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:57 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:57.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:57 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:47:57.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:57 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:57.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:57 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:47:57.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:57 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:57.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:57 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:47:57.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:57 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:57 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:47:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:57 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:57 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:47:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:57 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:47:58.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:58 smithi196 ceph-mon[36300]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:47:58.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:47:58 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:47:58.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:58 smithi131 ceph-mon[36065]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:47:58.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:47:58 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:47:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:58 smithi119 ceph-mon[31813]: pgmap v120: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-28T18:47:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:47:58 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:48:00.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:00 smithi196 ceph-mon[36300]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:48:00.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:00 smithi131 ceph-mon[36065]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:48:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:00 smithi119 ceph-mon[31813]: pgmap v121: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-28T18:48:02.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:02 smithi196 ceph-mon[36300]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:02 smithi131 ceph-mon[36065]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:02 smithi119 ceph-mon[31813]: pgmap v122: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:04.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:04 smithi196 ceph-mon[36300]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:04.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:04 smithi131 ceph-mon[36065]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:04 smithi119 ceph-mon[31813]: pgmap v123: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:48:06.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:06 smithi196 ceph-mon[36300]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:06.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:06 smithi131 ceph-mon[36065]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:06.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:06 smithi119 ceph-mon[31813]: pgmap v124: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:08.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:08 smithi196 ceph-mon[36300]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:08.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:08 smithi131 ceph-mon[36065]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:08 smithi119 ceph-mon[31813]: pgmap v125: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:10.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:10 smithi196 ceph-mon[36300]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:10.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:10 smithi131 ceph-mon[36065]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:10 smithi119 ceph-mon[31813]: pgmap v126: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:12.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:12 smithi196 ceph-mon[36300]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:12.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:12 smithi131 ceph-mon[36065]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:12 smithi119 ceph-mon[31813]: pgmap v127: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:14.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:14 smithi196 ceph-mon[36300]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:14.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:14 smithi131 ceph-mon[36065]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:14 smithi119 ceph-mon[31813]: pgmap v128: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:48:16.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:16 smithi131 ceph-mon[36065]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:16 smithi119 ceph-mon[31813]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:16 smithi196 ceph-mon[36300]: pgmap v129: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:48:17.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:17 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:17 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:17.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:17 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:48:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:18 smithi119 ceph-mon[31813]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:48:18.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:18 smithi196 ceph-mon[36300]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:48:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:18 smithi131 ceph-mon[36065]: pgmap v130: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:48:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:19 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:19.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:19 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:19.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:19 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:48:19.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:19 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:19.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:19 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:19.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:19 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:48:19.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:19 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:48:19.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:19 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:48:19.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:19 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:48:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:20 smithi119 ceph-mon[31813]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:48:20.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:20 smithi196 ceph-mon[36300]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:48:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:20 smithi131 ceph-mon[36065]: pgmap v131: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:48:22.746 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:22 smithi131 ceph-mon[36065]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:22 smithi119 ceph-mon[31813]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:22 smithi196 ceph-mon[36300]: pgmap v132: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:24 smithi119 ceph-mon[31813]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:24 smithi196 ceph-mon[36300]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:24 smithi131 ceph-mon[36065]: pgmap v133: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:25 smithi119 ceph-mon[31813]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:25.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:25 smithi196 ceph-mon[36300]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:25.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:25 smithi131 ceph-mon[36065]: pgmap v134: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:26.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:48:26.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.933+0000 7fb7d5600640 1 -- 172.21.15.119:0/479181069 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d00743f0 msgr2=0x7fb7d010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.933+0000 7fb7d5600640 1 --2- 172.21.15.119:0/479181069 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d00743f0 0x7fb7d010d930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009900 tx=0x7fb7bc02bcc0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 -- 172.21.15.119:0/479181069 shutdown_connections 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 --2- 172.21.15.119:0/479181069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d010de70 0x7fb7d01102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 --2- 172.21.15.119:0/479181069 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d00743f0 0x7fb7d010d930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 --2- 172.21.15.119:0/479181069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d0073ad0 0x7fb7d0073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 -- 172.21.15.119:0/479181069 >> 172.21.15.119:0/479181069 conn(0x7fb7d0078aa0 msgr2=0x7fb7d0078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 -- 172.21.15.119:0/479181069 shutdown_connections 2024-03-28T18:48:26.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.934+0000 7fb7d5600640 1 -- 172.21.15.119:0/479181069 wait complete. 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 Processor -- start 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 -- start start 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d00743f0 0x7fb7d019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 0x7fb7d019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0113c20 con 0x7fb7d00743f0 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0113aa0 con 0x7fb7d0073ad0 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7d5600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0113da0 con 0x7fb7d010de70 2024-03-28T18:48:26.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33494/0 (socket says 172.21.15.119:33494) 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.935+0000 7fb7cea00640 1 -- 172.21.15.119:0/972876544 learned_addr learned my addr 172.21.15.119:0/972876544 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 0x7fb7d019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7ce000640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d00743f0 0x7fb7d019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 -- 172.21.15.119:0/972876544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 msgr2=0x7fb7d019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 -- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d00743f0 msgr2=0x7fb7d019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d00743f0 0x7fb7d019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 -- 172.21.15.119:0/972876544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d019f110 con 0x7fb7d010de70 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cea00640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7cf400640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 0x7fb7d019e9e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009fd0 tx=0x7fb7c800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7c7400640 1 -- 172.21.15.119:0/972876544 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c800ce20 con 0x7fb7d010de70 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.936+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d019f400 con 0x7fb7d010de70 2024-03-28T18:48:26.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.937+0000 7fb7c7400640 1 -- 172.21.15.119:0/972876544 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c8005020 con 0x7fb7d010de70 2024-03-28T18:48:26.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.937+0000 7fb7c7400640 1 -- 172.21.15.119:0/972876544 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8010720 con 0x7fb7d010de70 2024-03-28T18:48:26.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.937+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d01a6c90 con 0x7fb7d010de70 2024-03-28T18:48:26.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.940+0000 7fb7d83dd740 1 Processor -- start 2024-03-28T18:48:26.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7c7400640 1 -- 172.21.15.119:0/972876544 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fb7c8005540 con 0x7fb7d010de70 2024-03-28T18:48:26.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 -- start start 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a60355c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 0x55a60355d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 0x55a603568680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6034c1250 con 0x55a60355cf20 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a603469a30 con 0x55a60355dc00 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.941+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6034bd2f0 con 0x55a603566290 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7ce000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a60355c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7ce000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a60355c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33502/0 (socket says 172.21.15.119:33502) 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7ce000640 1 -- 172.21.15.119:0/4213859482 learned_addr learned my addr 172.21.15.119:0/4213859482 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 0x55a60355d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7ce000640 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 0x55a603568680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 -- 172.21.15.119:0/4213859482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 msgr2=0x55a603568680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 0x55a603568680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 -- 172.21.15.119:0/4213859482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 msgr2=0x55a60355c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a60355c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 -- 172.21.15.119:0/4213859482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a603454690 con 0x55a60355cf20 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7ce000640 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 0x55a603568680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7cea00640 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 0x55a60355d3a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00114f0 tx=0x7fb7c00119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c6cc0a1de2dc6f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.942+0000 7fb7c7400640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 0x7fb7b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7bb400640 1 -- 172.21.15.119:0/4213859482 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c001e070 con 0x55a60355cf20 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7c7400640 1 -- 172.21.15.119:0/972876544 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb7c80cf590 con 0x7fb7d010de70 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7ce000640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 0x7fb7b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7bb400640 1 -- 172.21.15.119:0/4213859482 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c00035f0 con 0x55a60355cf20 2024-03-28T18:48:26.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7bb400640 1 -- 172.21.15.119:0/4213859482 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000f820 con 0x55a60355cf20 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7d83dd740 1 -- 172.21.15.119:0/4213859482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 msgr2=0x55a60355d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 0x55a60355d3a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00114f0 tx=0x7fb7c00119b0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.943+0000 7fb7ce000640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 0x7fb7b40a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc02c8b0 tx=0x7fb7bc005b50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 -- 172.21.15.119:0/4213859482 shutdown_connections 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a603566290 0x55a603568680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355cf20 0x55a60355d3a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/4213859482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a60355c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 -- 172.21.15.119:0/4213859482 >> 172.21.15.119:0/4213859482 conn(0x55a603524200 msgr2=0x55a603557970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 -- 172.21.15.119:0/4213859482 shutdown_connections 2024-03-28T18:48:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 -- 172.21.15.119:0/4213859482 wait complete. 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 -- start start 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.944+0000 7fb7d83dd740 1 Processor -- start 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a603562be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 0x55a603563120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a603569100 con 0x55a603566290 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a60347ce80 con 0x55a60355dc00 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a603569280 con 0x55a60355cf20 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7d83dd740 10 client.0 _ll_register_callbacks cb 0x55a60348f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7ce000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a603562be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 0x55a603563120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36356/0 (socket says 172.21.15.119:36356) 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 -- 172.21.15.119:0/1136719860 learned_addr learned my addr 172.21.15.119:0/1136719860 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 -- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 msgr2=0x55a603562be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a603562be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 -- 172.21.15.119:0/1136719860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 msgr2=0x55a603563120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 0x55a603563120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cea00640 1 -- 172.21.15.119:0/1136719860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a60329f7d0 con 0x55a60355cf20 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.945+0000 7fb7cf400640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 0x55a603563120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.946+0000 7fb7cea00640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00114c0 tx=0x7fb7c0003a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=245458c029cc9010 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.946+0000 7fb7afe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.946+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c001e030 con 0x55a60355cf20 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.946+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb7c0005260 con 0x55a60355cf20 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.946+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c00035f0 con 0x55a60355cf20 2024-03-28T18:48:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c0031050 con 0x55a60355cf20 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 10 client.26623 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a6031d53f0 con 0x55a60355cf20 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c00035f0 con 0x55a60355cf20 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb7c000f5c0 con 0x55a60355cf20 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 10 client.26623 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 10 client.26623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 20 client.26623 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a603576040 0x55a603578430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 4 client.26623 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.947+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6034ce790 con 0x55a603576040 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.948+0000 7fb7ce000640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a603576040 0x55a603578430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.948+0000 7fb7ce000640 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a603576040 0x55a603578430 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fe37da3210e64b8 server_cookie=ff8123d0f0ac25a2 in_seq=0 out_seq=0 2024-03-28T18:48:26.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.948+0000 7fb7afe00640 10 client.26623 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7afe00640 1 -- 172.21.15.119:0/1136719860 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a6034ce790 con 0x55a603576040 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7afe00640 10 client.26623 handle_command_reply: tid=0 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 client.26623 shutdown 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 msgr2=0x55a603565850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00114c0 tx=0x7fb7c0003a50 comp rx=0 tx=0).stop 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 shutdown_connections 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a603576040 0x55a603578430 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a603566290 0x55a603563120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a60355dc00 0x55a603562be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7afe00640 0 client.26623 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/1136719860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603565850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 >> 172.21.15.119:0/1136719860 conn(0x55a603524200 msgr2=0x55a6035596c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 shutdown_connections 2024-03-28T18:48:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 1 -- 172.21.15.119:0/1136719860 wait complete. 2024-03-28T18:48:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.949+0000 7fb7d83dd740 20 client.26623 trim_cache size 0 max 16384 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 Processor -- start 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 -- start start 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603562500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a603556910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01af520 con 0x55a60355dc00 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01b0ea0 con 0x55a603566290 2024-03-28T18:48:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01afb80 con 0x55a60355cf20 2024-03-28T18:48:26.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7cf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7ce000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a603556910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603562500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7cf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33522/0 (socket says 172.21.15.119:33522) 2024-03-28T18:48:26.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.967+0000 7fb7cf400640 1 -- 172.21.15.119:0/2287083910 learned_addr learned my addr 172.21.15.119:0/2287083910 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 -- 172.21.15.119:0/2287083910 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 msgr2=0x55a603562500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603562500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 -- 172.21.15.119:0/2287083910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 msgr2=0x55a603556910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a603556910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 -- 172.21.15.119:0/2287083910 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6031950f0 con 0x55a603566290 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7cf400640 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8097f90 tx=0x7fb7c8097e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ae09bda94e85fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7baa00640 1 -- 172.21.15.119:0/2287083910 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80b3070 con 0x55a603566290 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7baa00640 1 -- 172.21.15.119:0/2287083910 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c80051d0 con 0x55a603566290 2024-03-28T18:48:26.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.968+0000 7fb7baa00640 1 -- 172.21.15.119:0/2287083910 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c809ce60 con 0x55a603566290 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- 172.21.15.119:0/2287083910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 msgr2=0x55a603567680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8097f90 tx=0x7fb7c8097e50 comp rx=0 tx=0).stop 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- 172.21.15.119:0/2287083910 shutdown_connections 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a603567680 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a603556910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/2287083910 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a603562500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- 172.21.15.119:0/2287083910 >> 172.21.15.119:0/2287083910 conn(0x55a603524200 msgr2=0x55a603556200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- 172.21.15.119:0/2287083910 shutdown_connections 2024-03-28T18:48:26.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- 172.21.15.119:0/2287083910 wait complete. 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.969+0000 7fb7d83dd740 1 -- start start 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 Processor -- start 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a60355fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a60355ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01b0840 con 0x55a60355dc00 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01b01e0 con 0x55a603566290 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01ad6e0 con 0x55a60355cf20 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7d83dd740 10 client.0 _ll_register_callbacks cb 0x7fb7c001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36374/0 (socket says 172.21.15.119:36374) 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7cea00640 1 -- 172.21.15.119:0/887746377 learned_addr learned my addr 172.21.15.119:0/887746377 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7ce000640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a60355fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.970+0000 7fb7cf400640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a60355ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 -- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 msgr2=0x55a60355ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a60355ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 -- 172.21.15.119:0/887746377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 msgr2=0x55a60355fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a60355fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 -- 172.21.15.119:0/887746377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6035d09b0 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7cea00640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0004550 tx=0x7fb7c000fb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c3f5aac35b7f0ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7afe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0039020 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c80051d0 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fb7c0003ac0 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c0044050 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7d83dd740 10 client.26629 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.971+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a6035d1ca0 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c80051d0 con 0x55a60355cf20 2024-03-28T18:48:26.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb7c00046a0 con 0x55a60355cf20 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 10 client.26629 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 20 client.26629 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a6035d22a0 0x55a6035d4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 4 client.26629 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a6035634b0 con 0x55a6035d22a0 2024-03-28T18:48:26.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.972+0000 7fb7ce000640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a6035d22a0 0x55a6035d4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7ce000640 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a6035d22a0 0x55a6035d4690 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02be1c041383994 server_cookie=27f07d5b9baabb37 in_seq=0 out_seq=0 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7afe00640 10 client.26629 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7afe00640 1 -- 172.21.15.119:0/887746377 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a6035634b0 con 0x55a6035d22a0 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7afe00640 10 client.26629 handle_command_reply: tid=0 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 client.26629 shutdown 2024-03-28T18:48:26.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 msgr2=0x55a60355f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0004550 tx=0x7fb7c000fb80 comp rx=0 tx=0).stop 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 shutdown_connections 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a6035d22a0 0x55a6035d4690 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a603566290 0x55a60355ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a60355dc00 0x55a60355fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 --2- 172.21.15.119:0/887746377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a60355cf20 0x55a60355f510 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7afe00640 0 client.26629 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 >> 172.21.15.119:0/887746377 conn(0x55a603524200 msgr2=0x55a603556a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 shutdown_connections 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 1 -- 172.21.15.119:0/887746377 wait complete. 2024-03-28T18:48:26.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.973+0000 7fb7d83dd740 20 client.26629 trim_cache size 0 max 16384 2024-03-28T18:48:26.979 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:48:26.979 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:48:26.979 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:48:26.979 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:48:26.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 msgr2=0x7fb7b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 0x7fb7b40a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc02c8b0 tx=0x7fb7bc005b50 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 msgr2=0x7fb7d019e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 0x7fb7d019e9e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009fd0 tx=0x7fb7c800ef10 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 shutdown_connections 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fb7b40a2970 0x7fb7b40a4e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7d010de70 0x7fb7d019e9e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7d00743f0 0x7fb7d019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 --2- 172.21.15.119:0/972876544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb7d0073ad0 0x7fb7d019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 >> 172.21.15.119:0/972876544 conn(0x7fb7d0078aa0 msgr2=0x7fb7d00fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 shutdown_connections 2024-03-28T18:48:26.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:26.974+0000 7fb7d5600640 1 -- 172.21.15.119:0/972876544 wait complete. 2024-03-28T18:48:26.995 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:48:26.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:48:27.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 -- 172.21.15.119:0/4130151532 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70107670 msgr2=0x7f6b70107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 --2- 172.21.15.119:0/4130151532 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70107670 0x7f6b70107a50 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009930 tx=0x7f6b6402bc30 comp rx=0 tx=0).stop 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 -- 172.21.15.119:0/4130151532 shutdown_connections 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 --2- 172.21.15.119:0/4130151532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70102020 0x7f6b7010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 --2- 172.21.15.119:0/4130151532 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70101660 0x7f6b70101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 --2- 172.21.15.119:0/4130151532 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70107670 0x7f6b70107a50 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 -- 172.21.15.119:0/4130151532 >> 172.21.15.119:0/4130151532 conn(0x7f6b700fd110 msgr2=0x7f6b700ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.203+0000 7f6b78400640 1 -- 172.21.15.119:0/4130151532 shutdown_connections 2024-03-28T18:48:27.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.204+0000 7f6b78400640 1 -- 172.21.15.119:0/4130151532 wait complete. 2024-03-28T18:48:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 Processor -- start 2024-03-28T18:48:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 -- start start 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 0x7f6b7019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 0x7f6b7019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70111a90 con 0x7f6b70101660 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70111910 con 0x7f6b70107670 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.205+0000 7f6b78400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b70111c10 con 0x7f6b70102020 2024-03-28T18:48:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 0x7f6b7019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b76400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 0x7f6b7019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b76400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33558/0 (socket says 172.21.15.119:33558) 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b76400640 1 -- 172.21.15.119:0/3831871304 learned_addr learned my addr 172.21.15.119:0/3831871304 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 -- 172.21.15.119:0/3831871304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 msgr2=0x7f6b7019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 -- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 msgr2=0x7f6b7019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 0x7f6b7019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 -- 172.21.15.119:0/3831871304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7019f160 con 0x7f6b70102020 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75a00640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 0x7f6b7019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b76400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.206+0000 7f6b75000640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 0x7f6b7019a6b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00e9f0 tx=0x7f6b5c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.207+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c00ccc0 con 0x7f6b70102020 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.207+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b5c010910 con 0x7f6b70102020 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.207+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c018960 con 0x7f6b70102020 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.207+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7019f450 con 0x7f6b70102020 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.207+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b701a6ce0 con 0x7f6b70102020 2024-03-28T18:48:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.208+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b700688e0 con 0x7f6b70102020 2024-03-28T18:48:27.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.211+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6b5c010ac0 con 0x7f6b70102020 2024-03-28T18:48:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.212+0000 7f6b6e000640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 0x7f6b580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.212+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b5c0cf2b0 con 0x7f6b70102020 2024-03-28T18:48:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.212+0000 7f6b75a00640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 0x7f6b580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.213+0000 7f6b75a00640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 0x7f6b580a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009900 tx=0x7f6b6402df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.219+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b5c09b900 con 0x7f6b70102020 2024-03-28T18:48:27.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.460+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b701a6f80 con 0x7f6b70102020 2024-03-28T18:48:27.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.462+0000 7f6b6e000640 1 -- 172.21.15.119:0/3831871304 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f6b5c09f9a0 con 0x7f6b70102020 2024-03-28T18:48:27.462 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:48:27.462 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:48:27.463 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 msgr2=0x7f6b580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 0x7f6b580a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009900 tx=0x7f6b6402df90 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 msgr2=0x7f6b7019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 0x7f6b7019a6b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00e9f0 tx=0x7f6b5c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 shutdown_connections 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b580a2970 0x7f6b580a4e30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b70107670 0x7f6b7019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b70102020 0x7f6b7019a6b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 --2- 172.21.15.119:0/3831871304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b70101660 0x7f6b7019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 >> 172.21.15.119:0/3831871304 conn(0x7f6b700fd110 msgr2=0x7f6b7010cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 shutdown_connections 2024-03-28T18:48:27.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.464+0000 7f6b78400640 1 -- 172.21.15.119:0/3831871304 wait complete. 2024-03-28T18:48:27.486 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:48:27.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:48:27.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- 172.21.15.119:0/1065094949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 msgr2=0x7ffb7c1105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1065094949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c1105f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009ca0 tx=0x7ffb7402c100 comp rx=0 tx=0).stop 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- 172.21.15.119:0/1065094949 shutdown_connections 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1065094949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c1105f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1065094949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 0x7ffb7c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1065094949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- 172.21.15.119:0/1065094949 >> 172.21.15.119:0/1065094949 conn(0x7ffb7c0ff1d0 msgr2=0x7ffb7c1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- 172.21.15.119:0/1065094949 shutdown_connections 2024-03-28T18:48:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- 172.21.15.119:0/1065094949 wait complete. 2024-03-28T18:48:27.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 Processor -- start 2024-03-28T18:48:27.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.696+0000 7ffb82a00640 1 -- start start 2024-03-28T18:48:27.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 0x7ffb7c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c113b40 con 0x7ffb7c103720 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1139c0 con 0x7ffb7c1040e0 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c113cc0 con 0x7ffb7c109730 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33582/0 (socket says 172.21.15.119:33582) 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 -- 172.21.15.119:0/1232259512 learned_addr learned my addr 172.21.15.119:0/1232259512 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb81600640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 0x7ffb7c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82000640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 -- 172.21.15.119:0/1232259512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 msgr2=0x7ffb7c19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 -- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 msgr2=0x7ffb7c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 0x7ffb7c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb80c00640 1 -- 172.21.15.119:0/1232259512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c19efa0 con 0x7ffb7c1040e0 2024-03-28T18:48:27.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.697+0000 7ffb82000640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb80c00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e9f0 tx=0x7ffb6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb7a000640 1 -- 172.21.15.119:0/1232259512 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6800cd80 con 0x7ffb7c1040e0 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb7a000640 1 -- 172.21.15.119:0/1232259512 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb68010910 con 0x7ffb7c1040e0 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb7a000640 1 -- 172.21.15.119:0/1232259512 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68018990 con 0x7ffb7c1040e0 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c19f290 con 0x7ffb7c1040e0 2024-03-28T18:48:27.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.698+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c1a6b20 con 0x7ffb7c1040e0 2024-03-28T18:48:27.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.701+0000 7ffb7a000640 1 -- 172.21.15.119:0/1232259512 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ffb68010ac0 con 0x7ffb7c1040e0 2024-03-28T18:48:27.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.701+0000 7ffb86d69740 1 Processor -- start 2024-03-28T18:48:27.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.701+0000 7ffb86d69740 1 -- start start 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bc4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bd0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2b282c0 con 0x555ac2bcdde0 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2ad0c90 con 0x555ac2bc48b0 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb86d69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2b24360 con 0x555ac2bc5260 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bc4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36418/0 (socket says 172.21.15.119:36418) 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb81600640 1 -- 172.21.15.119:0/3600264495 learned_addr learned my addr 172.21.15.119:0/3600264495 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bd0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 -- 172.21.15.119:0/3600264495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 msgr2=0x555ac2bcd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 -- 172.21.15.119:0/3600264495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 msgr2=0x555ac2bd0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bd0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb82000640 1 -- 172.21.15.119:0/3600264495 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac2af5080 con 0x555ac2bc48b0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bd0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.702+0000 7ffb81600640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb7a000640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 0x7ffb640a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb82000640 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bc4c90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009bf0 tx=0x7ffb740323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6a12b6223cd6f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb7a000640 1 -- 172.21.15.119:0/1232259512 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb680cf210 con 0x7ffb7c1040e0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb81600640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 0x7ffb640a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb6e600640 1 -- 172.21.15.119:0/3600264495 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74044020 con 0x555ac2bc48b0 2024-03-28T18:48:27.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb6e600640 1 -- 172.21.15.119:0/3600264495 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb74032c90 con 0x555ac2bc48b0 2024-03-28T18:48:27.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.703+0000 7ffb6e600640 1 -- 172.21.15.119:0/3600264495 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74039040 con 0x555ac2bc48b0 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- 172.21.15.119:0/3600264495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 msgr2=0x555ac2bc4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bc4c90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009bf0 tx=0x7ffb740323d0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb81600640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 0x7ffb640a4eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x555ac2bbfff0 tx=0x7ffb70005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- 172.21.15.119:0/3600264495 shutdown_connections 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bd0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3600264495 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bc4c90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- 172.21.15.119:0/3600264495 >> 172.21.15.119:0/3600264495 conn(0x555ac2bbef80 msgr2=0x555ac2bbf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- 172.21.15.119:0/3600264495 shutdown_connections 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- 172.21.15.119:0/3600264495 wait complete. 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.704+0000 7ffb86d69740 1 -- start start 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 Processor -- start 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2bcaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2bcaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2bd0cf0 con 0x555ac2bc5260 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2ae40d0 con 0x555ac2bcdde0 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac2bd0e70 con 0x555ac2bc48b0 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb86d69740 10 client.0 _ll_register_callbacks cb 0x555ac2ae8e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:27.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36424/0 (socket says 172.21.15.119:36424) 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb81600640 1 -- 172.21.15.119:0/3526443970 learned_addr learned my addr 172.21.15.119:0/3526443970 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb82000640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2bcaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.705+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2bcaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 -- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 msgr2=0x555ac2bcd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 -- 172.21.15.119:0/3526443970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 msgr2=0x555ac2bcaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2bcaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 -- 172.21.15.119:0/3526443970 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac283c3f0 con 0x555ac2bcdde0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb82000640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2bcaff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb74037040 tx=0x7ffb740095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5cca71646ab97ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb62a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7404f070 con 0x555ac2bcdde0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb74032c90 con 0x555ac2bcdde0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb74032790 con 0x555ac2bcdde0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7405b050 con 0x555ac2bcdde0 2024-03-28T18:48:27.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb86d69740 10 client.26666 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.706+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555ac29067d0 con 0x555ac2bcdde0 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb74032c90 con 0x555ac2bcdde0 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb740494e0 con 0x555ac2bcdde0 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 10 client.26666 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 10 client.26666 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 20 client.26666 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2bdda70 0x555ac2bdfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 4 client.26666 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ac2b35800 con 0x555ac2bdda70 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.707+0000 7ffb81600640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2bdda70 0x555ac2bdfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.708+0000 7ffb81600640 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2bdda70 0x555ac2bdfe60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4c1b295e6606a57b server_cookie=8bffb0544f26912 in_seq=0 out_seq=0 2024-03-28T18:48:27.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.708+0000 7ffb62a00640 10 client.26666 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:48:27.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb62a00640 1 -- 172.21.15.119:0/3526443970 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555ac2b35800 con 0x555ac2bdda70 2024-03-28T18:48:27.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb62a00640 10 client.26666 handle_command_reply: tid=0 2024-03-28T18:48:27.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 client.26666 shutdown 2024-03-28T18:48:27.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 msgr2=0x555ac2bcaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2bcaff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb74037040 tx=0x7ffb740095c0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 shutdown_connections 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2bdda70 0x555ac2bdfe60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2bcaff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2bcaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3526443970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 >> 172.21.15.119:0/3526443970 conn(0x555ac2bbef80 msgr2=0x555ac2bc0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb62a00640 0 client.26666 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 shutdown_connections 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 1 -- 172.21.15.119:0/3526443970 wait complete. 2024-03-28T18:48:27.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.709+0000 7ffb86d69740 20 client.26666 trim_cache size 0 max 16384 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 Processor -- start 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 -- start start 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bc7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1af370 con 0x555ac2bcdde0 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1b0cf0 con 0x555ac2bc48b0 2024-03-28T18:48:27.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb86d69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1af9d0 con 0x555ac2bc5260 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb80c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb81600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb81600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33618/0 (socket says 172.21.15.119:33618) 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb82000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bc7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb81600640 1 -- 172.21.15.119:0/1906463243 learned_addr learned my addr 172.21.15.119:0/1906463243 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb81600640 1 -- 172.21.15.119:0/1906463243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 msgr2=0x555ac2bcd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.723+0000 7ffb81600640 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb81600640 1 -- 172.21.15.119:0/1906463243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 msgr2=0x555ac2bc7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb81600640 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bc7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb81600640 1 -- 172.21.15.119:0/1906463243 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac2bbece0 con 0x555ac2bc48b0 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb81600640 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005bc0 tx=0x7ffb7001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9964609034704428 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb6dc00640 1 -- 172.21.15.119:0/1906463243 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70004430 con 0x555ac2bc48b0 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb6dc00640 1 -- 172.21.15.119:0/1906463243 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb70030070 con 0x555ac2bc48b0 2024-03-28T18:48:27.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.724+0000 7ffb6dc00640 1 -- 172.21.15.119:0/1906463243 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7000f790 con 0x555ac2bc48b0 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- 172.21.15.119:0/1906463243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 msgr2=0x555ac2bcd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005bc0 tx=0x7ffb7001b000 comp rx=0 tx=0).stop 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- 172.21.15.119:0/1906463243 shutdown_connections 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bcdde0 0x555ac2bc7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc5260 0x555ac2bcd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 --2- 172.21.15.119:0/1906463243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bc48b0 0x555ac2bcd1a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- 172.21.15.119:0/1906463243 >> 172.21.15.119:0/1906463243 conn(0x555ac2bbef80 msgr2=0x555ac2bbe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- 172.21.15.119:0/1906463243 shutdown_connections 2024-03-28T18:48:27.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- 172.21.15.119:0/1906463243 wait complete. 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 -- start start 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.725+0000 7ffb86d69740 1 Processor -- start 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2be2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2be0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1b0690 con 0x555ac2bc5260 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1b0030 con 0x555ac2bcdde0 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7c1ad530 con 0x555ac2bc48b0 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb86d69740 10 client.0 _ll_register_callbacks cb 0x7ffb700097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb81600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36438/0 (socket says 172.21.15.119:36438) 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb80c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2be2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb81600640 1 -- 172.21.15.119:0/3534592932 learned_addr learned my addr 172.21.15.119:0/3534592932 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb82000640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2be0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb80c00640 1 -- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 msgr2=0x555ac2be26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.726+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb80c00640 1 -- 172.21.15.119:0/3534592932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 msgr2=0x555ac2be0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2be0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb80c00640 1 -- 172.21.15.119:0/3534592932 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac2be1370 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb81600640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb80c00640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2be2c00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb680cc900 tx=0x7ffb6809b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca0ed48fff2474d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb62a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6809bf10 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb680a3b20 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb70030070 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb6809b810 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb86d69740 10 client.16449 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555ac2c39b20 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.727+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb70030070 con 0x555ac2bc5260 2024-03-28T18:48:27.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb6809cc60 con 0x555ac2bc5260 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 10 client.16449 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 10 client.16449 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 20 client.16449 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2c3a220 0x555ac2c3c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 4 client.16449 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555ac2b2fb50 con 0x555ac2c3a220 2024-03-28T18:48:27.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.728+0000 7ffb81600640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2c3a220 0x555ac2c3c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb81600640 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2c3a220 0x555ac2c3c640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=aee33c0c556c81b2 server_cookie=eeee84301e9a8d18 in_seq=0 out_seq=0 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb62a00640 10 client.16449 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb62a00640 1 -- 172.21.15.119:0/3534592932 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555ac2b2fb50 con 0x555ac2c3a220 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb62a00640 10 client.16449 handle_command_reply: tid=0 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 client.16449 shutdown 2024-03-28T18:48:27.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 msgr2=0x555ac2be2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2be2c00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb680cc900 tx=0x7ffb6809b5e0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 shutdown_connections 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x555ac2c3a220 0x555ac2c3c640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555ac2bcdde0 0x555ac2be0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac2bc5260 0x555ac2be2c00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 --2- 172.21.15.119:0/3534592932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ac2bc48b0 0x555ac2be26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb62a00640 0 client.16449 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 >> 172.21.15.119:0/3534592932 conn(0x555ac2bbef80 msgr2=0x555ac2bc02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 shutdown_connections 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 1 -- 172.21.15.119:0/3534592932 wait complete. 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.729+0000 7ffb86d69740 20 client.16449 trim_cache size 0 max 16384 2024-03-28T18:48:27.734 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 msgr2=0x7ffb640a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 0x7ffb640a4eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x555ac2bbfff0 tx=0x7ffb70005f90 comp rx=0 tx=0).stop 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 msgr2=0x7ffb7c19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e9f0 tx=0x7ffb6800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 shutdown_connections 2024-03-28T18:48:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb640a29f0 0x7ffb640a4eb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb7c109730 0x7ffb7c19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb7c1040e0 0x7ffb7c19a540 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 --2- 172.21.15.119:0/1232259512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb7c103720 0x7ffb7c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 >> 172.21.15.119:0/1232259512 conn(0x7ffb7c0ff1d0 msgr2=0x7ffb7c0ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 shutdown_connections 2024-03-28T18:48:27.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.730+0000 7ffb82a00640 1 -- 172.21.15.119:0/1232259512 wait complete. 2024-03-28T18:48:27.748 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:48:27.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 -- 172.21.15.119:0/1069131433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 msgr2=0x7fc834107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 --2- 172.21.15.119:0/1069131433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc834107a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc824009930 tx=0x7fc82402bc30 comp rx=0 tx=0).stop 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 -- 172.21.15.119:0/1069131433 shutdown_connections 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 --2- 172.21.15.119:0/1069131433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 0x7fc83410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 --2- 172.21.15.119:0/1069131433 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc834101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 --2- 172.21.15.119:0/1069131433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc834107a20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.950+0000 7fc839400640 1 -- 172.21.15.119:0/1069131433 >> 172.21.15.119:0/1069131433 conn(0x7fc834078aa0 msgr2=0x7fc834078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.951+0000 7fc839400640 1 -- 172.21.15.119:0/1069131433 shutdown_connections 2024-03-28T18:48:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.951+0000 7fc839400640 1 -- 172.21.15.119:0/1069131433 wait complete. 2024-03-28T18:48:27.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 Processor -- start 2024-03-28T18:48:27.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 -- start start 2024-03-28T18:48:27.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 0x7fc83419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834111a60 con 0x7fc834101ff0 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341118e0 con 0x7fc834101630 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc839400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834111be0 con 0x7fc834107640 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc833200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc832800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc831e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 0x7fc83419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc832800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33656/0 (socket says 172.21.15.119:33656) 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc833200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36448/0 (socket says 172.21.15.119:36448) 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.952+0000 7fc832800640 1 -- 172.21.15.119:0/788492968 learned_addr learned my addr 172.21.15.119:0/788492968 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 -- 172.21.15.119:0/788492968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 msgr2=0x7fc83419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 -- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 msgr2=0x7fc83419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 0x7fc83419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 -- 172.21.15.119:0/788492968 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83419f130 con 0x7fc834101630 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc833200640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc832800640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc824009900 tx=0x7fc824004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc82b400640 1 -- 172.21.15.119:0/788492968 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824043020 con 0x7fc834101630 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc82b400640 1 -- 172.21.15.119:0/788492968 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8240054e0 con 0x7fc834101630 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc82b400640 1 -- 172.21.15.119:0/788492968 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824003b80 con 0x7fc834101630 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.953+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83419f3c0 con 0x7fc834101630 2024-03-28T18:48:27.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.954+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8341a6cb0 con 0x7fc834101630 2024-03-28T18:48:27.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.956+0000 7fc82b400640 1 -- 172.21.15.119:0/788492968 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc824046020 con 0x7fc834101630 2024-03-28T18:48:27.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 Processor -- start 2024-03-28T18:48:27.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 -- start start 2024-03-28T18:48:27.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49a6d0 0x5635ca49aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca3fe2c0 con 0x5635ca49a6d0 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca3a6c90 con 0x5635ca49b080 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc83c006740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca3fa360 con 0x5635ca4a3c00 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc831e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc832800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36458/0 (socket says 172.21.15.119:36458) 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.957+0000 7fc833200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49a6d0 0x5635ca49aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 -- 172.21.15.119:0/1418991763 learned_addr learned my addr 172.21.15.119:0/1418991763 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 -- 172.21.15.119:0/1418991763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 msgr2=0x5635ca4a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 -- 172.21.15.119:0/1418991763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49a6d0 msgr2=0x5635ca49aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49a6d0 0x5635ca49aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 -- 172.21.15.119:0/1418991763 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635ca3cb080 con 0x5635ca4a3c00 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc82b400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 0x7fc8180a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc831e00640 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc820009c90 tx=0x7fc820011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f3ad92044201f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc82b400640 1 -- 172.21.15.119:0/788492968 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc8240f0e10 con 0x7fc834101630 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc81f400640 1 -- 172.21.15.119:0/1418991763 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82001e070 con 0x5635ca4a3c00 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc833200640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 0x7fc8180a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc81f400640 1 -- 172.21.15.119:0/1418991763 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc820019b10 con 0x5635ca4a3c00 2024-03-28T18:48:27.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.958+0000 7fc81f400640 1 -- 172.21.15.119:0/1418991763 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82000eac0 con 0x5635ca4a3c00 2024-03-28T18:48:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 -- 172.21.15.119:0/1418991763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 msgr2=0x5635ca4a6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc833200640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 0x7fc8180a4eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005fd0 tx=0x7fc82c009600 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc820009c90 tx=0x7fc820011810 comp rx=0 tx=0).stop 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 -- 172.21.15.119:0/1418991763 shutdown_connections 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4a6050 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 --2- 172.21.15.119:0/1418991763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49a6d0 0x5635ca49aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 -- 172.21.15.119:0/1418991763 >> 172.21.15.119:0/1418991763 conn(0x5635ca494da0 msgr2=0x5635ca495180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 -- 172.21.15.119:0/1418991763 shutdown_connections 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.959+0000 7fc83c006740 1 -- 172.21.15.119:0/1418991763 wait complete. 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.960+0000 7fc83c006740 1 -- start start 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.960+0000 7fc83c006740 1 Processor -- start 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca4a3c00 0x5635ca4a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca4a6b70 con 0x5635ca4a3c00 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca3ba0d0 con 0x5635ca49b080 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635ca4a6cf0 con 0x5635ca49a6d0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc83c006740 10 client.0 _ll_register_callbacks cb 0x5635ca3bee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36464/0 (socket says 172.21.15.119:36464) 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 -- 172.21.15.119:0/1734657453 learned_addr learned my addr 172.21.15.119:0/1734657453 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc831e00640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc833200640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca4a3c00 0x5635ca4a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 -- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 msgr2=0x5635ca4a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 -- 172.21.15.119:0/1734657453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca4a3c00 msgr2=0x5635ca4a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca4a3c00 0x5635ca4a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 -- 172.21.15.119:0/1734657453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635ca1123f0 con 0x5635ca49a6d0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.961+0000 7fc832800640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc824000c00 tx=0x7fc8240b90a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d3d503f33ce9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8240bd000 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc8240ee720 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc820019b10 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8240bce20 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 10 client.26665 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5635ca1dc7d0 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc820019b10 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc8240c57d0 con 0x5635ca49a6d0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 10 client.26665 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 10 client.26665 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 20 client.26665 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca4b3ac0 0x5635ca4b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 4 client.26665 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.962+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635ca40b800 con 0x5635ca4b3ac0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.963+0000 7fc831e00640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca4b3ac0 0x5635ca4b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.963+0000 7fc831e00640 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca4b3ac0 0x5635ca4b5f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=860e45b38dcd5570 server_cookie=4796721eade3e858 in_seq=0 out_seq=0 2024-03-28T18:48:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.963+0000 7fc813e00640 10 client.26665 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.964+0000 7fc813e00640 1 -- 172.21.15.119:0/1734657453 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5635ca40b800 con 0x5635ca4b3ac0 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.964+0000 7fc813e00640 10 client.26665 handle_command_reply: tid=0 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.964+0000 7fc83c006740 1 client.26665 shutdown 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 msgr2=0x5635ca4a3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc824000c00 tx=0x7fc8240b90a0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 shutdown_connections 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca4b3ac0 0x5635ca4b5f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca4a3c00 0x5635ca4a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49b080 0x5635ca4a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 --2- 172.21.15.119:0/1734657453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a3520 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc813e00640 0 client.26665 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 >> 172.21.15.119:0/1734657453 conn(0x5635ca494da0 msgr2=0x5635ca496af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 shutdown_connections 2024-03-28T18:48:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 1 -- 172.21.15.119:0/1734657453 wait complete. 2024-03-28T18:48:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.965+0000 7fc83c006740 20 client.26665 trim_cache size 0 max 16384 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.978+0000 7fc83c006740 1 Processor -- start 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.978+0000 7fc83c006740 1 -- start start 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.978+0000 7fc83c006740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.978+0000 7fc83c006740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc83c006740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca4a3c00 0x5635ca49db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc83c006740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341af510 con 0x5635ca49b080 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc83c006740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341b0e90 con 0x5635ca4a3c00 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc83c006740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341afb70 con 0x5635ca49a6d0 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36480/0 (socket says 172.21.15.119:36480) 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 -- 172.21.15.119:0/870299412 learned_addr learned my addr 172.21.15.119:0/870299412 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc833200640 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca4a3c00 0x5635ca49db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc831e00640 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 -- 172.21.15.119:0/870299412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca4a3c00 msgr2=0x5635ca49db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca4a3c00 0x5635ca49db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 -- 172.21.15.119:0/870299412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 msgr2=0x5635ca4a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 -- 172.21.15.119:0/870299412 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635ca495f40 con 0x5635ca49a6d0 2024-03-28T18:48:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.979+0000 7fc832800640 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc8240bd450 tx=0x7fc824003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ee101d5311d554 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc81ea00640 1 -- 172.21.15.119:0/870299412 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8240bd000 con 0x5635ca49a6d0 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc81ea00640 1 -- 172.21.15.119:0/870299412 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc824004c80 con 0x5635ca49a6d0 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc81ea00640 1 -- 172.21.15.119:0/870299412 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82402cb90 con 0x5635ca49a6d0 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 -- 172.21.15.119:0/870299412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 msgr2=0x5635ca4a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc8240bd450 tx=0x7fc824003680 comp rx=0 tx=0).stop 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 -- 172.21.15.119:0/870299412 shutdown_connections 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca4a3c00 0x5635ca49db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 --2- 172.21.15.119:0/870299412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca49a6d0 0x5635ca4a2fe0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 -- 172.21.15.119:0/870299412 >> 172.21.15.119:0/870299412 conn(0x5635ca494da0 msgr2=0x5635ca496a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 -- 172.21.15.119:0/870299412 shutdown_connections 2024-03-28T18:48:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.980+0000 7fc83c006740 1 -- 172.21.15.119:0/870299412 wait complete. 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 -- start start 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 Processor -- start 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 0x5635ca4abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4aa460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341b0830 con 0x5635ca49b080 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341b01d0 con 0x5635ca49a6d0 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341ad6d0 con 0x5635ca4a3c00 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.981+0000 7fc83c006740 10 client.0 _ll_register_callbacks cb 0x7fc82402ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc832800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 0x5635ca4abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc833200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc832800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 0x5635ca4abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33696/0 (socket says 172.21.15.119:33696) 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc832800640 1 -- 172.21.15.119:0/1739376002 learned_addr learned my addr 172.21.15.119:0/1739376002 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 -- 172.21.15.119:0/1739376002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 msgr2=0x5635ca4aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 -- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 msgr2=0x5635ca4abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 0x5635ca4abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 -- 172.21.15.119:0/1739376002 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635ca4aaad0 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc831e00640 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a9f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x5635ca4a0110 tx=0x7fc820005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9200f2ceb2142e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.982+0000 7fc813e00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82001e040 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc8200052c0 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc824004c80 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8200054e0 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc83c006740 10 client.16467 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5635ca510270 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc824004c80 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.983+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc82000e730 con 0x5635ca49b080 2024-03-28T18:48:27.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 10 client.16467 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 10 client.16467 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 20 client.16467 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca510790 0x5635ca512bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 4 client.16467 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5635ca405b50 con 0x5635ca510790 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc832800640 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca510790 0x5635ca512bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc832800640 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca510790 0x5635ca512bb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f00693ad3f7bff59 server_cookie=fe87a2e07b91cd9d in_seq=0 out_seq=0 2024-03-28T18:48:27.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc813e00640 10 client.16467 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc813e00640 1 -- 172.21.15.119:0/1739376002 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5635ca405b50 con 0x5635ca510790 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc813e00640 10 client.16467 handle_command_reply: tid=0 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.984+0000 7fc83c006740 1 client.16467 shutdown 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 msgr2=0x5635ca4a9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a9f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x5635ca4a0110 tx=0x7fc820005850 comp rx=0 tx=0).stop 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 shutdown_connections 2024-03-28T18:48:27.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5635ca4a3c00 0x5635ca4aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5635ca510790 0x5635ca512bb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5635ca49b080 0x5635ca4a9f20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 --2- 172.21.15.119:0/1739376002 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5635ca49a6d0 0x5635ca4abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc813e00640 0 client.16467 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 >> 172.21.15.119:0/1739376002 conn(0x5635ca494da0 msgr2=0x5635ca493620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 shutdown_connections 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 1 -- 172.21.15.119:0/1739376002 wait complete. 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc83c006740 20 client.16467 trim_cache size 0 max 16384 2024-03-28T18:48:27.990 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 msgr2=0x7fc8180a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 0x7fc8180a4eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc82c005fd0 tx=0x7fc82c009600 comp rx=0 tx=0).stop 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 msgr2=0x7fc83419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.985+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc824009900 tx=0x7fc824004360 comp rx=0 tx=0).stop 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 shutdown_connections 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc8180a29f0 0x7fc8180a4eb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc834107640 0x7fc83419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc834101ff0 0x7fc83419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 --2- 172.21.15.119:0/788492968 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc834101630 0x7fc83419a190 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:27.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 >> 172.21.15.119:0/788492968 conn(0x7fc834078aa0 msgr2=0x7fc834103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:27.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 shutdown_connections 2024-03-28T18:48:27.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:27.986+0000 7fc839400640 1 -- 172.21.15.119:0/788492968 wait complete. 2024-03-28T18:48:28.003 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:48:28.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:48:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 -- 172.21.15.119:0/777581651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5014108800 msgr2=0x7f501410c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 --2- 172.21.15.119:0/777581651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5014108800 0x7f501410c7c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5000009900 tx=0x7f500002bc40 comp rx=0 tx=0).stop 2024-03-28T18:48:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 -- 172.21.15.119:0/777581651 shutdown_connections 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 --2- 172.21.15.119:0/777581651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f501410cea0 0x7f5014114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 --2- 172.21.15.119:0/777581651 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5014108800 0x7f501410c7c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 --2- 172.21.15.119:0/777581651 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 0x7f5014108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 -- 172.21.15.119:0/777581651 >> 172.21.15.119:0/777581651 conn(0x7f50140774c0 msgr2=0x7f50141064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 -- 172.21.15.119:0/777581651 shutdown_connections 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.205+0000 7f5013e00640 1 -- 172.21.15.119:0/777581651 wait complete. 2024-03-28T18:48:28.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.206+0000 7f5013e00640 1 Processor -- start 2024-03-28T18:48:28.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 -- start start 2024-03-28T18:48:28.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 0x7f501419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 0x7f50141a3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014117fa0 con 0x7f5014107e50 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014117e20 con 0x7f501410cea0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014118120 con 0x7f50141a2cb0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5012000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5012000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33714/0 (socket says 172.21.15.119:33714) 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5012000640 1 -- 172.21.15.119:0/1294333596 learned_addr learned my addr 172.21.15.119:0/1294333596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5013400640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 0x7f50141a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.207+0000 7f5012a00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 0x7f501419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 -- 172.21.15.119:0/1294333596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 msgr2=0x7f50141a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 0x7f50141a3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 -- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 msgr2=0x7f501419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 0x7f501419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 -- 172.21.15.119:0/1294333596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50141a37a0 con 0x7f501410cea0 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5013400640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 0x7f50141a3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5012000640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f500002c250 tx=0x7f5000004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f500b400640 1 -- 172.21.15.119:0/1294333596 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000041070 con 0x7f501410cea0 2024-03-28T18:48:28.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f500b400640 1 -- 172.21.15.119:0/1294333596 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f500002cca0 con 0x7f501410cea0 2024-03-28T18:48:28.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50141a3a30 con 0x7f501410cea0 2024-03-28T18:48:28.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.208+0000 7f500b400640 1 -- 172.21.15.119:0/1294333596 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000003da0 con 0x7f501410cea0 2024-03-28T18:48:28.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.209+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50141ab1a0 con 0x7f501410cea0 2024-03-28T18:48:28.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.212+0000 7f501c585740 1 Processor -- start 2024-03-28T18:48:28.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.212+0000 7f501c585740 1 -- start start 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.212+0000 7f501c585740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292455c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.212+0000 7f501c585740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.212+0000 7f500b400640 1 -- 172.21.15.119:0/1294333596 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5000044070 con 0x7f501410cea0 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f501c585740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292461230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f501c585740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622923b92c0 con 0x56229245ede0 2024-03-28T18:48:28.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f501c585740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562292361c90 con 0x5622924558b0 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f501c585740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622923b5360 con 0x562292456260 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5013400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292455c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36514/0 (socket says 172.21.15.119:36514) 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 -- 172.21.15.119:0/3307737718 learned_addr learned my addr 172.21.15.119:0/3307737718 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012000640 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292461230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 -- 172.21.15.119:0/3307737718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 msgr2=0x562292455c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292455c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 -- 172.21.15.119:0/3307737718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 msgr2=0x562292461230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292461230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 -- 172.21.15.119:0/3307737718 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562292386080 con 0x562292456260 2024-03-28T18:48:28.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.213+0000 7f5012a00640 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f500400bcd0 tx=0x7f50040107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1b276b4f06cdad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f4fffe00640 1 -- 172.21.15.119:0/3307737718 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004005220 con 0x562292456260 2024-03-28T18:48:28.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f4fffe00640 1 -- 172.21.15.119:0/3307737718 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5004004c80 con 0x562292456260 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f4fffe00640 1 -- 172.21.15.119:0/3307737718 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004011780 con 0x562292456260 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f500b400640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 0x7f4ff80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f500b400640 1 -- 172.21.15.119:0/1294333596 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f50000f0a90 con 0x7f501410cea0 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f5013400640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 0x7f4ff80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f501c585740 1 -- 172.21.15.119:0/3307737718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 msgr2=0x56229245e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.214+0000 7f501c585740 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f500400bcd0 tx=0x7f50040107a0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 -- 172.21.15.119:0/3307737718 shutdown_connections 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292461230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e810 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 --2- 172.21.15.119:0/3307737718 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292455c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 -- 172.21.15.119:0/3307737718 >> 172.21.15.119:0/3307737718 conn(0x56229244ff80 msgr2=0x562292450360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 -- 172.21.15.119:0/3307737718 shutdown_connections 2024-03-28T18:48:28.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 -- 172.21.15.119:0/3307737718 wait complete. 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f5013400640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 0x7f4ff80a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f500c005fd0 tx=0x7f500c009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.215+0000 7f501c585740 1 -- start start 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 Processor -- start 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5622924558b0 0x56229245e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56229245ede0 0x56229245bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562292461d50 con 0x5622924558b0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622923750d0 con 0x56229245ede0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562292461ed0 con 0x562292456260 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f501c585740 10 client.0 _ll_register_callbacks cb 0x562292379e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f5012000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f5012a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5622924558b0 0x56229245e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f5012000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36530/0 (socket says 172.21.15.119:36530) 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f5012000640 1 -- 172.21.15.119:0/1363315641 learned_addr learned my addr 172.21.15.119:0/1363315641 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.216+0000 7f5013400640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56229245ede0 0x56229245bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 -- 172.21.15.119:0/1363315641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56229245ede0 msgr2=0x56229245bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56229245ede0 0x56229245bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 -- 172.21.15.119:0/1363315641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5622924558b0 msgr2=0x56229245e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5622924558b0 0x56229245e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 -- 172.21.15.119:0/1363315641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622920cd3f0 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f5012000640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f50000bd1e0 tx=0x7f5000004ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3dd019a15b70a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f4ffcc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500004b070 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f50000b98c0 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5004004c80 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50000bd020 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f501c585740 10 client.26683 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.217+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5622921977d0 con 0x562292456260 2024-03-28T18:48:28.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5004004c80 con 0x562292456260 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f50000bd4b0 con 0x562292456260 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 10 client.26683 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 10 client.26683 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 20 client.26683 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x56229246ebc0 0x562292470ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 4 client.26683 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622923c6800 con 0x56229246ebc0 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.218+0000 7f5012a00640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x56229246ebc0 0x562292470ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.219+0000 7f4ffcc00640 10 client.26683 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:48:28.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.219+0000 7f5012a00640 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x56229246ebc0 0x562292470ff0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17bdf99513a1be13 server_cookie=6ae85018c825667c in_seq=0 out_seq=0 2024-03-28T18:48:28.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.219+0000 7f4ffcc00640 1 -- 172.21.15.119:0/1363315641 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5622923c6800 con 0x56229246ebc0 2024-03-28T18:48:28.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.219+0000 7f4ffcc00640 10 client.26683 handle_command_reply: tid=0 2024-03-28T18:48:28.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 client.26683 shutdown 2024-03-28T18:48:28.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 msgr2=0x56229245ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f50000bd1e0 tx=0x7f5000004ee0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 shutdown_connections 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x56229246ebc0 0x562292470ff0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56229245ede0 0x56229245bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245ba50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 --2- 172.21.15.119:0/1363315641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5622924558b0 0x56229245e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 >> 172.21.15.119:0/1363315641 conn(0x56229244ff80 msgr2=0x562292451cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f4ffcc00640 0 client.26683 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 shutdown_connections 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 1 -- 172.21.15.119:0/1363315641 wait complete. 2024-03-28T18:48:28.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.221+0000 7f501c585740 20 client.26683 trim_cache size 0 max 16384 2024-03-28T18:48:28.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.233+0000 7f501c585740 1 Processor -- start 2024-03-28T18:48:28.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 -- start start 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292456da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b3710 con 0x56229245ede0 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b5090 con 0x5622924558b0 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f501c585740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b3d70 con 0x562292456260 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33736/0 (socket says 172.21.15.119:33736) 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 -- 172.21.15.119:0/100931739 learned_addr learned my addr 172.21.15.119:0/100931739 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012000640 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5013400640 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292456da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 -- 172.21.15.119:0/100931739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 msgr2=0x56229245e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 -- 172.21.15.119:0/100931739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 msgr2=0x562292456da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292456da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.234+0000 7f5012a00640 1 -- 172.21.15.119:0/100931739 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56229244fab0 con 0x5622924558b0 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f5012a00640 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5004011010 tx=0x7f500400bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ba81fdc66e96609 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f4fff400640 1 -- 172.21.15.119:0/100931739 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004004020 con 0x5622924558b0 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f4fff400640 1 -- 172.21.15.119:0/100931739 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50040041c0 con 0x5622924558b0 2024-03-28T18:48:28.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f4fff400640 1 -- 172.21.15.119:0/100931739 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50040114b0 con 0x5622924558b0 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f501c585740 1 -- 172.21.15.119:0/100931739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 msgr2=0x56229245e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.235+0000 7f501c585740 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5004011010 tx=0x7f500400bde0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 -- 172.21.15.119:0/100931739 shutdown_connections 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292456da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x56229245e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 --2- 172.21.15.119:0/100931739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x56229245e180 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 -- 172.21.15.119:0/100931739 >> 172.21.15.119:0/100931739 conn(0x56229244ff80 msgr2=0x562292456690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 -- 172.21.15.119:0/100931739 shutdown_connections 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 -- 172.21.15.119:0/100931739 wait complete. 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 -- start start 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.236+0000 7f501c585740 1 Processor -- start 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x562292466fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292464f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b4a30 con 0x56229245ede0 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b43d0 con 0x5622924558b0 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50141b18d0 con 0x562292456260 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f501c585740 10 client.0 _ll_register_callbacks cb 0x7f5000004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x562292466fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33748/0 (socket says 172.21.15.119:33748) 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 -- 172.21.15.119:0/53398126 learned_addr learned my addr 172.21.15.119:0/53398126 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5013400640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292464f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 -- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 msgr2=0x562292466fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x562292466fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 -- 172.21.15.119:0/53398126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 msgr2=0x562292464f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292464f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.237+0000 7f5012a00640 1 -- 172.21.15.119:0/53398126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562292465730 con 0x5622924558b0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f5012a00640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5004003cc0 tx=0x7f5004005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=301a581be2b480bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500400b710 con 0x5622924558b0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50040041c0 con 0x5622924558b0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f5004003d60 con 0x5622924558b0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5004038050 con 0x5622924558b0 2024-03-28T18:48:28.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f501c585740 10 client.26708 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5622924cb080 con 0x5622924558b0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50040041c0 con 0x5622924558b0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.238+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5004029b70 con 0x5622924558b0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 10 client.26708 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 10 client.26708 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 20 client.26708 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5622924cb4a0 0x5622924cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 4 client.26708 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5622923c0b50 con 0x5622924cb4a0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f5012000640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5622924cb4a0 0x5622924cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f5012000640 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5622924cb4a0 0x5622924cd890 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eabd20d02279af0b server_cookie=955beb7336108ea3 in_seq=0 out_seq=0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f4ffcc00640 10 client.26708 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f4ffcc00640 1 -- 172.21.15.119:0/53398126 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5622923c0b50 con 0x5622924cb4a0 2024-03-28T18:48:28.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f4ffcc00640 10 client.26708 handle_command_reply: tid=0 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.239+0000 7f501c585740 1 client.26708 shutdown 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 msgr2=0x562292466aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5004003cc0 tx=0x7f5004005d90 comp rx=0 tx=0).stop 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 shutdown_connections 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5622924cb4a0 0x5622924cd890 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56229245ede0 0x562292464f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562292456260 0x562292466fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f4ffcc00640 0 client.26708 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 --2- 172.21.15.119:0/53398126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5622924558b0 0x562292466aa0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 >> 172.21.15.119:0/53398126 conn(0x56229244ff80 msgr2=0x562292456ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 shutdown_connections 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 1 -- 172.21.15.119:0/53398126 wait complete. 2024-03-28T18:48:28.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.240+0000 7f501c585740 20 client.26708 trim_cache size 0 max 16384 2024-03-28T18:48:28.246 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:48:28.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 msgr2=0x7f4ff80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 0x7f4ff80a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f500c005fd0 tx=0x7f500c009500 comp rx=0 tx=0).stop 2024-03-28T18:48:28.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 msgr2=0x7f501419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f500002c250 tx=0x7f5000004360 comp rx=0 tx=0).stop 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 shutdown_connections 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4ff80a2970 0x7f4ff80a4e30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50141a2cb0 0x7f50141a3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f501410cea0 0x7f501419e980 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 --2- 172.21.15.119:0/1294333596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014107e50 0x7f501419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 >> 172.21.15.119:0/1294333596 conn(0x7f50140774c0 msgr2=0x7f50140790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 shutdown_connections 2024-03-28T18:48:28.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.241+0000 7f5013e00640 1 -- 172.21.15.119:0/1294333596 wait complete. 2024-03-28T18:48:28.259 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:48:28.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:48:28.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 -- 172.21.15.119:0/154023955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a4075bd0 msgr2=0x7fe5a4075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 --2- 172.21.15.119:0/154023955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a4075bd0 0x7fe5a4075fb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe594009930 tx=0x7fe59402bc20 comp rx=0 tx=0).stop 2024-03-28T18:48:28.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 -- 172.21.15.119:0/154023955 shutdown_connections 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 --2- 172.21.15.119:0/154023955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 --2- 172.21.15.119:0/154023955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a40764f0 0x7fe5a4111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 --2- 172.21.15.119:0/154023955 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a4075bd0 0x7fe5a4075fb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 -- 172.21.15.119:0/154023955 >> 172.21.15.119:0/154023955 conn(0x7fe5a40ff2f0 msgr2=0x7fe5a4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 -- 172.21.15.119:0/154023955 shutdown_connections 2024-03-28T18:48:28.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.467+0000 7fe5a9000640 1 -- 172.21.15.119:0/154023955 wait complete. 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.468+0000 7fe5a9000640 1 Processor -- start 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.468+0000 7fe5a9000640 1 -- start start 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a40764f0 0x7fe5a419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4118090 con 0x7fe5a41124e0 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4117f10 con 0x7fe5a40764f0 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a9000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4118210 con 0x7fe5a4075bd0 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a40764f0 0x7fe5a419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55418/0 (socket says 172.21.15.119:55418) 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36576/0 (socket says 172.21.15.119:36576) 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a2800640 1 -- 172.21.15.119:0/2008137296 learned_addr learned my addr 172.21.15.119:0/2008137296 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2008137296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a40764f0 msgr2=0x7fe5a419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a40764f0 0x7fe5a419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 msgr2=0x7fe5a41a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2008137296 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a41a3590 con 0x7fe5a4075bd0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.469+0000 7fe5a2800640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe594036040 tx=0x7fe594004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe59aa00640 1 -- 172.21.15.119:0/2008137296 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594043020 con 0x7fe5a4075bd0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe59aa00640 1 -- 172.21.15.119:0/2008137296 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe594005000 con 0x7fe5a4075bd0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe59aa00640 1 -- 172.21.15.119:0/2008137296 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594003a50 con 0x7fe5a4075bd0 2024-03-28T18:48:28.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a41a3820 con 0x7fe5a4075bd0 2024-03-28T18:48:28.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.470+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a41ab110 con 0x7fe5a4075bd0 2024-03-28T18:48:28.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.473+0000 7fe59aa00640 1 -- 172.21.15.119:0/2008137296 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe594046020 con 0x7fe5a4075bd0 2024-03-28T18:48:28.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 Processor -- start 2024-03-28T18:48:28.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 -- start start 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513349050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5132a12c0 con 0x55f513346c00 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f513249c90 con 0x55f51333d6d0 2024-03-28T18:48:28.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.474+0000 7fe5abe14740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f51329d360 con 0x55f51333e080 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33786/0 (socket says 172.21.15.119:33786) 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a2800640 1 -- 172.21.15.119:0/1272914400 learned_addr learned my addr 172.21.15.119:0/1272914400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1400640 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513349050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe59aa00640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 0x7fe5880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1272914400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 msgr2=0x55f51333dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1272914400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 msgr2=0x55f513349050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513349050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1272914400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f51326e080 con 0x55f51333e080 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a2800640 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe59aa00640 1 -- 172.21.15.119:0/2008137296 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe5940f0bf0 con 0x7fe5a4075bd0 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a2800640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 0x7fe5880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.475+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe5940ee390 tx=0x7fe594088010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec0559e2f6bfe3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe593400640 1 -- 172.21.15.119:0/1272914400 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940f4050 con 0x55f51333e080 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe593400640 1 -- 172.21.15.119:0/1272914400 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5940dbda0 con 0x55f51333e080 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe593400640 1 -- 172.21.15.119:0/1272914400 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940bcc00 con 0x55f51333e080 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 -- 172.21.15.119:0/1272914400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 msgr2=0x55f513346630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346630 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe5940ee390 tx=0x7fe594088010 comp rx=0 tx=0).stop 2024-03-28T18:48:28.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5a2800640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 0x7fe5880a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0094d0 tx=0x7fe59c004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 -- 172.21.15.119:0/1272914400 shutdown_connections 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513349050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346630 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1272914400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333d6d0 0x55f51333dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 -- 172.21.15.119:0/1272914400 >> 172.21.15.119:0/1272914400 conn(0x55f513337da0 msgr2=0x55f513338180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 -- 172.21.15.119:0/1272914400 shutdown_connections 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.476+0000 7fe5abe14740 1 -- 172.21.15.119:0/1272914400 wait complete. 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.477+0000 7fe5abe14740 1 -- start start 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.477+0000 7fe5abe14740 1 Processor -- start 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.477+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f5133438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513343ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f513349930 con 0x55f513346c00 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f51325d0d0 con 0x55f51333e080 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f513349ab0 con 0x55f51333d6d0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5abe14740 10 client.0 _ll_register_callbacks cb 0x55f513261e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f5133438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36588/0 (socket says 172.21.15.119:36588) 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2205549170 learned_addr learned my addr 172.21.15.119:0/2205549170 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a2800640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513343ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2205549170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 msgr2=0x55f5133438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f5133438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2205549170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 msgr2=0x55f513343ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513343ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2205549170 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f512fb53f0 con 0x55f51333d6d0 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f5133438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe5940b9700 tx=0x7fe5940e1180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5246da780a47e9ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.478+0000 7fe587e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940bcc00 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe59408e0c0 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5940dbda0 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5940bd9a0 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 10 client.26701 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f51307f7d0 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5940dbda0 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe5940b9910 con 0x55f51333d6d0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 10 client.26701 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 10 client.26701 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 20 client.26701 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133565c0 0x55f5133589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 4 client.26701 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.479+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5132ae800 con 0x55f5133565c0 2024-03-28T18:48:28.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.480+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133565c0 0x55f5133589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.480+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133565c0 0x55f5133589b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6362f2325cbf555 server_cookie=1e12afed416dc3a7 in_seq=0 out_seq=0 2024-03-28T18:48:28.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.480+0000 7fe587e00640 10 client.26701 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.481+0000 7fe587e00640 1 -- 172.21.15.119:0/2205549170 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f5132ae800 con 0x55f5133565c0 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.481+0000 7fe587e00640 10 client.26701 handle_command_reply: tid=0 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.481+0000 7fe5abe14740 1 client.26701 shutdown 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 msgr2=0x55f513346530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe5940b9700 tx=0x7fe5940e1180 comp rx=0 tx=0).stop 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 shutdown_connections 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133565c0 0x55f5133589b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f513343ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f5133438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2205549170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f513346530 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 >> 172.21.15.119:0/2205549170 conn(0x55f513337da0 msgr2=0x55f513339af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe587e00640 0 client.26701 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 shutdown_connections 2024-03-28T18:48:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 1 -- 172.21.15.119:0/2205549170 wait complete. 2024-03-28T18:48:28.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.482+0000 7fe5abe14740 20 client.26701 trim_cache size 0 max 16384 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.495+0000 7fe5abe14740 1 Processor -- start 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.495+0000 7fe5abe14740 1 -- start start 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f513346c00 0x55f513340b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b3970 con 0x55f51333d6d0 2024-03-28T18:48:28.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b52f0 con 0x55f513346c00 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5abe14740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b3fd0 con 0x55f51333e080 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55452/0 (socket says 172.21.15.119:55452) 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1891414017 learned_addr learned my addr 172.21.15.119:0/1891414017 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f513346c00 0x55f513340b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1400640 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1891414017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 msgr2=0x55f513346530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1891414017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f513346c00 msgr2=0x55f513340b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f513346c00 0x55f513340b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.496+0000 7fe5a1e00640 1 -- 172.21.15.119:0/1891414017 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f513338fd0 con 0x55f51333d6d0 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe594005ab0 tx=0x7fe5940d1550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66467b0b2157e355 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe592a00640 1 -- 172.21.15.119:0/1891414017 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940bcc00 con 0x55f51333d6d0 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe592a00640 1 -- 172.21.15.119:0/1891414017 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5940dbda0 con 0x55f51333d6d0 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe592a00640 1 -- 172.21.15.119:0/1891414017 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940054b0 con 0x55f51333d6d0 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe5abe14740 1 -- 172.21.15.119:0/1891414017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 msgr2=0x55f513345ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.497+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe594005ab0 tx=0x7fe5940d1550 comp rx=0 tx=0).stop 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 -- 172.21.15.119:0/1891414017 shutdown_connections 2024-03-28T18:48:28.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f513346c00 0x55f513340b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333e080 0x55f513346530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 --2- 172.21.15.119:0/1891414017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f51333d6d0 0x55f513345ff0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 -- 172.21.15.119:0/1891414017 >> 172.21.15.119:0/1891414017 conn(0x55f513337da0 msgr2=0x55f513336eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 -- 172.21.15.119:0/1891414017 shutdown_connections 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 -- 172.21.15.119:0/1891414017 wait complete. 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 -- start start 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.498+0000 7fe5abe14740 1 Processor -- start 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f51334dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b4c90 con 0x55f513346c00 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b4630 con 0x55f51333e080 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41b1b30 con 0x55f51333d6d0 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5abe14740 10 client.0 _ll_register_callbacks cb 0x7fe58c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33816/0 (socket says 172.21.15.119:33816) 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36602/0 (socket says 172.21.15.119:36602) 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1400640 1 -- 172.21.15.119:0/2975139362 learned_addr learned my addr 172.21.15.119:0/2975139362 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a2800640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f51334dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2975139362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 msgr2=0x55f51334d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2975139362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 msgr2=0x55f51334dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f51334dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1e00640 1 -- 172.21.15.119:0/2975139362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5133b1e30 con 0x55f51333d6d0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.499+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5a1e00640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5940bb550 tx=0x7fe5940bd130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de8f868dc75b03d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5940bcc00 con 0x55f51333d6d0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe5940054b0 con 0x55f51333d6d0 2024-03-28T18:48:28.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5940dbda0 con 0x55f51333d6d0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5940dd0b0 con 0x55f51333d6d0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 10 client.26707 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f5133b1e30 con 0x55f51333d6d0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5940dbda0 con 0x55f51333d6d0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe594003bf0 con 0x55f51333d6d0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 10 client.26707 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 10 client.26707 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 20 client.26707 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133b3380 0x55f5133b5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 4 client.26707 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.500+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f5132a8b50 con 0x55f5133b3380 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133b3380 0x55f5133b5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe5a1400640 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133b3380 0x55f5133b5770 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d87b3b4fa2fc83ac server_cookie=25e6340341293c0a in_seq=0 out_seq=0 2024-03-28T18:48:28.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe587e00640 10 client.26707 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe587e00640 1 -- 172.21.15.119:0/2975139362 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f5132a8b50 con 0x55f5133b3380 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe587e00640 10 client.26707 handle_command_reply: tid=0 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.501+0000 7fe5abe14740 1 client.26707 shutdown 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 msgr2=0x55f51334d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5940bb550 tx=0x7fe5940bd130 comp rx=0 tx=0).stop 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 shutdown_connections 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55f5133b3380 0x55f5133b5770 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f513346c00 0x55f51334dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe587e00640 0 client.26707 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f51333e080 0x55f51334d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 --2- 172.21.15.119:0/2975139362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f51333d6d0 0x55f51334d1b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 >> 172.21.15.119:0/2975139362 conn(0x55f513337da0 msgr2=0x55f513336620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 shutdown_connections 2024-03-28T18:48:28.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 1 -- 172.21.15.119:0/2975139362 wait complete. 2024-03-28T18:48:28.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5abe14740 20 client.26707 trim_cache size 0 max 16384 2024-03-28T18:48:28.507 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 msgr2=0x7fe5880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 0x7fe5880a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0094d0 tx=0x7fe59c004a40 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 msgr2=0x7fe5a419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.502+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe594036040 tx=0x7fe594004250 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 shutdown_connections 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe5880a2970 0x7fe5880a4e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a41124e0 0x7fe5a41a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe5a40764f0 0x7fe5a419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 --2- 172.21.15.119:0/2008137296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5a4075bd0 0x7fe5a419e5f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 >> 172.21.15.119:0/2008137296 conn(0x7fe5a40ff2f0 msgr2=0x7fe5a41016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 shutdown_connections 2024-03-28T18:48:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.503+0000 7fe5a9000640 1 -- 172.21.15.119:0/2008137296 wait complete. 2024-03-28T18:48:28.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:28 smithi119 ceph-mon[31813]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:28.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:28 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3831871304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:48:28.520 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:48:28.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:48:28.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:28 smithi196 ceph-mon[36300]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:28.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:28 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3831871304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:48:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:28 smithi131 ceph-mon[36065]: pgmap v135: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:28 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3831871304' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:48:28.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 -- 172.21.15.119:0/759439466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604103840 msgr2=0x7f7604103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 --2- 172.21.15.119:0/759439466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604103840 0x7f7604103cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009890 tx=0x7f75f002bb70 comp rx=0 tx=0).stop 2024-03-28T18:48:28.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 -- 172.21.15.119:0/759439466 shutdown_connections 2024-03-28T18:48:28.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 --2- 172.21.15.119:0/759439466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604104200 0x7f7604110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 --2- 172.21.15.119:0/759439466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604103840 0x7f7604103cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 --2- 172.21.15.119:0/759439466 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f7604109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 -- 172.21.15.119:0/759439466 >> 172.21.15.119:0/759439466 conn(0x7f76040ff2f0 msgr2=0x7f7604101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 -- 172.21.15.119:0/759439466 shutdown_connections 2024-03-28T18:48:28.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.721+0000 7f760a600640 1 -- 172.21.15.119:0/759439466 wait complete. 2024-03-28T18:48:28.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.722+0000 7f760a600640 1 Processor -- start 2024-03-28T18:48:28.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.722+0000 7f760a600640 1 -- start start 2024-03-28T18:48:28.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 0x7f760419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604113c20 con 0x7f7604103840 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604113aa0 con 0x7f7604109850 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f760a600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604113da0 con 0x7f7604104200 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7603400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 0x7f760419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7603e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36622/0 (socket says 172.21.15.119:36622) 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7603e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33838/0 (socket says 172.21.15.119:33838) 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 -- 172.21.15.119:0/4044449035 learned_addr learned my addr 172.21.15.119:0/4044449035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 -- 172.21.15.119:0/4044449035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 msgr2=0x7f760419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 -- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 msgr2=0x7f760419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 0x7f760419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7602a00640 1 -- 172.21.15.119:0/4044449035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760419f1b0 con 0x7f7604104200 2024-03-28T18:48:28.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7603e00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.723+0000 7f7603400640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 0x7f760419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f7602a00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75f00057c0 tx=0x7f75f00045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f75fbe00640 1 -- 172.21.15.119:0/4044449035 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0041020 con 0x7f7604104200 2024-03-28T18:48:28.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f75fbe00640 1 -- 172.21.15.119:0/4044449035 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75f002c9f0 con 0x7f7604104200 2024-03-28T18:48:28.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f75fbe00640 1 -- 172.21.15.119:0/4044449035 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0039040 con 0x7f7604104200 2024-03-28T18:48:28.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760419f440 con 0x7f7604104200 2024-03-28T18:48:28.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.724+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76041a6d30 con 0x7f7604104200 2024-03-28T18:48:28.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.727+0000 7f760d3c3740 1 Processor -- start 2024-03-28T18:48:28.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.727+0000 7f760d3c3740 1 -- start start 2024-03-28T18:48:28.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32855630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32858050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f327b02c0 con 0x555f32855c00 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f32758c90 con 0x555f3284c6d0 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f760d3c3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f327ac360 con 0x555f3284d080 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f75fbe00640 1 -- 172.21.15.119:0/4044449035 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f75f0044020 con 0x7f7604104200 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7602a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32858050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32855630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33840/0 (socket says 172.21.15.119:33840) 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 -- 172.21.15.119:0/3610635220 learned_addr learned my addr 172.21.15.119:0/3610635220 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 -- 172.21.15.119:0/3610635220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 msgr2=0x555f32855630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32855630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 -- 172.21.15.119:0/3610635220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 msgr2=0x555f32858050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32858050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.728+0000 7f7603e00640 1 -- 172.21.15.119:0/3610635220 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f3277d080 con 0x555f3284c6d0 2024-03-28T18:48:28.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f7603e00640 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0117e0 tx=0x7f75fc011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eff815b950fd6e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f75f8c00640 1 -- 172.21.15.119:0/3610635220 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc011310 con 0x555f3284c6d0 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f75f8c00640 1 -- 172.21.15.119:0/3610635220 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75fc003e50 con 0x555f3284c6d0 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f75f8c00640 1 -- 172.21.15.119:0/3610635220 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0045c0 con 0x555f3284c6d0 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f760d3c3740 1 -- 172.21.15.119:0/3610635220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 msgr2=0x555f3284cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.729+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0117e0 tx=0x7f75fc011ca0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f75fbe00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 0x7f75e80a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 -- 172.21.15.119:0/3610635220 shutdown_connections 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32858050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32855630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3610635220 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f3284cab0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 -- 172.21.15.119:0/3610635220 >> 172.21.15.119:0/3610635220 conn(0x555f32846da0 msgr2=0x555f32847180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 -- 172.21.15.119:0/3610635220 shutdown_connections 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f75fbe00640 1 -- 172.21.15.119:0/4044449035 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f75f00f06c0 con 0x7f7604104200 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 -- 172.21.15.119:0/3610635220 wait complete. 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f7603e00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 0x7f75e80a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.730+0000 7f760d3c3740 1 -- start start 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f7603e00640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 0x7f75e80a4f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75fc010010 tx=0x7f75fc005bd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 Processor -- start 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32852940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32853080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f32858b70 con 0x555f32855c00 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f3276c0d0 con 0x555f3284c6d0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f32858cf0 con 0x555f3284d080 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f760d3c3740 10 client.0 _ll_register_callbacks cb 0x555f32770e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f7603400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f7602a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32852940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f7603400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33852/0 (socket says 172.21.15.119:33852) 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.731+0000 7f7603400640 1 -- 172.21.15.119:0/3633525381 learned_addr learned my addr 172.21.15.119:0/3633525381 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603e00640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32853080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 -- 172.21.15.119:0/3633525381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 msgr2=0x555f32852940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32852940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 -- 172.21.15.119:0/3633525381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 msgr2=0x555f32853080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32853080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 -- 172.21.15.119:0/3633525381 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f324c43f0 con 0x555f3284c6d0 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7602a00640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32852940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603e00640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32853080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f7603400640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f75f4012d20 tx=0x7f75f400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b7f7a4eaea7068 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f75ed600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4012110 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f75f4004050 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.732+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75fc003e50 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75f40122b0 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 10 client.26738 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555f3258e7d0 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75fc003e50 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75f40104c0 con 0x555f3284c6d0 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 10 client.26738 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 10 client.26738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 20 client.26738 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f32865b60 0x555f32867f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 4 client.26738 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.733+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f327bd800 con 0x555f32865b60 2024-03-28T18:48:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.734+0000 7f7602a00640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f32865b60 0x555f32867f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.734+0000 7f75ed600640 10 client.26738 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:28.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.734+0000 7f7602a00640 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f32865b60 0x555f32867f70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28864926ee40c724 server_cookie=da61972990935efd in_seq=0 out_seq=0 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f75ed600640 1 -- 172.21.15.119:0/3633525381 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555f327bd800 con 0x555f32865b60 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f75ed600640 10 client.26738 handle_command_reply: tid=0 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 client.26738 shutdown 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 msgr2=0x555f328555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f75f4012d20 tx=0x7f75f400f790 comp rx=0 tx=0).stop 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 shutdown_connections 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f32865b60 0x555f32867f70 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f32855c00 0x555f32853080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284d080 0x555f32852940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 --2- 172.21.15.119:0/3633525381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f328555b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f75ed600640 0 client.26738 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 >> 172.21.15.119:0/3633525381 conn(0x555f32846da0 msgr2=0x555f32848af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 shutdown_connections 2024-03-28T18:48:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 1 -- 172.21.15.119:0/3633525381 wait complete. 2024-03-28T18:48:28.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.735+0000 7f760d3c3740 20 client.26738 trim_cache size 0 max 16384 2024-03-28T18:48:28.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 Processor -- start 2024-03-28T18:48:28.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 -- start start 2024-03-28T18:48:28.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041af660 con 0x555f3284d080 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041b0fe0 con 0x555f3284c6d0 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f760d3c3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041afcc0 con 0x555f32855c00 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7602a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7603400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7603e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7603400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:33864/0 (socket says 172.21.15.119:33864) 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7603e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36660/0 (socket says 172.21.15.119:36660) 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7602a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55486/0 (socket says 172.21.15.119:55486) 2024-03-28T18:48:28.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.749+0000 7f7603e00640 1 -- 172.21.15.119:0/315834722 learned_addr learned my addr 172.21.15.119:0/315834722 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 -- 172.21.15.119:0/315834722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 msgr2=0x555f3284fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 -- 172.21.15.119:0/315834722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 msgr2=0x555f32855070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 -- 172.21.15.119:0/315834722 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f328468d0 con 0x555f3284d080 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7603e00640 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7603400640 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f7602a00640 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75f00ede60 tx=0x7f75f00f6530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a19ba3cd5be8010 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f75efe00640 1 -- 172.21.15.119:0/315834722 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f00bc8b0 con 0x555f3284d080 2024-03-28T18:48:28.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f75efe00640 1 -- 172.21.15.119:0/315834722 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75f00bd470 con 0x555f3284d080 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.750+0000 7f75efe00640 1 -- 172.21.15.119:0/315834722 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f00bc6d0 con 0x555f3284d080 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- 172.21.15.119:0/315834722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 msgr2=0x555f328555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f75f00ede60 tx=0x7f75f00f6530 comp rx=0 tx=0).stop 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- 172.21.15.119:0/315834722 shutdown_connections 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f32855c00 0x555f3284fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f328555b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 --2- 172.21.15.119:0/315834722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f3284c6d0 0x555f32855070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- 172.21.15.119:0/315834722 >> 172.21.15.119:0/315834722 conn(0x555f32846da0 msgr2=0x555f328489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- 172.21.15.119:0/315834722 shutdown_connections 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- 172.21.15.119:0/315834722 wait complete. 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.751+0000 7f760d3c3740 1 -- start start 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 Processor -- start 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f3285bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f32855c00 0x555f328c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041b0980 con 0x555f3284d080 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041b0320 con 0x555f32855c00 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041ad820 con 0x555f3284c6d0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f760d3c3740 10 client.0 _ll_register_callbacks cb 0x7f75f4009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f7603400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f7603400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:36670/0 (socket says 172.21.15.119:36670) 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f7603400640 1 -- 172.21.15.119:0/641483943 learned_addr learned my addr 172.21.15.119:0/641483943 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f7603e00640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f32855c00 0x555f328c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.752+0000 7f7602a00640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f3285bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 -- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f32855c00 msgr2=0x555f328c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f32855c00 0x555f328c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 -- 172.21.15.119:0/641483943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 msgr2=0x555f3285bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f3285bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 -- 172.21.15.119:0/641483943 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f328c0950 con 0x555f3284c6d0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f7603400640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75f4010960 tx=0x7f75f40106c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84da2422078e2ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f75ed600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:48:28.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4031060 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75f00bd470 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.753+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f75f40052d0 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75f40054d0 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 10 client.26725 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555f328c1bc0 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75f00bd470 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75f401c420 con 0x555f3284c6d0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 10 client.26725 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 10 client.26725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 20 client.26725 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f328c2210 0x555f328c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 4 client.26725 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.754+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555f327b7b50 con 0x555f328c2210 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f7602a00640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f328c2210 0x555f328c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:28.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f7602a00640 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f328c2210 0x555f328c4630 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a568767f44d7f491 server_cookie=558c74379a135ce4 in_seq=0 out_seq=0 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f75ed600640 10 client.26725 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f75ed600640 1 -- 172.21.15.119:0/641483943 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555f327b7b50 con 0x555f328c2210 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f75ed600640 10 client.26725 handle_command_reply: tid=0 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 client.26725 shutdown 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 msgr2=0x555f3285ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75f4010960 tx=0x7f75f40106c0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 shutdown_connections 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555f328c2210 0x555f328c4630 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555f32855c00 0x555f328c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f75ed600640 0 client.26725 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555f3284d080 0x555f3285bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 --2- 172.21.15.119:0/641483943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555f3284c6d0 0x555f3285ba30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.755+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 >> 172.21.15.119:0/641483943 conn(0x555f32846da0 msgr2=0x555f32848120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 shutdown_connections 2024-03-28T18:48:28.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760d3c3740 1 -- 172.21.15.119:0/641483943 wait complete. 2024-03-28T18:48:28.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760d3c3740 20 client.26725 trim_cache size 0 max 16384 2024-03-28T18:48:28.760 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 msgr2=0x7f75e80a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 0x7f75e80a4f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75fc010010 tx=0x7f75fc005bd0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 msgr2=0x7f760419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75f00057c0 tx=0x7f75f00045d0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 shutdown_connections 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75e80a2a70 0x7f75e80a4f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7604109850 0x7f760419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7604104200 0x7f760419a700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 --2- 172.21.15.119:0/4044449035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7604103840 0x7f760419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.756+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 >> 172.21.15.119:0/4044449035 conn(0x7f76040ff2f0 msgr2=0x7f76040ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.757+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 shutdown_connections 2024-03-28T18:48:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:28.757+0000 7f760a600640 1 -- 172.21.15.119:0/4044449035 wait complete. 2024-03-28T18:48:29.774 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:48:29.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:48:29.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:48:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 -- 172.21.15.119:0/3144957291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 msgr2=0x7f6bb8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3144957291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb8109c30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009ca0 tx=0x7f6ba802c100 comp rx=0 tx=0).stop 2024-03-28T18:48:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 -- 172.21.15.119:0/3144957291 shutdown_connections 2024-03-28T18:48:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3144957291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 0x7f6bb8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3144957291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3144957291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb8109c30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 -- 172.21.15.119:0/3144957291 >> 172.21.15.119:0/3144957291 conn(0x7f6bb80ff2f0 msgr2=0x7f6bb8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 -- 172.21.15.119:0/3144957291 shutdown_connections 2024-03-28T18:48:30.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.003+0000 7f6bbd600640 1 -- 172.21.15.119:0/3144957291 wait complete. 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 Processor -- start 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 -- start start 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 0x7f6bb819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8113c20 con 0x7f6bb8104200 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8113aa0 con 0x7f6bb8109850 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.004+0000 7f6bbd600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8113da0 con 0x7f6bb8103840 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41356/0 (socket says 172.21.15.119:41356) 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 -- 172.21.15.119:0/3387009843 learned_addr learned my addr 172.21.15.119:0/3387009843 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bae000640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 0x7f6bb819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 -- 172.21.15.119:0/3387009843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 msgr2=0x7f6bb819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 -- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 msgr2=0x7f6bb819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 0x7f6bb819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 -- 172.21.15.119:0/3387009843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb819f130 con 0x7f6bb8103840 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.005+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8005e30 tx=0x7f6ba80346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.006+0000 7f6bafe00640 1 -- 172.21.15.119:0/3387009843 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8039050 con 0x7f6bb8103840 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.006+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb819f3c0 con 0x7f6bb8103840 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.006+0000 7f6bafe00640 1 -- 172.21.15.119:0/3387009843 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ba8045070 con 0x7f6bb8103840 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.006+0000 7f6bafe00640 1 -- 172.21.15.119:0/3387009843 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8004580 con 0x7f6bb8103840 2024-03-28T18:48:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.006+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb81a6cb0 con 0x7f6bb8103840 2024-03-28T18:48:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.009+0000 7f6bafe00640 1 -- 172.21.15.119:0/3387009843 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6ba8003970 con 0x7f6bb8103840 2024-03-28T18:48:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.009+0000 7f6bc03d7740 1 Processor -- start 2024-03-28T18:48:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.009+0000 7f6bc03d7740 1 -- start start 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d9a0 0x55780de96040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de98970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780ddf1eb0 con 0x55780de96580 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780dd9aa60 con 0x55780de8d080 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780ddedf50 con 0x55780de8d9a0 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58874/0 (socket says 172.21.15.119:58874) 2024-03-28T18:48:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 -- 172.21.15.119:0/3912794514 learned_addr learned my addr 172.21.15.119:0/3912794514 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bae000640 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de98970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d9a0 0x55780de96040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 -- 172.21.15.119:0/3912794514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d9a0 msgr2=0x55780de96040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d9a0 0x55780de96040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 -- 172.21.15.119:0/3912794514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 msgr2=0x55780de98970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de98970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.010+0000 7f6bb7400640 1 -- 172.21.15.119:0/3912794514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55780dbce7e0 con 0x55780de8d080 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0011440 tx=0x7f6bb0011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae0d30230f857fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6bafe00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 0x7f6b9c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6ba7400640 1 -- 172.21.15.119:0/3912794514 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0004020 con 0x55780de8d080 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6ba7400640 1 -- 172.21.15.119:0/3912794514 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bb00041c0 con 0x55780de8d080 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6ba7400640 1 -- 172.21.15.119:0/3912794514 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000f940 con 0x55780de8d080 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6bafe00640 1 -- 172.21.15.119:0/3387009843 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6ba80f1400 con 0x7f6bb8103840 2024-03-28T18:48:30.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.011+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 0x7f6b9c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3912794514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 msgr2=0x55780de8d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0011440 tx=0x7f6bb0011900 comp rx=0 tx=0).stop 2024-03-28T18:48:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3912794514 shutdown_connections 2024-03-28T18:48:30.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de98970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d9a0 0x55780de96040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3912794514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de8d460 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bb6a00640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 0x7f6b9c0a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55780de8cb50 tx=0x7f6ba80f4eb0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3912794514 >> 172.21.15.119:0/3912794514 conn(0x55780de877b0 msgr2=0x55780de87b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3912794514 shutdown_connections 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3912794514 wait complete. 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.012+0000 7f6bc03d7740 1 -- start start 2024-03-28T18:48:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 Processor -- start 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de93230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de93770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780de99370 con 0x55780de96580 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780ddadeb0 con 0x55780de8d9a0 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55780de994f0 con 0x55780de8d080 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bc03d7740 10 client.0 _ll_register_callbacks cb 0x55780dda2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:30.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bae000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de93230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41360/0 (socket says 172.21.15.119:41360) 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.013+0000 7f6bb7400640 1 -- 172.21.15.119:0/3031188548 learned_addr learned my addr 172.21.15.119:0/3031188548 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de93770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 -- 172.21.15.119:0/3031188548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 msgr2=0x55780de95ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 -- 172.21.15.119:0/3031188548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 msgr2=0x55780de93770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de93770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 -- 172.21.15.119:0/3031188548 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55780dc2ef70 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de93770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bae000640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de93230 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8101990 tx=0x7f6ba0012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac474b5448400ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6b9be00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000f920 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6bb00041c0 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.014+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6ba000fac0 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ba000fcc0 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 10 client.26756 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55780dac6170 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bb00041c0 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6ba001c420 con 0x55780de8d9a0 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 10 client.26756 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 20 client.26756 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780dea6680 0x55780dea8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 4 client.26756 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55780ddff3f0 con 0x55780dea6680 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.015+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780dea6680 0x55780dea8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.016+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780dea6680 0x55780dea8aa0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d373cba3624c04 server_cookie=c7e3c036d334d0c7 in_seq=0 out_seq=0 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.016+0000 7f6b9be00640 10 client.26756 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6b9be00640 1 -- 172.21.15.119:0/3031188548 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55780ddff3f0 con 0x55780dea6680 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6b9be00640 10 client.26756 handle_command_reply: tid=0 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 client.26756 shutdown 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 msgr2=0x55780de93230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de93230 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8101990 tx=0x7f6ba0012b60 comp rx=0 tx=0).stop 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 shutdown_connections 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780dea6680 0x55780dea8aa0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de93770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de93230 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6b9be00640 0 client.26756 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3031188548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 >> 172.21.15.119:0/3031188548 conn(0x55780de877b0 msgr2=0x55780de89500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 shutdown_connections 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3031188548 wait complete. 2024-03-28T18:48:30.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.017+0000 7f6bc03d7740 20 client.26756 trim_cache size 0 max 16384 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 Processor -- start 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 -- start start 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81af560 con 0x55780de96580 2024-03-28T18:48:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81b0ee0 con 0x55780de8d9a0 2024-03-28T18:48:30.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.036+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81afbc0 con 0x55780de8d080 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58888/0 (socket says 172.21.15.119:58888) 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 -- 172.21.15.119:0/383581336 learned_addr learned my addr 172.21.15.119:0/383581336 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bb7400640 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 -- 172.21.15.119:0/383581336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 msgr2=0x55780de95960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 -- 172.21.15.119:0/383581336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 msgr2=0x55780de866b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de866b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 -- 172.21.15.119:0/383581336 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55780de82c20 con 0x55780de8d9a0 2024-03-28T18:48:30.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bb7400640 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.037+0000 7f6bae000640 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000f810 tx=0x7f6ba000f840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=170c483d7284b392 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6ba6a00640 1 -- 172.21.15.119:0/383581336 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba001c420 con 0x55780de8d9a0 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6ba6a00640 1 -- 172.21.15.119:0/383581336 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ba001c5c0 con 0x55780de8d9a0 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6ba6a00640 1 -- 172.21.15.119:0/383581336 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0003850 con 0x55780de8d9a0 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 -- 172.21.15.119:0/383581336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 msgr2=0x55780de95ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000f810 tx=0x7f6ba000f840 comp rx=0 tx=0).stop 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 -- 172.21.15.119:0/383581336 shutdown_connections 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de96580 0x55780de866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d9a0 0x55780de95ea0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/383581336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de8d080 0x55780de95960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 -- 172.21.15.119:0/383581336 >> 172.21.15.119:0/383581336 conn(0x55780de877b0 msgr2=0x55780de85fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 -- 172.21.15.119:0/383581336 shutdown_connections 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.038+0000 7f6bc03d7740 1 -- 172.21.15.119:0/383581336 wait complete. 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.039+0000 7f6bc03d7740 1 -- start start 2024-03-28T18:48:30.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.039+0000 7f6bc03d7740 1 Processor -- start 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 0x55780de9e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de96580 0x55780de9b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81b0880 con 0x55780de8d9a0 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81b0220 con 0x55780de8d080 2024-03-28T18:48:30.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81ad720 con 0x55780de96580 2024-03-28T18:48:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bc03d7740 10 client.0 _ll_register_callbacks cb 0x7f6ba0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bae000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 0x55780de9e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58900/0 (socket says 172.21.15.119:58900) 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 -- 172.21.15.119:0/3049916578 learned_addr learned my addr 172.21.15.119:0/3049916578 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de96580 0x55780de9b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 -- 172.21.15.119:0/3049916578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de96580 msgr2=0x55780de9b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de96580 0x55780de9b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 -- 172.21.15.119:0/3049916578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 msgr2=0x55780de9e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 0x55780de9e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bb7400640 1 -- 172.21.15.119:0/3049916578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55780de9beb0 con 0x55780de8d080 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.040+0000 7f6bae000640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 0x55780de9e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6bb7400640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009250 tx=0x7f6bb0011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c76c4f5a91ed9cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb001d040 con 0x55780de8d080 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6bb0005040 con 0x55780de8d080 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ba001c5c0 con 0x55780de8d080 2024-03-28T18:48:30.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bb002c2f0 con 0x55780de8d080 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6bc03d7740 10 client.26762 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55780df0c1e0 con 0x55780de8d080 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ba001c5c0 con 0x55780de8d080 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.041+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6bb0003680 con 0x55780de8d080 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 10 client.26762 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 20 client.26762 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780df0c5c0 0x55780df0e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 4 client.26762 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55780ddf9740 con 0x55780df0c5c0 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bae000640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780df0c5c0 0x55780df0e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6bae000640 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780df0c5c0 0x55780df0e9b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=759d28326b0136fc server_cookie=6c81226d71f3b3c in_seq=0 out_seq=0 2024-03-28T18:48:30.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.042+0000 7f6b9be00640 10 client.26762 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6b9be00640 1 -- 172.21.15.119:0/3049916578 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55780ddf9740 con 0x55780df0c5c0 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6b9be00640 10 client.26762 handle_command_reply: tid=0 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 client.26762 shutdown 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 msgr2=0x55780de9dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009250 tx=0x7f6bb0011900 comp rx=0 tx=0).stop 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 shutdown_connections 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55780df0c5c0 0x55780df0e9b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55780de96580 0x55780de9b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55780de8d9a0 0x55780de9e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 --2- 172.21.15.119:0/3049916578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55780de8d080 0x55780de9dd70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 >> 172.21.15.119:0/3049916578 conn(0x55780de877b0 msgr2=0x55780de86830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6b9be00640 0 client.26762 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 shutdown_connections 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 1 -- 172.21.15.119:0/3049916578 wait complete. 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.044+0000 7f6bc03d7740 20 client.26762 trim_cache size 0 max 16384 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "e8ec3066-50e4-4c20-8066-027513a79026", 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:48:30.051 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 msgr2=0x7f6b9c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 0x7f6b9c0a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55780de8cb50 tx=0x7f6ba80f4eb0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 msgr2=0x7f6bb819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8005e30 tx=0x7f6ba80346f0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 shutdown_connections 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b9c0a2970 0x7f6b9c0a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb8109850 0x7f6bb819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bb8104200 0x7f6bb819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 --2- 172.21.15.119:0/3387009843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bb8103840 0x7f6bb819a140 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 >> 172.21.15.119:0/3387009843 conn(0x7f6bb80ff2f0 msgr2=0x7f6bb80ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 shutdown_connections 2024-03-28T18:48:30.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.045+0000 7f6bbd600640 1 -- 172.21.15.119:0/3387009843 wait complete. 2024-03-28T18:48:30.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:48:30.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:48:30.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 -- 172.21.15.119:0/2606825898 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 msgr2=0x7f09f4073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 --2- 172.21.15.119:0/2606825898 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f4073e50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009ca0 tx=0x7f09ec02c180 comp rx=0 tx=0).stop 2024-03-28T18:48:30.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 -- 172.21.15.119:0/2606825898 shutdown_connections 2024-03-28T18:48:30.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 --2- 172.21.15.119:0/2606825898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f410ded0 0x7f09f41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 --2- 172.21.15.119:0/2606825898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f4074390 0x7f09f410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 --2- 172.21.15.119:0/2606825898 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f4073e50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 -- 172.21.15.119:0/2606825898 >> 172.21.15.119:0/2606825898 conn(0x7f09f4078aa0 msgr2=0x7f09f4078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 -- 172.21.15.119:0/2606825898 shutdown_connections 2024-03-28T18:48:30.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.277+0000 7f09fc800640 1 -- 172.21.15.119:0/2606825898 wait complete. 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 Processor -- start 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 -- start start 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f4074390 0x7f09f419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 0x7f09f419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f4113c30 con 0x7f09f4074390 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f4113ab0 con 0x7f09f4073a70 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.278+0000 7f09fc800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f4113db0 con 0x7f09f410ded0 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09fa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 0x7f09f419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f4074390 0x7f09f419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58920/0 (socket says 172.21.15.119:58920) 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 -- 172.21.15.119:0/543288202 learned_addr learned my addr 172.21.15.119:0/543288202 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 -- 172.21.15.119:0/543288202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 msgr2=0x7f09f419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 0x7f09f419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 -- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f4074390 msgr2=0x7f09f419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f4074390 0x7f09f419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 -- 172.21.15.119:0/543288202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f419f160 con 0x7f09f4073a70 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09fa800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 0x7f09f419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.279+0000 7f09f9e00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f09ec005e30 tx=0x7f09ec035e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.280+0000 7f09eaa00640 1 -- 172.21.15.119:0/543288202 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec039040 con 0x7f09f4073a70 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.280+0000 7f09eaa00640 1 -- 172.21.15.119:0/543288202 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09ec044070 con 0x7f09f4073a70 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.280+0000 7f09eaa00640 1 -- 172.21.15.119:0/543288202 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec032960 con 0x7f09f4073a70 2024-03-28T18:48:30.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.280+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f419f3f0 con 0x7f09f4073a70 2024-03-28T18:48:30.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.280+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f41a6ce0 con 0x7f09f4073a70 2024-03-28T18:48:30.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.283+0000 7f09eaa00640 1 -- 172.21.15.119:0/543288202 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f09ec032b00 con 0x7f09f4073a70 2024-03-28T18:48:30.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.283+0000 7f09ff5f9740 1 Processor -- start 2024-03-28T18:48:30.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.283+0000 7f09ff5f9740 1 -- start start 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca57b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca586680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca4df250 con 0x5609ca584290 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca487a30 con 0x5609ca57bc00 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca4db2f0 con 0x5609ca57af20 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09f9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca57b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58926/0 (socket says 172.21.15.119:58926) 2024-03-28T18:48:30.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09f9400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca586680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 -- 172.21.15.119:0/1894625806 learned_addr learned my addr 172.21.15.119:0/1894625806 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 -- 172.21.15.119:0/1894625806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 msgr2=0x5609ca57b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca57b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 -- 172.21.15.119:0/1894625806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 msgr2=0x5609ca586680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca586680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09fa800640 1 -- 172.21.15.119:0/1894625806 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609ca472690 con 0x5609ca57bc00 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.284+0000 7f09f9e00640 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca57b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09fa800640 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009f50 tx=0x7f09f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41584c319284a46f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09eaa00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09df000640 1 -- 172.21.15.119:0/1894625806 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0004310 con 0x5609ca57bc00 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09df000640 1 -- 172.21.15.119:0/1894625806 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09f00044b0 con 0x5609ca57bc00 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09df000640 1 -- 172.21.15.119:0/1894625806 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f00035f0 con 0x5609ca57bc00 2024-03-28T18:48:30.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09f9e00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.285+0000 7f09eaa00640 1 -- 172.21.15.119:0/543288202 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f09ec0f1170 con 0x7f09f4073a70 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- 172.21.15.119:0/1894625806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 msgr2=0x5609ca57a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009f50 tx=0x7f09f00118d0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09f9e00640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 0x7f09e00a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5609ca57c150 tx=0x7f09ec0f4c20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- 172.21.15.119:0/1894625806 shutdown_connections 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca586680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca57b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/1894625806 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca57a950 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- 172.21.15.119:0/1894625806 >> 172.21.15.119:0/1894625806 conn(0x5609ca542200 msgr2=0x5609ca575970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- 172.21.15.119:0/1894625806 shutdown_connections 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- 172.21.15.119:0/1894625806 wait complete. 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 -- start start 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.286+0000 7f09ff5f9740 1 Processor -- start 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca583970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca581240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca587100 con 0x5609ca584290 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca49ae80 con 0x5609ca57bc00 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609ca587280 con 0x5609ca57af20 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09ff5f9740 10 client.0 _ll_register_callbacks cb 0x5609ca4ad130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09f9400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09fa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca583970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09f9400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58940/0 (socket says 172.21.15.119:58940) 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09f9400640 1 -- 172.21.15.119:0/2679146247 learned_addr learned my addr 172.21.15.119:0/2679146247 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.287+0000 7f09fa800640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca581240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 -- 172.21.15.119:0/2679146247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 msgr2=0x5609ca583970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca583970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 -- 172.21.15.119:0/2679146247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 msgr2=0x5609ca581240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca581240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 -- 172.21.15.119:0/2679146247 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609ca2bd7d0 con 0x5609ca57bc00 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09fa800640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca581240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09f9400640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f09e40126e0 tx=0x7f09e4012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c7442f9b7e1a91b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09d3400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40120d0 con 0x5609ca57bc00 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09f00044b0 con 0x5609ca57bc00 2024-03-28T18:48:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f09e400f920 con 0x5609ca57bc00 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09e4012270 con 0x5609ca57bc00 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09ff5f9740 10 client.26774 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.288+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5609ca1f33f0 con 0x5609ca57bc00 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09f00044b0 con 0x5609ca57bc00 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f09e401c420 con 0x5609ca57bc00 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 10 client.26774 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 10 client.26774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 20 client.26774 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5942d0 0x5609ca5966c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 4 client.26774 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609ca4ec790 con 0x5609ca5942d0 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.289+0000 7f09fa800640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5942d0 0x5609ca5966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.290+0000 7f09fa800640 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5942d0 0x5609ca5966c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53b72adb6de2126b server_cookie=4aca5fdc2d8e9ce8 in_seq=0 out_seq=0 2024-03-28T18:48:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.290+0000 7f09d3400640 10 client.26774 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09d3400640 1 -- 172.21.15.119:0/2679146247 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5609ca4ec790 con 0x5609ca5942d0 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09d3400640 10 client.26774 handle_command_reply: tid=0 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 client.26774 shutdown 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 msgr2=0x5609ca580d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f09e40126e0 tx=0x7f09e4012ba0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 shutdown_connections 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5942d0 0x5609ca5966c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca581240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09d3400640 0 client.26774 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57bc00 0x5609ca580d00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/2679146247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57af20 0x5609ca583970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 >> 172.21.15.119:0/2679146247 conn(0x5609ca542200 msgr2=0x5609ca5776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 shutdown_connections 2024-03-28T18:48:30.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 1 -- 172.21.15.119:0/2679146247 wait complete. 2024-03-28T18:48:30.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.291+0000 7f09ff5f9740 20 client.26774 trim_cache size 0 max 16384 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 Processor -- start 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 -- start start 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca5774d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca583970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41af550 con 0x5609ca584290 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41b0ed0 con 0x5609ca57af20 2024-03-28T18:48:30.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41afbb0 con 0x5609ca57bc00 2024-03-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09f9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09f9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41414/0 (socket says 172.21.15.119:41414) 2024-03-28T18:48:30.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09f9400640 1 -- 172.21.15.119:0/3422175323 learned_addr learned my addr 172.21.15.119:0/3422175323 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.309+0000 7f09fa800640 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca5774d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09fa800640 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca583970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 -- 172.21.15.119:0/3422175323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 msgr2=0x5609ca5774d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca5774d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 -- 172.21.15.119:0/3422175323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 msgr2=0x5609ca583970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca583970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 -- 172.21.15.119:0/3422175323 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609ca1b30f0 con 0x5609ca57bc00 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09f9400640 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e4009420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61aca3c593f3991 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09de600640 1 -- 172.21.15.119:0/3422175323 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e401caf0 con 0x5609ca57bc00 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09de600640 1 -- 172.21.15.119:0/3422175323 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09e400fe00 con 0x5609ca57bc00 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.310+0000 7f09de600640 1 -- 172.21.15.119:0/3422175323 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4005680 con 0x5609ca57bc00 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 -- 172.21.15.119:0/3422175323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 msgr2=0x5609ca583430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f09e4000c00 tx=0x7f09e4009420 comp rx=0 tx=0).stop 2024-03-28T18:48:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 -- 172.21.15.119:0/3422175323 shutdown_connections 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca584290 0x5609ca583970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca57bc00 0x5609ca583430 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/3422175323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca5774d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 -- 172.21.15.119:0/3422175323 >> 172.21.15.119:0/3422175323 conn(0x5609ca542200 msgr2=0x5609ca576dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 -- 172.21.15.119:0/3422175323 shutdown_connections 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.311+0000 7f09ff5f9740 1 -- 172.21.15.119:0/3422175323 wait complete. 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 -- start start 2024-03-28T18:48:30.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 Processor -- start 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca57bc00 0x5609ca57daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca584290 0x5609ca57e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41b0870 con 0x5609ca57bc00 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41b0210 con 0x5609ca57af20 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f41ad710 con 0x5609ca584290 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09ff5f9740 10 client.0 _ll_register_callbacks cb 0x7f09e4009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09fa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09fa800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58968/0 (socket says 172.21.15.119:58968) 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09fa800640 1 -- 172.21.15.119:0/4111074576 learned_addr learned my addr 172.21.15.119:0/4111074576 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.312+0000 7f09f9400640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca57bc00 0x5609ca57daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca584290 0x5609ca57e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 -- 172.21.15.119:0/4111074576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca584290 msgr2=0x5609ca57e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca584290 0x5609ca57e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 -- 172.21.15.119:0/4111074576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca57bc00 msgr2=0x5609ca57daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca57bc00 0x5609ca57daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 -- 172.21.15.119:0/4111074576 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609ca5eed20 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09fa800640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009710 tx=0x7f09f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad8dd33dbe23c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09d3400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0032040 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09e400fe00 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.313+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f09f00035f0 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09f00037f0 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 10 client.26780 fetch_fsmap learned FSMap version 28 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5609ca5f0070 con 0x5609ca57af20 2024-03-28T18:48:30.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09e400fe00 con 0x5609ca57af20 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f09f00289d0 con 0x5609ca57af20 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 10 client.26780 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 10 client.26780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 20 client.26780 populate_metadata read hostname 'smithi119' 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5f0560 0x5609ca5f2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 4 client.26780 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.314+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5609ca4e6ae0 con 0x5609ca5f0560 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09f9400640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5f0560 0x5609ca5f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09f9400640 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5f0560 0x5609ca5f2980 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=809941b218ed62e8 server_cookie=5e298565f74528ab in_seq=0 out_seq=0 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09d3400640 10 client.26780 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09d3400640 1 -- 172.21.15.119:0/4111074576 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5609ca4e6ae0 con 0x5609ca5f0560 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09d3400640 10 client.26780 handle_command_reply: tid=0 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.315+0000 7f09ff5f9740 1 client.26780 shutdown 2024-03-28T18:48:30.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 msgr2=0x5609ca57d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009710 tx=0x7f09f00118d0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 shutdown_connections 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5609ca5f0560 0x5609ca5f2980 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5609ca584290 0x5609ca57e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5609ca57bc00 0x5609ca57daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09d3400640 0 client.26780 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 --2- 172.21.15.119:0/4111074576 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5609ca57af20 0x5609ca57d5b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 >> 172.21.15.119:0/4111074576 conn(0x5609ca542200 msgr2=0x5609ca576d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 shutdown_connections 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 1 -- 172.21.15.119:0/4111074576 wait complete. 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09ff5f9740 20 client.26780 trim_cache size 0 max 16384 2024-03-28T18:48:30.320 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "e8ec3066-50e4-4c20-8066-027513a79026": { 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "e8ec3066-50e4-4c20-8066-027513a79026", 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:48:30.321 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 msgr2=0x7f09e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 0x7f09e00a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5609ca57c150 tx=0x7f09ec0f4c20 comp rx=0 tx=0).stop 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 msgr2=0x7f09f419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.316+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f09ec005e30 tx=0x7f09ec035e00 comp rx=0 tx=0).stop 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 shutdown_connections 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09f410ded0 0x7f09f419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f4074390 0x7f09f419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 --2- 172.21.15.119:0/543288202 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f09f4073a70 0x7f09f419a170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:48:30.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 >> 172.21.15.119:0/543288202 conn(0x7f09f4078aa0 msgr2=0x7f09f40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:48:30.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 shutdown_connections 2024-03-28T18:48:30.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:48:30.317+0000 7f09fc800640 1 -- 172.21.15.119:0/543288202 wait complete. 2024-03-28T18:48:30.335 INFO:tasks.cephfs.filesystem:scrub status for tag:e8ec3066-50e4-4c20-8066-027513a79026 - {'path': '/', 'tag': 'e8ec3066-50e4-4c20-8066-027513a79026', 'options': 'recursive,force'} 2024-03-28T18:48:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:30 smithi119 ceph-mon[31813]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:30 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:48:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:30 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:30 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:48:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:30 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:30.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:30 smithi196 ceph-mon[36300]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:30 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:48:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:30 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:30 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:48:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:30 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:30 smithi131 ceph-mon[36065]: pgmap v136: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:30 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:48:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:30 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:30.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:30 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:48:30.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:30 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:48:32.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:32 smithi196 ceph-mon[36300]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:32 smithi131 ceph-mon[36065]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:32 smithi119 ceph-mon[31813]: pgmap v137: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:33.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:33 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:48:33.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:33 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:48:33.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:33 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:48:34.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:34 smithi196 ceph-mon[36300]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:34 smithi131 ceph-mon[36065]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:34 smithi119 ceph-mon[31813]: pgmap v138: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:36.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:36 smithi196 ceph-mon[36300]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:36 smithi131 ceph-mon[36065]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:36 smithi119 ceph-mon[31813]: pgmap v139: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:38.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:38 smithi196 ceph-mon[36300]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:38 smithi131 ceph-mon[36065]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:38 smithi119 ceph-mon[31813]: pgmap v140: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:40 smithi196 ceph-mon[36300]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:40 smithi131 ceph-mon[36065]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:40 smithi119 ceph-mon[31813]: pgmap v141: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:42.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:42 smithi196 ceph-mon[36300]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:42 smithi131 ceph-mon[36065]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:42 smithi119 ceph-mon[31813]: pgmap v142: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:44.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:44 smithi196 ceph-mon[36300]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:44 smithi131 ceph-mon[36065]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:44 smithi119 ceph-mon[31813]: pgmap v143: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:48:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:46 smithi196 ceph-mon[36300]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:46 smithi131 ceph-mon[36065]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:46 smithi119 ceph-mon[31813]: pgmap v144: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:48:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:48 smithi196 ceph-mon[36300]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:48 smithi131 ceph-mon[36065]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:48 smithi119 ceph-mon[31813]: pgmap v145: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:50.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:50 smithi196 ceph-mon[36300]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:50 smithi131 ceph-mon[36065]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:50 smithi119 ceph-mon[31813]: pgmap v146: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:52.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:52 smithi196 ceph-mon[36300]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:52.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:52 smithi131 ceph-mon[36065]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:52 smithi119 ceph-mon[31813]: pgmap v147: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:54.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:54 smithi196 ceph-mon[36300]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:54.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:54 smithi131 ceph-mon[36065]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:54 smithi119 ceph-mon[31813]: pgmap v148: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:48:56.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:56 smithi196 ceph-mon[36300]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:56.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:56 smithi131 ceph-mon[36065]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:56 smithi119 ceph-mon[31813]: pgmap v149: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:48:58.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:48:58 smithi196 ceph-mon[36300]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:48:58.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:48:58 smithi131 ceph-mon[36065]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:48:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:48:58 smithi119 ceph-mon[31813]: pgmap v150: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:00.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:49:00.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:49:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 -- 172.21.15.119:0/103291074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90103250 msgr2=0x7f6b9010b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 --2- 172.21.15.119:0/103291074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90103250 0x7f6b9010b730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009900 tx=0x7f6b7c02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 -- 172.21.15.119:0/103291074 shutdown_connections 2024-03-28T18:49:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 --2- 172.21.15.119:0/103291074 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 0x7f6b9010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 --2- 172.21.15.119:0/103291074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90103250 0x7f6b9010b730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 --2- 172.21.15.119:0/103291074 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90102930 0x7f6b90102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 -- 172.21.15.119:0/103291074 >> 172.21.15.119:0/103291074 conn(0x7f6b90077700 msgr2=0x7f6b90077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 -- 172.21.15.119:0/103291074 shutdown_connections 2024-03-28T18:49:00.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.561+0000 7f6b95600640 1 -- 172.21.15.119:0/103291074 wait complete. 2024-03-28T18:49:00.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.562+0000 7f6b95600640 1 Processor -- start 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.562+0000 7f6b95600640 1 -- start start 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 0x7f6b901984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 0x7f6b9019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90111a90 con 0x7f6b90103250 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90111910 con 0x7f6b9010bc70 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b95600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90111c10 con 0x7f6b90102930 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37568/0 (socket says 172.21.15.119:37568) 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 -- 172.21.15.119:0/3460774499 learned_addr learned my addr 172.21.15.119:0/3460774499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 0x7f6b9019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8e000640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 0x7f6b901984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 -- 172.21.15.119:0/3460774499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 msgr2=0x7f6b9019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 0x7f6b9019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 -- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 msgr2=0x7f6b901984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 0x7f6b901984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8ea00640 1 -- 172.21.15.119:0/3460774499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9019cf30 con 0x7f6b90102930 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.563+0000 7f6b8e000640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 0x7f6b901984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000e9f0 tx=0x7f6b8000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b87400640 1 -- 172.21.15.119:0/3460774499 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000cd80 con 0x7f6b90102930 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b87400640 1 -- 172.21.15.119:0/3460774499 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b80010910 con 0x7f6b90102930 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9019d220 con 0x7f6b90102930 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b87400640 1 -- 172.21.15.119:0/3460774499 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80018990 con 0x7f6b90102930 2024-03-28T18:49:00.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.564+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b901a4ab0 con 0x7f6b90102930 2024-03-28T18:49:00.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.568+0000 7f6b87400640 1 -- 172.21.15.119:0/3460774499 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6b80010ac0 con 0x7f6b90102930 2024-03-28T18:49:00.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.568+0000 7f6b983b0740 1 Processor -- start 2024-03-28T18:49:00.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.568+0000 7f6b983b0740 1 -- start start 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdef43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdeff680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fde58250 con 0x55f0fdef3f20 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fde00a30 con 0x55f0fdef4c00 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b983b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fde542f0 con 0x55f0fdefd290 2024-03-28T18:49:00.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60736/0 (socket says 172.21.15.119:60736) 2024-03-28T18:49:00.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 -- 172.21.15.119:0/206931669 learned_addr learned my addr 172.21.15.119:0/206931669 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8e000640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdeff680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdef43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b87400640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 0x7f6b740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 -- 172.21.15.119:0/206931669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 msgr2=0x55f0fdeff680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdeff680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 -- 172.21.15.119:0/206931669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 msgr2=0x55f0fdef43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.569+0000 7f6b8f400640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdef43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b8f400640 1 -- 172.21.15.119:0/206931669 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0fddeb690 con 0x55f0fdef4c00 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdef43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:00.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b87400640 1 -- 172.21.15.119:0/3460774499 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b800cf2b0 con 0x7f6b90102930 2024-03-28T18:49:00.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b8f400640 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009f40 tx=0x7f6b880118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e44ed596575360 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 0x7f6b740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b7b400640 1 -- 172.21.15.119:0/206931669 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8801e070 con 0x55f0fdef4c00 2024-03-28T18:49:00.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.570+0000 7f6b7b400640 1 -- 172.21.15.119:0/206931669 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88003d70 con 0x55f0fdef4c00 2024-03-28T18:49:00.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.571+0000 7f6b7b400640 1 -- 172.21.15.119:0/206931669 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8800fca0 con 0x55f0fdef4c00 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 -- 172.21.15.119:0/206931669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 msgr2=0x55f0fdef3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 0x7f6b740a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6b88004020 tx=0x7f6b8800fe40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009f40 tx=0x7f6b880118d0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 -- 172.21.15.119:0/206931669 shutdown_connections 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdeff680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdef43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 --2- 172.21.15.119:0/206931669 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdef3950 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 -- 172.21.15.119:0/206931669 >> 172.21.15.119:0/206931669 conn(0x55f0fdebb200 msgr2=0x55f0fdeee970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 -- 172.21.15.119:0/206931669 shutdown_connections 2024-03-28T18:49:00.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.572+0000 7f6b983b0740 1 -- 172.21.15.119:0/206931669 wait complete. 2024-03-28T18:49:00.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 -- start start 2024-03-28T18:49:00.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 Processor -- start 2024-03-28T18:49:00.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdefd290 0x55f0fdef9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fdf00100 con 0x55f0fdef3f20 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fde13e80 con 0x55f0fdefd290 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.573+0000 7f6b983b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0fdf00280 con 0x55f0fdef4c00 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b983b0740 10 client.0 _ll_register_callbacks cb 0x55f0fde26130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37586/0 (socket says 172.21.15.119:37586) 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46058/0 (socket says 172.21.15.119:46058) 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 -- 172.21.15.119:0/1870560799 learned_addr learned my addr 172.21.15.119:0/1870560799 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8f400640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdefd290 0x55f0fdef9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 -- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdefd290 msgr2=0x55f0fdef9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdefd290 0x55f0fdef9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 -- 172.21.15.119:0/1870560799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 msgr2=0x55f0fdefc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8e000640 1 -- 172.21.15.119:0/1870560799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0fdc367d0 con 0x55f0fdef4c00 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.574+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b8e000640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c005c40 tx=0x7f6b7c005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9070da528059b30d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c043070 con 0x55f0fdef4c00 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b88003d70 con 0x55f0fdef4c00 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b7c005030 con 0x55f0fdef4c00 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b7c04f050 con 0x55f0fdef4c00 2024-03-28T18:49:00.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b983b0740 10 client.26773 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f0fdb6c3f0 con 0x55f0fdef4c00 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.575+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88003d70 con 0x55f0fdef4c00 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6b7c0035f0 con 0x55f0fdef4c00 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 10 client.26773 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 10 client.26773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 20 client.26773 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdf15970 0x55f0fdf17d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 4 client.26773 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.576+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0fde65790 con 0x55f0fdf15970 2024-03-28T18:49:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.577+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdf15970 0x55f0fdf17d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.577+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdf15970 0x55f0fdf17d60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=249bc09348918c57 server_cookie=9b5237b5f848a779 in_seq=0 out_seq=0 2024-03-28T18:49:00.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.577+0000 7f6b6fe00640 10 client.26773 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:00.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b6fe00640 1 -- 172.21.15.119:0/1870560799 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f0fde65790 con 0x55f0fdf15970 2024-03-28T18:49:00.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b6fe00640 10 client.26773 handle_command_reply: tid=0 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 client.26773 shutdown 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 msgr2=0x55f0fdefcbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c005c40 tx=0x7f6b7c005e30 comp rx=0 tx=0).stop 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 shutdown_connections 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdf15970 0x55f0fdf17d60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdefd290 0x55f0fdef9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef4c00 0x55f0fdefcbe0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 --2- 172.21.15.119:0/1870560799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef3f20 0x55f0fdefc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 >> 172.21.15.119:0/1870560799 conn(0x55f0fdebb200 msgr2=0x55f0fdef06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.578+0000 7f6b6fe00640 0 client.26773 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.579+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 shutdown_connections 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.579+0000 7f6b983b0740 1 -- 172.21.15.119:0/1870560799 wait complete. 2024-03-28T18:49:00.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.579+0000 7f6b983b0740 20 client.26773 trim_cache size 0 max 16384 2024-03-28T18:49:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 Processor -- start 2024-03-28T18:49:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 -- start start 2024-03-28T18:49:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 0x55f0fdefe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef4c00 0x55f0fdefebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901ab140 con 0x55f0fdef4c00 2024-03-28T18:49:00.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901acac0 con 0x55f0fdef3f20 2024-03-28T18:49:00.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.600+0000 7f6b983b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901ab7a0 con 0x55f0fdefd290 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 0x55f0fdefe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37594/0 (socket says 172.21.15.119:37594) 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 0x55f0fdefe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60748/0 (socket says 172.21.15.119:60748) 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 -- 172.21.15.119:0/3608120114 learned_addr learned my addr 172.21.15.119:0/3608120114 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8e000640 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef4c00 0x55f0fdefebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 -- 172.21.15.119:0/3608120114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 msgr2=0x55f0fdefe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 0x55f0fdefe680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 -- 172.21.15.119:0/3608120114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef4c00 msgr2=0x55f0fdefebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef4c00 0x55f0fdefebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.601+0000 7f6b8f400640 1 -- 172.21.15.119:0/3608120114 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0fdb2c0f0 con 0x55f0fdefd290 2024-03-28T18:49:00.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.602+0000 7f6b8f400640 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b8801c040 tx=0x7f6b88027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df9c056679e078fc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.602+0000 7f6b7aa00640 1 -- 172.21.15.119:0/3608120114 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88037070 con 0x55f0fdefd290 2024-03-28T18:49:00.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.602+0000 7f6b7aa00640 1 -- 172.21.15.119:0/3608120114 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88003b70 con 0x55f0fdefd290 2024-03-28T18:49:00.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.602+0000 7f6b7aa00640 1 -- 172.21.15.119:0/3608120114 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8801e040 con 0x55f0fdefd290 2024-03-28T18:49:00.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 -- 172.21.15.119:0/3608120114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 msgr2=0x55f0fdefc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b8801c040 tx=0x7f6b88027000 comp rx=0 tx=0).stop 2024-03-28T18:49:00.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 -- 172.21.15.119:0/3608120114 shutdown_connections 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdefd290 0x55f0fdefc3c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdef4c00 0x55f0fdefebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 --2- 172.21.15.119:0/3608120114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef3f20 0x55f0fdefe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 -- 172.21.15.119:0/3608120114 >> 172.21.15.119:0/3608120114 conn(0x55f0fdebb200 msgr2=0x55f0fdeee4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 -- 172.21.15.119:0/3608120114 shutdown_connections 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.603+0000 7f6b983b0740 1 -- 172.21.15.119:0/3608120114 wait complete. 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 -- start start 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 Processor -- start 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdf126f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 0x55f0fdf12c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901ac460 con 0x55f0fdefd290 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901abe00 con 0x55f0fdef4c00 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901a9300 con 0x55f0fdef3f20 2024-03-28T18:49:00.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.604+0000 7f6b983b0740 10 client.0 _ll_register_callbacks cb 0x7f6b880093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:00.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 0x55f0fdf12c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8e000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdf126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37610/0 (socket says 172.21.15.119:37610) 2024-03-28T18:49:00.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 -- 172.21.15.119:0/456356259 learned_addr learned my addr 172.21.15.119:0/456356259 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 -- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 msgr2=0x55f0fdf126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdf126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 -- 172.21.15.119:0/456356259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 msgr2=0x55f0fdf12c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 0x55f0fdf12c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 -- 172.21.15.119:0/456356259 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0fdef5ff0 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8f400640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 0x55f0fdf12c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b8ea00640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c6a0 tx=0x7f6b80018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a78c54f53e2cf6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b6fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.605+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b800ccc20 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6b800ccdc0 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b88003b70 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b80086070 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 10 client.26779 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f0fdef7360 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88003b70 con 0x55f0fdef3f20 2024-03-28T18:49:00.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6b80018c50 con 0x55f0fdef3f20 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 10 client.26779 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 10 client.26779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 20 client.26779 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdef79e0 0x55f0fdf741a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 4 client.26779 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.606+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f0fdefabd0 con 0x55f0fdef79e0 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b8e000640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdef79e0 0x55f0fdf741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b8e000640 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdef79e0 0x55f0fdf741a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ca1ef32a4369cc4 server_cookie=8118e94e9098ec28 in_seq=0 out_seq=0 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b6fe00640 10 client.26779 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b6fe00640 1 -- 172.21.15.119:0/456356259 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f0fdefabd0 con 0x55f0fdef79e0 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b6fe00640 10 client.26779 handle_command_reply: tid=0 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.607+0000 7f6b983b0740 1 client.26779 shutdown 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 msgr2=0x55f0fdf121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c6a0 tx=0x7f6b80018420 comp rx=0 tx=0).stop 2024-03-28T18:49:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 shutdown_connections 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f0fdef79e0 0x55f0fdf741a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0fdefd290 0x55f0fdf12c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f0fdef4c00 0x55f0fdf126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 --2- 172.21.15.119:0/456356259 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0fdef3f20 0x55f0fdf121b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b6fe00640 0 client.26779 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 >> 172.21.15.119:0/456356259 conn(0x55f0fdebb200 msgr2=0x55f0fdef8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 shutdown_connections 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 1 -- 172.21.15.119:0/456356259 wait complete. 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.608+0000 7f6b983b0740 20 client.26779 trim_cache size 0 max 16384 2024-03-28T18:49:00.611 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:00.612 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:49:00.612 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:49:00.612 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:00.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 msgr2=0x7f6b740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 0x7f6b740a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6b88004020 tx=0x7f6b8800fe40 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 msgr2=0x7f6b90197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000e9f0 tx=0x7f6b8000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 shutdown_connections 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b740a2970 0x7f6b740a4e30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9010bc70 0x7f6b9019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b90103250 0x7f6b901984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 --2- 172.21.15.119:0/3460774499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b90102930 0x7f6b90197f90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 >> 172.21.15.119:0/3460774499 conn(0x7f6b90077700 msgr2=0x7f6b900fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 shutdown_connections 2024-03-28T18:49:00.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.609+0000 7f6b95600640 1 -- 172.21.15.119:0/3460774499 wait complete. 2024-03-28T18:49:00.628 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:49:00.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:49:00.690 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:00 smithi119 ceph-mon[31813]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:00.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:00 smithi131 ceph-mon[36065]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:00.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 -- 172.21.15.119:0/458348541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e080764f0 msgr2=0x7f6e08111fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 --2- 172.21.15.119:0/458348541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e080764f0 0x7f6e08111fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009c10 tx=0x7f6df002bef0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 -- 172.21.15.119:0/458348541 shutdown_connections 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 --2- 172.21.15.119:0/458348541 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e081124f0 0x7f6e081148e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 --2- 172.21.15.119:0/458348541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e080764f0 0x7f6e08111fb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 --2- 172.21.15.119:0/458348541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e08075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 -- 172.21.15.119:0/458348541 >> 172.21.15.119:0/458348541 conn(0x7f6e080ff2e0 msgr2=0x7f6e08101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 -- 172.21.15.119:0/458348541 shutdown_connections 2024-03-28T18:49:00.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.819+0000 7f6e0de00640 1 -- 172.21.15.119:0/458348541 wait complete. 2024-03-28T18:49:00.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.820+0000 7f6e0de00640 1 Processor -- start 2024-03-28T18:49:00.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.820+0000 7f6e0de00640 1 -- start start 2024-03-28T18:49:00.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e0819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 0x7f6e0819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 0x7f6e081a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081180a0 con 0x7f6e08075bd0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08117f20 con 0x7f6e080764f0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e0de00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08118220 con 0x7f6e081124f0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e0819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e07400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 0x7f6e081a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 0x7f6e0819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e0819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46092/0 (socket says 172.21.15.119:46092) 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06a00640 1 -- 172.21.15.119:0/1077450659 learned_addr learned my addr 172.21.15.119:0/1077450659 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 -- 172.21.15.119:0/1077450659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 msgr2=0x7f6e081a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 0x7f6e081a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 -- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 msgr2=0x7f6e0819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e0819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e06000640 1 -- 172.21.15.119:0/1077450659 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e081a3610 con 0x7f6e080764f0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.821+0000 7f6e07400640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 0x7f6e081a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e06000640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 0x7f6e0819eb60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6df002c430 tx=0x7f6df0033730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0043020 con 0x7f6e080764f0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6df002ca60 con 0x7f6e080764f0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df003e6b0 con 0x7f6e080764f0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e081a38a0 con 0x7f6e080764f0 2024-03-28T18:49:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.822+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e081ab190 con 0x7f6e080764f0 2024-03-28T18:49:00.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.823+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e08077310 con 0x7f6e080764f0 2024-03-28T18:49:00.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.825+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6df0046020 con 0x7f6e080764f0 2024-03-28T18:49:00.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.826+0000 7f6e0ca00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 0x7f6dec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:00.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.827+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6df00f1a40 con 0x7f6e080764f0 2024-03-28T18:49:00.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.827+0000 7f6e06a00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 0x7f6dec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:00.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.827+0000 7f6e06a00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 0x7f6dec0a4eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009c90 tx=0x7f6df8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:00.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:00.829+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6df004d020 con 0x7f6e080764f0 2024-03-28T18:49:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:00 smithi196 ceph-mon[36300]: pgmap v151: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:01.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.065+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e08068490 con 0x7f6e080764f0 2024-03-28T18:49:01.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.066+0000 7f6e0ca00640 1 -- 172.21.15.119:0/1077450659 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f6df00be7f0 con 0x7f6e080764f0 2024-03-28T18:49:01.066 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:49:01.067 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:49:01.067 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 msgr2=0x7f6dec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 0x7f6dec0a4eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009c90 tx=0x7f6df8005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 msgr2=0x7f6e0819eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 0x7f6e0819eb60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6df002c430 tx=0x7f6df0033730 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 shutdown_connections 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dec0a29f0 0x7f6dec0a4eb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e081124f0 0x7f6e081a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e080764f0 0x7f6e0819eb60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 --2- 172.21.15.119:0/1077450659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e08075bd0 0x7f6e0819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 >> 172.21.15.119:0/1077450659 conn(0x7f6e080ff2e0 msgr2=0x7f6e08113c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 shutdown_connections 2024-03-28T18:49:01.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.069+0000 7f6e0de00640 1 -- 172.21.15.119:0/1077450659 wait complete. 2024-03-28T18:49:01.088 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:01.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 -- 172.21.15.119:0/1523802037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c1124e0 msgr2=0x7ffb8c1148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 --2- 172.21.15.119:0/1523802037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1148d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009cd0 tx=0x7ffb8402c0e0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 -- 172.21.15.119:0/1523802037 shutdown_connections 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 --2- 172.21.15.119:0/1523802037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1148d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 --2- 172.21.15.119:0/1523802037 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 --2- 172.21.15.119:0/1523802037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 0x7ffb8c075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.305+0000 7ffb91800640 1 -- 172.21.15.119:0/1523802037 >> 172.21.15.119:0/1523802037 conn(0x7ffb8c0ff2f0 msgr2=0x7ffb8c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.306+0000 7ffb91800640 1 -- 172.21.15.119:0/1523802037 shutdown_connections 2024-03-28T18:49:01.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.306+0000 7ffb91800640 1 -- 172.21.15.119:0/1523802037 wait complete. 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.306+0000 7ffb91800640 1 Processor -- start 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 -- start start 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 0x7ffb8c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c118040 con 0x7ffb8c075bd0 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c117ec0 con 0x7ffb8c1124e0 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb91800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1181c0 con 0x7ffb8c0764f0 2024-03-28T18:49:01.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37650/0 (socket says 172.21.15.119:37650) 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb8a000640 1 -- 172.21.15.119:0/705201515 learned_addr learned my addr 172.21.15.119:0/705201515 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 0x7ffb8c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.307+0000 7ffb8b400640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 -- 172.21.15.119:0/705201515 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 msgr2=0x7ffb8c1a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 -- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 msgr2=0x7ffb8c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 0x7ffb8c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 -- 172.21.15.119:0/705201515 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c1a3570 con 0x7ffb8c0764f0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8b400640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb8a000640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800ea60 tx=0x7ffb7800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb83400640 1 -- 172.21.15.119:0/705201515 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7800cde0 con 0x7ffb8c0764f0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb83400640 1 -- 172.21.15.119:0/705201515 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb780035f0 con 0x7ffb8c0764f0 2024-03-28T18:49:01.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c1a3860 con 0x7ffb8c0764f0 2024-03-28T18:49:01.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb83400640 1 -- 172.21.15.119:0/705201515 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78010670 con 0x7ffb8c0764f0 2024-03-28T18:49:01.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.308+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c1ab0f0 con 0x7ffb8c0764f0 2024-03-28T18:49:01.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.311+0000 7ffb94414740 1 Processor -- start 2024-03-28T18:49:01.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 -- start start 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108852ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb83400640 1 -- 172.21.15.119:0/705201515 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ffb780037a0 con 0x7ffb8c0764f0 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x56310885e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631087b62c0 con 0x56310885bc00 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310875ec90 con 0x5631088526d0 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb94414740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631087b2360 con 0x563108853080 2024-03-28T18:49:01.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb8b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108852ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37654/0 (socket says 172.21.15.119:37654) 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb8aa00640 1 -- 172.21.15.119:0/230686383 learned_addr learned my addr 172.21.15.119:0/230686383 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.312+0000 7ffb8a000640 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x56310885e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 -- 172.21.15.119:0/230686383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 msgr2=0x56310885b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 -- 172.21.15.119:0/230686383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 msgr2=0x56310885e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x56310885e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 -- 172.21.15.119:0/230686383 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563108783080 con 0x5631088526d0 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb8b400640 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108852ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb8402c6f0 tx=0x7ffb84005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73220648b10ea8cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb77e00640 1 -- 172.21.15.119:0/230686383 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84043070 con 0x5631088526d0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb83400640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 0x7ffb700a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb77e00640 1 -- 172.21.15.119:0/230686383 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84003970 con 0x5631088526d0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.313+0000 7ffb77e00640 1 -- 172.21.15.119:0/230686383 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84039040 con 0x5631088526d0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.314+0000 7ffb83400640 1 -- 172.21.15.119:0/705201515 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb780cf140 con 0x7ffb8c0764f0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.314+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 0x7ffb700a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.314+0000 7ffb94414740 1 -- 172.21.15.119:0/230686383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 msgr2=0x563108852ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.314+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 0x7ffb700a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x56310884de10 tx=0x7ffb7c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.314+0000 7ffb94414740 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108852ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb8402c6f0 tx=0x7ffb84005f80 comp rx=0 tx=0).stop 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 -- 172.21.15.119:0/230686383 shutdown_connections 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x56310885e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 --2- 172.21.15.119:0/230686383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108852ab0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 -- 172.21.15.119:0/230686383 >> 172.21.15.119:0/230686383 conn(0x56310884cda0 msgr2=0x56310884d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 -- 172.21.15.119:0/230686383 shutdown_connections 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 -- 172.21.15.119:0/230686383 wait complete. 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 -- start start 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.315+0000 7ffb94414740 1 Processor -- start 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5631088526d0 0x56310885b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 0x563108859070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310885eb10 con 0x5631088526d0 2024-03-28T18:49:01.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631087720d0 con 0x563108853080 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310885ec90 con 0x56310885bc00 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb94414740 10 client.0 _ll_register_callbacks cb 0x563108776e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5631088526d0 0x56310885b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60814/0 (socket says 172.21.15.119:60814) 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 -- 172.21.15.119:0/2617975644 learned_addr learned my addr 172.21.15.119:0/2617975644 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8b400640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 0x563108859070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 -- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 msgr2=0x563108859070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 0x563108859070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 -- 172.21.15.119:0/2617975644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5631088526d0 msgr2=0x56310885b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5631088526d0 0x56310885b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8a000640 1 -- 172.21.15.119:0/2617975644 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631084ca3f0 con 0x563108853080 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.316+0000 7ffb8b400640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 0x563108859070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb8a000640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffb780980b0 tx=0x7ffb78009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed7d7cfbdd3e15b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb74c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:01.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7809b220 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb78003b30 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb84003970 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7809b040 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb94414740 10 client.26816 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5631085947d0 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.317+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84003970 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb78097d50 con 0x563108853080 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 10 client.26816 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 10 client.26816 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 20 client.26816 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56310886ba90 0x56310886deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 4 client.26816 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631087c3800 con 0x56310886ba90 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.318+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56310886ba90 0x56310886deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.319+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56310886ba90 0x56310886deb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a64cc82b5cc90c8 server_cookie=d92a65bbf8540b97 in_seq=0 out_seq=0 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.319+0000 7ffb74c00640 10 client.26816 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:01.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.319+0000 7ffb74c00640 1 -- 172.21.15.119:0/2617975644 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5631087c3800 con 0x56310886ba90 2024-03-28T18:49:01.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.319+0000 7ffb74c00640 10 client.26816 handle_command_reply: tid=0 2024-03-28T18:49:01.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.319+0000 7ffb94414740 1 client.26816 shutdown 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 msgr2=0x563108858930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffb780980b0 tx=0x7ffb78009490 comp rx=0 tx=0).stop 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 shutdown_connections 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56310886ba90 0x56310886deb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56310885bc00 0x563108859070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563108853080 0x563108858930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 --2- 172.21.15.119:0/2617975644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5631088526d0 0x56310885b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb74c00640 0 client.26816 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 >> 172.21.15.119:0/2617975644 conn(0x56310884cda0 msgr2=0x56310884eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 shutdown_connections 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 1 -- 172.21.15.119:0/2617975644 wait complete. 2024-03-28T18:49:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.320+0000 7ffb94414740 20 client.26816 trim_cache size 0 max 16384 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 Processor -- start 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 -- start start 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x56310885aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108855b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b39b0 con 0x56310885bc00 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b5330 con 0x5631088526d0 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.334+0000 7ffb94414740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b4010 con 0x563108853080 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37668/0 (socket says 172.21.15.119:37668) 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 -- 172.21.15.119:0/869505607 learned_addr learned my addr 172.21.15.119:0/869505607 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x56310885aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8b400640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108855b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 -- 172.21.15.119:0/869505607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 msgr2=0x56310885aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x56310885aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 -- 172.21.15.119:0/869505607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 msgr2=0x563108855b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108855b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 -- 172.21.15.119:0/869505607 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56310884c8d0 con 0x563108853080 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8b400640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108855b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb8a000640 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009710 tx=0x7ffb780cf410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f887bd2acd0d807f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb77400640 1 -- 172.21.15.119:0/869505607 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7809b220 con 0x563108853080 2024-03-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb77400640 1 -- 172.21.15.119:0/869505607 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb780ccc40 con 0x563108853080 2024-03-28T18:49:01.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.335+0000 7ffb77400640 1 -- 172.21.15.119:0/869505607 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb780cca50 con 0x563108853080 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 -- 172.21.15.119:0/869505607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 msgr2=0x56310885b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009710 tx=0x7ffb780cf410 comp rx=0 tx=0).stop 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 -- 172.21.15.119:0/869505607 shutdown_connections 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108855b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x56310885b530 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 --2- 172.21.15.119:0/869505607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x56310885aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 -- 172.21.15.119:0/869505607 >> 172.21.15.119:0/869505607 conn(0x56310884cda0 msgr2=0x56310884beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 -- 172.21.15.119:0/869505607 shutdown_connections 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.336+0000 7ffb94414740 1 -- 172.21.15.119:0/869505607 wait complete. 2024-03-28T18:49:01.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 -- start start 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 Processor -- start 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108863710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108861b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b4cd0 con 0x56310885bc00 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b4670 con 0x5631088526d0 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.337+0000 7ffb94414740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b1b70 con 0x563108853080 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb94414740 10 client.0 _ll_register_callbacks cb 0x7ffb780d28e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108863710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37674/0 (socket says 172.21.15.119:37674) 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108861b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 -- 172.21.15.119:0/3804417011 learned_addr learned my addr 172.21.15.119:0/3804417011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 -- 172.21.15.119:0/3804417011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 msgr2=0x563108863710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108863710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 -- 172.21.15.119:0/3804417011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 msgr2=0x563108861b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108861b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 -- 172.21.15.119:0/3804417011 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563108862370 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108863710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb8a000640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffb7809b670 tx=0x7ffb7809ade0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b51b69e963f131 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.338+0000 7ffb74c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb780cca50 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7ffb78004210 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631087bdb50 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7809b220 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 10 client.26803 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5631088c7ed0 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5631087bdb50 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb780afac0 con 0x563108853080 2024-03-28T18:49:01.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 10 client.26803 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 10 client.26803 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 20 client.26803 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5631088c8300 0x5631088ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 4 client.26803 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.339+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56310873b500 con 0x5631088c8300 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5631088c8300 0x5631088ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb8aa00640 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5631088c8300 0x5631088ca6f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=257ac67ba90f3bf1 server_cookie=125c58c91020d996 in_seq=0 out_seq=0 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb74c00640 10 client.26803 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb74c00640 1 -- 172.21.15.119:0/3804417011 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56310873b500 con 0x5631088c8300 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb74c00640 10 client.26803 handle_command_reply: tid=0 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb94414740 1 client.26803 shutdown 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 msgr2=0x563108863c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.340+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ffb7809b670 tx=0x7ffb7809ade0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 shutdown_connections 2024-03-28T18:49:01.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5631088c8300 0x5631088ca6f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56310885bc00 0x563108861b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb74c00640 0 client.26803 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563108853080 0x563108863c50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 --2- 172.21.15.119:0/3804417011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5631088526d0 0x563108863710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 >> 172.21.15.119:0/3804417011 conn(0x56310884cda0 msgr2=0x5631088534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 shutdown_connections 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 1 -- 172.21.15.119:0/3804417011 wait complete. 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb94414740 20 client.26803 trim_cache size 0 max 16384 2024-03-28T18:49:01.345 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 msgr2=0x7ffb700a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 0x7ffb700a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x56310884de10 tx=0x7ffb7c005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 msgr2=0x7ffb8c19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.341+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800ea60 tx=0x7ffb7800ef20 comp rx=0 tx=0).stop 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 shutdown_connections 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ffb700a29f0 0x7ffb700a4eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffb8c1124e0 0x7ffb8c1a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb8c0764f0 0x7ffb8c19eac0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 --2- 172.21.15.119:0/705201515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb8c075bd0 0x7ffb8c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 >> 172.21.15.119:0/705201515 conn(0x7ffb8c0ff2f0 msgr2=0x7ffb8c113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 shutdown_connections 2024-03-28T18:49:01.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.342+0000 7ffb91800640 1 -- 172.21.15.119:0/705201515 wait complete. 2024-03-28T18:49:01.359 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:01.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 -- 172.21.15.119:0/2953844434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101610 msgr2=0x7fdefc101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 --2- 172.21.15.119:0/2953844434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101610 0x7fdefc101a90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009930 tx=0x7fdee802bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 -- 172.21.15.119:0/2953844434 shutdown_connections 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 --2- 172.21.15.119:0/2953844434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc101fd0 0x7fdefc10e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 --2- 172.21.15.119:0/2953844434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101610 0x7fdefc101a90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 --2- 172.21.15.119:0/2953844434 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc1076b0 0x7fdefc107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 -- 172.21.15.119:0/2953844434 >> 172.21.15.119:0/2953844434 conn(0x7fdefc0fd100 msgr2=0x7fdefc0ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 -- 172.21.15.119:0/2953844434 shutdown_connections 2024-03-28T18:49:01.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.575+0000 7fdf02a00640 1 -- 172.21.15.119:0/2953844434 wait complete. 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 Processor -- start 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 -- start start 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc101610 0x7fdefc0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc1076b0 0x7fdefc074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc111a40 con 0x7fdefc1076b0 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1118c0 con 0x7fdefc101610 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf02a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc111bc0 con 0x7fdefc101fd0 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.576+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc1076b0 0x7fdefc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37696/0 (socket says 172.21.15.119:37696) 2024-03-28T18:49:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 -- 172.21.15.119:0/3721468112 learned_addr learned my addr 172.21.15.119:0/3721468112 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefbe00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc101610 0x7fdefc0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 -- 172.21.15.119:0/3721468112 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc101610 msgr2=0x7fdefc0739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc101610 0x7fdefc0739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 -- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc1076b0 msgr2=0x7fdefc074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc1076b0 0x7fdefc074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 -- 172.21.15.119:0/3721468112 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc1a6400 con 0x7fdefc101fd0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.577+0000 7fdefb400640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009900 tx=0x7fdee8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.578+0000 7fdef8c00640 1 -- 172.21.15.119:0/3721468112 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8043020 con 0x7fdefc101fd0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.578+0000 7fdef8c00640 1 -- 172.21.15.119:0/3721468112 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdee80054e0 con 0x7fdefc101fd0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.578+0000 7fdef8c00640 1 -- 172.21.15.119:0/3721468112 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8003b80 con 0x7fdefc101fd0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.578+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc1a6690 con 0x7fdefc101fd0 2024-03-28T18:49:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.578+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc1a6b90 con 0x7fdefc101fd0 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.581+0000 7fdf057ec740 1 Processor -- start 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.581+0000 7fdf057ec740 1 -- start start 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.581+0000 7fdef8c00640 1 -- 172.21.15.119:0/3721468112 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fdee8046020 con 0x7fdefc101fd0 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 0x558084b1a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b1d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084a752c0 con 0x558084b12080 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084a1dc90 con 0x558084b116d0 2024-03-28T18:49:01.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf057ec740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084a71360 con 0x558084b1ac00 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 0x558084b1a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdefb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b1d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60874/0 (socket says 172.21.15.119:60874) 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 -- 172.21.15.119:0/1175971142 learned_addr learned my addr 172.21.15.119:0/1175971142 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 -- 172.21.15.119:0/1175971142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 msgr2=0x558084b1d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b1d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 -- 172.21.15.119:0/1175971142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 msgr2=0x558084b1a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.582+0000 7fdf00a00640 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 0x558084b1a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdf00a00640 1 -- 172.21.15.119:0/1175971142 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558084a42080 con 0x558084b116d0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdefbe00640 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 0x558084b1a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdf00a00640 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdef40039f0 tx=0x7fdef4003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59714b74545dcf85 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdef1200640 1 -- 172.21.15.119:0/1175971142 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef401a470 con 0x558084b116d0 2024-03-28T18:49:01.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdef1200640 1 -- 172.21.15.119:0/1175971142 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdef4015070 con 0x558084b116d0 2024-03-28T18:49:01.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdef1200640 1 -- 172.21.15.119:0/1175971142 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4010900 con 0x558084b116d0 2024-03-28T18:49:01.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.583+0000 7fdef8c00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 0x7fdee00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdef8c00640 1 -- 172.21.15.119:0/3721468112 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdee80f0e10 con 0x7fdefc101fd0 2024-03-28T18:49:01.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- 172.21.15.119:0/1175971142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 msgr2=0x558084b11ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fdef40039f0 tx=0x7fdef4003eb0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdefbe00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 0x7fdee00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- 172.21.15.119:0/1175971142 shutdown_connections 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b1d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b12080 0x558084b1a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 --2- 172.21.15.119:0/1175971142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b11ab0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- 172.21.15.119:0/1175971142 >> 172.21.15.119:0/1175971142 conn(0x558084b0bda0 msgr2=0x558084b0c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- 172.21.15.119:0/1175971142 shutdown_connections 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- 172.21.15.119:0/1175971142 wait complete. 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.584+0000 7fdf057ec740 1 -- start start 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdefbe00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 0x7fdee00a4eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x558084b0ce10 tx=0x7fdeec005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 Processor -- start 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 0x558084b1a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b17e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084b1db70 con 0x558084b116d0 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084a310d0 con 0x558084b12080 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558084b1dcf0 con 0x558084b1ac00 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf057ec740 10 client.0 _ll_register_callbacks cb 0x558084a35e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b17e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdefb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdefb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60884/0 (socket says 172.21.15.119:60884) 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b17e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37716/0 (socket says 172.21.15.119:37716) 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdefb400640 1 -- 172.21.15.119:0/2285989180 learned_addr learned my addr 172.21.15.119:0/2285989180 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.585+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 0x558084b1a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 -- 172.21.15.119:0/2285989180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 msgr2=0x558084b17e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b17e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 -- 172.21.15.119:0/2285989180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 msgr2=0x558084b1a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 0x558084b1a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 -- 172.21.15.119:0/2285989180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580847893f0 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 0x558084b1a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdefb400640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdee80bd450 tx=0x7fdee80b9160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7338d269e21acf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdee5600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee80bd000 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdee80ee720 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdef4015070 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.586+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee80bce20 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 10 client.26834 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5580848537d0 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdef4015070 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdee80c57d0 con 0x558084b12080 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 10 client.26834 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 10 client.26834 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 20 client.26834 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b2a960 0x558084b2cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 4 client.26834 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558084a82800 con 0x558084b2a960 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.587+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b2a960 0x558084b2cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.588+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b2a960 0x558084b2cd70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3fa356e37aae4cd6 server_cookie=152894c3e1dba403 in_seq=0 out_seq=0 2024-03-28T18:49:01.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.588+0000 7fdee5600640 10 client.26834 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdee5600640 1 -- 172.21.15.119:0/2285989180 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558084a82800 con 0x558084b2a960 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdee5600640 10 client.26834 handle_command_reply: tid=0 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 client.26834 shutdown 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 msgr2=0x558084b17890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdee80bd450 tx=0x7fdee80b9160 comp rx=0 tx=0).stop 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 shutdown_connections 2024-03-28T18:49:01.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b1ac00 0x558084b17e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b17890 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b2a960 0x558084b2cd70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2285989180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b116d0 0x558084b1a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 >> 172.21.15.119:0/2285989180 conn(0x558084b0bda0 msgr2=0x558084b0daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdee5600640 0 client.26834 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 shutdown_connections 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.589+0000 7fdf057ec740 1 -- 172.21.15.119:0/2285989180 wait complete. 2024-03-28T18:49:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.590+0000 7fdf057ec740 20 client.26834 trim_cache size 0 max 16384 2024-03-28T18:49:01.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 Processor -- start 2024-03-28T18:49:01.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 -- start start 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b19f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 0x558084b1a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b14af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1af5f0 con 0x558084b1ac00 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1b0f70 con 0x558084b116d0 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf057ec740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1afc50 con 0x558084b12080 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b19f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdefb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 0x558084b1a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdf00a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b14af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b19f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60888/0 (socket says 172.21.15.119:60888) 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.603+0000 7fdefbe00640 1 -- 172.21.15.119:0/2860891863 learned_addr learned my addr 172.21.15.119:0/2860891863 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 -- 172.21.15.119:0/2860891863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 msgr2=0x558084b19f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b19f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 -- 172.21.15.119:0/2860891863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 msgr2=0x558084b14af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b14af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 -- 172.21.15.119:0/2860891863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558084b15de0 con 0x558084b12080 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdefb400640 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 0x558084b1a490 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdee80bd1c0 tx=0x7fdee8004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce0d7228f1515eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdee7e00640 1 -- 172.21.15.119:0/2860891863 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee8004c80 con 0x558084b12080 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdee7e00640 1 -- 172.21.15.119:0/2860891863 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdee80d5070 con 0x558084b12080 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.604+0000 7fdee7e00640 1 -- 172.21.15.119:0/2860891863 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee80c57d0 con 0x558084b12080 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- 172.21.15.119:0/2860891863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 msgr2=0x558084b1a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 0x558084b1a490 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdee80bd1c0 tx=0x7fdee8004610 comp rx=0 tx=0).stop 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- 172.21.15.119:0/2860891863 shutdown_connections 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b14af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b12080 0x558084b1a490 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2860891863 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b116d0 0x558084b19f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- 172.21.15.119:0/2860891863 >> 172.21.15.119:0/2860891863 conn(0x558084b0bda0 msgr2=0x558084b0da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- 172.21.15.119:0/2860891863 shutdown_connections 2024-03-28T18:49:01.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- 172.21.15.119:0/2860891863 wait complete. 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 -- start start 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.605+0000 7fdf057ec740 1 Processor -- start 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b21d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1b0910 con 0x558084b1ac00 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1b02b0 con 0x558084b12080 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc1ad7b0 con 0x558084b116d0 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf057ec740 10 client.0 _ll_register_callbacks cb 0x7fdee802ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37748/0 (socket says 172.21.15.119:37748) 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 -- 172.21.15.119:0/2999323645 learned_addr learned my addr 172.21.15.119:0/2999323645 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefb400640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdf00a00640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b21d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 -- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 msgr2=0x558084b217d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b217d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 -- 172.21.15.119:0/2999323645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 msgr2=0x558084b21d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.606+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b21d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdefbe00640 1 -- 172.21.15.119:0/2999323645 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558084b86220 con 0x558084b116d0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdefb400640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b217d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdefbe00640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009fb0 tx=0x7fdeec017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3d744e664d8c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdee5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec030070 con 0x558084b116d0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdee80d5070 con 0x558084b116d0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fdeec01dda0 con 0x558084b116d0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdeec03b050 con 0x558084b116d0 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdf057ec740 10 client.26821 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.607+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558084b87480 con 0x558084b116d0 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdee80d5070 con 0x558084b116d0 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdeec02a690 con 0x558084b116d0 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 10 client.26821 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 10 client.26821 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 20 client.26821 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b87b90 0x558084b89fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 4 client.26821 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558084a7cb50 con 0x558084b87b90 2024-03-28T18:49:01.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.608+0000 7fdefb400640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b87b90 0x558084b89fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdefb400640 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b87b90 0x558084b89fb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=37e4982321eee165 server_cookie=bce6cba8b4f2b2c in_seq=0 out_seq=0 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdee5600640 10 client.26821 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdee5600640 1 -- 172.21.15.119:0/2999323645 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558084a7cb50 con 0x558084b87b90 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdee5600640 10 client.26821 handle_command_reply: tid=0 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 client.26821 shutdown 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 msgr2=0x558084b21290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdeec009fb0 tx=0x7fdeec017000 comp rx=0 tx=0).stop 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 shutdown_connections 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x558084b87b90 0x558084b89fb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558084b1ac00 0x558084b21d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558084b12080 0x558084b217d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdee5600640 0 client.26821 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 --2- 172.21.15.119:0/2999323645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558084b116d0 0x558084b21290 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 >> 172.21.15.119:0/2999323645 conn(0x558084b0bda0 msgr2=0x558084b124b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 shutdown_connections 2024-03-28T18:49:01.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 1 -- 172.21.15.119:0/2999323645 wait complete. 2024-03-28T18:49:01.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.609+0000 7fdf057ec740 20 client.26821 trim_cache size 0 max 16384 2024-03-28T18:49:01.614 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 msgr2=0x7fdee00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 0x7fdee00a4eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x558084b0ce10 tx=0x7fdeec005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 msgr2=0x7fdefc073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009900 tx=0x7fdee8004360 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 shutdown_connections 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdee00a29f0 0x7fdee00a4eb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdefc1076b0 0x7fdefc074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdefc101fd0 0x7fdefc073ee0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 --2- 172.21.15.119:0/3721468112 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdefc101610 0x7fdefc0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 >> 172.21.15.119:0/3721468112 conn(0x7fdefc0fd100 msgr2=0x7fdefc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 shutdown_connections 2024-03-28T18:49:01.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.610+0000 7fdf02a00640 1 -- 172.21.15.119:0/3721468112 wait complete. 2024-03-28T18:49:01.625 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:01 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1077450659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:01.629 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:01.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:49:01.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:01 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1077450659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 -- 172.21.15.119:0/2129803524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 msgr2=0x7f6e54101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 --2- 172.21.15.119:0/2129803524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e54101ab0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009cd0 tx=0x7f6e3c02c0e0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 -- 172.21.15.119:0/2129803524 shutdown_connections 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 --2- 172.21.15.119:0/2129803524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54101ff0 0x7f6e5410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 --2- 172.21.15.119:0/2129803524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e54101ab0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 --2- 172.21.15.119:0/2129803524 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54107640 0x7f6e54107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 -- 172.21.15.119:0/2129803524 >> 172.21.15.119:0/2129803524 conn(0x7f6e54078aa0 msgr2=0x7f6e54078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 -- 172.21.15.119:0/2129803524 shutdown_connections 2024-03-28T18:49:01.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.845+0000 7f6e59e00640 1 -- 172.21.15.119:0/2129803524 wait complete. 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 Processor -- start 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 -- start start 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 0x7f6e5419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 0x7f6e5419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54111a50 con 0x7f6e54101630 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541118d0 con 0x7f6e54101ff0 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.847+0000 7f6e59e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54111bd0 con 0x7f6e54107640 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 0x7f6e5419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 0x7f6e5419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46204/0 (socket says 172.21.15.119:46204) 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52a00640 1 -- 172.21.15.119:0/3071986632 learned_addr learned my addr 172.21.15.119:0/3071986632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 -- 172.21.15.119:0/3071986632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 msgr2=0x7f6e5419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 0x7f6e5419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 -- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 msgr2=0x7f6e5419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 -- 172.21.15.119:0/3071986632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5419f140 con 0x7f6e54107640 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52a00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e52000640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 0x7f6e5419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.848+0000 7f6e53400640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 0x7f6e5419ea10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00e970 tx=0x7f6e4c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.849+0000 7f6e58a00640 1 -- 172.21.15.119:0/3071986632 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c00ccc0 con 0x7f6e54107640 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.849+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5419f430 con 0x7f6e54107640 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.849+0000 7f6e58a00640 1 -- 172.21.15.119:0/3071986632 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c005020 con 0x7f6e54107640 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.849+0000 7f6e58a00640 1 -- 172.21.15.119:0/3071986632 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c004a00 con 0x7f6e54107640 2024-03-28T18:49:01.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.849+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e541a6cc0 con 0x7f6e54107640 2024-03-28T18:49:01.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.852+0000 7f6e5cbea740 1 Processor -- start 2024-03-28T18:49:01.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.852+0000 7f6e58a00640 1 -- 172.21.15.119:0/3071986632 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6e4c0051d0 con 0x7f6e54107640 2024-03-28T18:49:01.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.852+0000 7f6e5cbea740 1 -- start start 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed04ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 0x55d96ed10050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ec682c0 con 0x55d96ed046d0 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ec10c90 con 0x55d96ed05080 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ec64360 con 0x55d96ed0dc00 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60942/0 (socket says 172.21.15.119:60942) 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 -- 172.21.15.119:0/2586476637 learned_addr learned my addr 172.21.15.119:0/2586476637 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52000640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed04ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52000640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 0x55d96ed10050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 -- 172.21.15.119:0/2586476637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 msgr2=0x55d96ed10050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 0x55d96ed10050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 -- 172.21.15.119:0/2586476637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 msgr2=0x55d96ed04ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed04ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52a00640 1 -- 172.21.15.119:0/2586476637 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d96ec35080 con 0x55d96ed05080 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.853+0000 7f6e52000640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 0x55d96ed10050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e52a00640 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6e440115c0 tx=0x7f6e44011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d55192fc67ea0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e58a00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 0x7f6e340a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e43e00640 1 -- 172.21.15.119:0/2586476637 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400f660 con 0x55d96ed05080 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e43e00640 1 -- 172.21.15.119:0/2586476637 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4400fce0 con 0x55d96ed05080 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e43e00640 1 -- 172.21.15.119:0/2586476637 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44019db0 con 0x55d96ed05080 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e58a00640 1 -- 172.21.15.119:0/3071986632 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6e4c0cedf0 con 0x7f6e54107640 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.854+0000 7f6e52000640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 0x7f6e340a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- 172.21.15.119:0/2586476637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 msgr2=0x55d96ed0d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6e440115c0 tx=0x7f6e44011a80 comp rx=0 tx=0).stop 2024-03-28T18:49:01.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- 172.21.15.119:0/2586476637 shutdown_connections 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed0dc00 0x55d96ed10050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed05080 0x55d96ed0d630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/2586476637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed04ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- 172.21.15.119:0/2586476637 >> 172.21.15.119:0/2586476637 conn(0x55d96ecfeda0 msgr2=0x55d96ecff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- 172.21.15.119:0/2586476637 shutdown_connections 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e52000640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 0x7f6e340a4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009ca0 tx=0x7f6e3c0324f0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- 172.21.15.119:0/2586476637 wait complete. 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 -- start start 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.855+0000 7f6e5cbea740 1 Processor -- start 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed05080 0x55d96ed0a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed0af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ed10b70 con 0x55d96ed05080 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ec240d0 con 0x55d96ed0dc00 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d96ed10cf0 con 0x55d96ed046d0 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e5cbea740 10 client.0 _ll_register_callbacks cb 0x55d96ec28e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37792/0 (socket says 172.21.15.119:37792) 2024-03-28T18:49:01.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e52a00640 1 -- 172.21.15.119:0/3423659953 learned_addr learned my addr 172.21.15.119:0/3423659953 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e53400640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed0af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.856+0000 7f6e52000640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed05080 0x55d96ed0a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 -- 172.21.15.119:0/3423659953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 msgr2=0x55d96ed0af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed0af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 -- 172.21.15.119:0/3423659953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed05080 msgr2=0x55d96ed0a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed05080 0x55d96ed0a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 -- 172.21.15.119:0/3423659953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d96e97c3f0 con 0x55d96ed046d0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e53400640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed0af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e52a00640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6e44000c00 tx=0x7f6e4400f890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee93b78a0da272bf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400fe90 con 0x55d96ed046d0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e4400fce0 con 0x55d96ed046d0 2024-03-28T18:49:01.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.857+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6e44016050 con 0x55d96ed046d0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e4402f050 con 0x55d96ed046d0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 10 client.26830 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d96ea467d0 con 0x55d96ed046d0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4400fce0 con 0x55d96ed046d0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6e440051d0 con 0x55d96ed046d0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 10 client.26830 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 10 client.26830 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 20 client.26830 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed1d9e0 0x55d96ed1fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 4 client.26830 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.858+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d96ec75800 con 0x55d96ed1d9e0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.859+0000 7f6e52000640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed1d9e0 0x55d96ed1fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.859+0000 7f6e52000640 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed1d9e0 0x55d96ed1fdd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88bdda4dad991ab5 server_cookie=acf940b58fe7a in_seq=0 out_seq=0 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.859+0000 7f6e40c00640 10 client.26830 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:01.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e40c00640 1 -- 172.21.15.119:0/3423659953 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d96ec75800 con 0x55d96ed1d9e0 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e40c00640 10 client.26830 handle_command_reply: tid=0 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 client.26830 shutdown 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 msgr2=0x55d96ed0d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6e44000c00 tx=0x7f6e4400f890 comp rx=0 tx=0).stop 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 shutdown_connections 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed1d9e0 0x55d96ed1fdd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed0af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed05080 0x55d96ed0a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/3423659953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed046d0 0x55d96ed0d590 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e40c00640 0 client.26830 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 >> 172.21.15.119:0/3423659953 conn(0x55d96ecfeda0 msgr2=0x55d96ed00af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 shutdown_connections 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 1 -- 172.21.15.119:0/3423659953 wait complete. 2024-03-28T18:49:01.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.860+0000 7f6e5cbea740 20 client.26830 trim_cache size 0 max 16384 2024-03-28T18:49:01.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 Processor -- start 2024-03-28T18:49:01.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 -- start start 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed0d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541af690 con 0x55d96ed046d0 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541b1010 con 0x55d96ed0dc00 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541afcf0 con 0x55d96ed05080 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e53400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e52000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed0d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46224/0 (socket says 172.21.15.119:46224) 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e53400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60968/0 (socket says 172.21.15.119:60968) 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.874+0000 7f6e52a00640 1 -- 172.21.15.119:0/680202775 learned_addr learned my addr 172.21.15.119:0/680202775 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 -- 172.21.15.119:0/680202775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 msgr2=0x55d96ed0d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed0d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 -- 172.21.15.119:0/680202775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 msgr2=0x55d96ed0d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 -- 172.21.15.119:0/680202775 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d96ed104b0 con 0x55d96ed0dc00 2024-03-28T18:49:01.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e52a00640 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e53400640 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0cc480 tx=0x7f6e4c097760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a821ac23d82743 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e43400640 1 -- 172.21.15.119:0/680202775 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0b4070 con 0x55d96ed0dc00 2024-03-28T18:49:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e43400640 1 -- 172.21.15.119:0/680202775 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c0b4210 con 0x55d96ed0dc00 2024-03-28T18:49:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.875+0000 7f6e43400640 1 -- 172.21.15.119:0/680202775 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c01c040 con 0x55d96ed0dc00 2024-03-28T18:49:01.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- 172.21.15.119:0/680202775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 msgr2=0x55d96ed07ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0cc480 tx=0x7f6e4c097760 comp rx=0 tx=0).stop 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- 172.21.15.119:0/680202775 shutdown_connections 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed07ba0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed0d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/680202775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed0d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- 172.21.15.119:0/680202775 >> 172.21.15.119:0/680202775 conn(0x55d96ecfeda0 msgr2=0x55d96ecfdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- 172.21.15.119:0/680202775 shutdown_connections 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- 172.21.15.119:0/680202775 wait complete. 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.876+0000 7f6e5cbea740 1 -- start start 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 Processor -- start 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed13af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed14030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541b09b0 con 0x55d96ed046d0 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541b0350 con 0x55d96ed0dc00 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541ad850 con 0x55d96ed05080 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e5cbea740 10 client.0 _ll_register_callbacks cb 0x7f6e44011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e52000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed14030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e53400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e52a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed13af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e53400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:60980/0 (socket says 172.21.15.119:60980) 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.877+0000 7f6e53400640 1 -- 172.21.15.119:0/823251137 learned_addr learned my addr 172.21.15.119:0/823251137 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 -- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 msgr2=0x55d96ed14030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed14030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 -- 172.21.15.119:0/823251137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 msgr2=0x55d96ed13af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed13af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 -- 172.21.15.119:0/823251137 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d96ed78570 con 0x55d96ed0dc00 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e52a00640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed13af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e52000640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed14030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e53400640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0cc4e0 tx=0x7f6e4c005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c190ddfd158f768 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0b9060 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f6e4c0ccdd0 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e4c0b4210 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e4c01c070 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e5cbea740 10 client.26858 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.878+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d96ed79870 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c0b4210 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6e4c09a7e0 con 0x55d96ed0dc00 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 10 client.26858 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 10 client.26858 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 20 client.26858 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed79e80 0x55d96ed7c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 4 client.26858 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d96ec6fb50 con 0x55d96ed79e80 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.879+0000 7f6e52a00640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed79e80 0x55d96ed7c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e52a00640 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed79e80 0x55d96ed7c270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2cf5f68cdc81a8d server_cookie=1fd1661259c35b23 in_seq=0 out_seq=0 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e40c00640 10 client.26858 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e40c00640 1 -- 172.21.15.119:0/823251137 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d96ec6fb50 con 0x55d96ed79e80 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e40c00640 10 client.26858 handle_command_reply: tid=0 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 client.26858 shutdown 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 msgr2=0x55d96ed78030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0cc4e0 tx=0x7f6e4c005750 comp rx=0 tx=0).stop 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 shutdown_connections 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d96ed79e80 0x55d96ed7c270 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d96ed0dc00 0x55d96ed78030 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d96ed05080 0x55d96ed14030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 --2- 172.21.15.119:0/823251137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d96ed046d0 0x55d96ed13af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e40c00640 0 client.26858 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:01.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 >> 172.21.15.119:0/823251137 conn(0x55d96ecfeda0 msgr2=0x55d96ed05460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 shutdown_connections 2024-03-28T18:49:01.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 1 -- 172.21.15.119:0/823251137 wait complete. 2024-03-28T18:49:01.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.880+0000 7f6e5cbea740 20 client.26858 trim_cache size 0 max 16384 2024-03-28T18:49:01.885 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 msgr2=0x7f6e340a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 0x7f6e340a4eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009ca0 tx=0x7f6e3c0324f0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 msgr2=0x7f6e5419ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 0x7f6e5419ea10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00e970 tx=0x7f6e4c00ee30 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 shutdown_connections 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6e340a29f0 0x7f6e340a4eb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6e54107640 0x7f6e5419ea10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e54101ff0 0x7f6e5419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 --2- 172.21.15.119:0/3071986632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e54101630 0x7f6e5419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 >> 172.21.15.119:0/3071986632 conn(0x7f6e54078aa0 msgr2=0x7f6e5410cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 shutdown_connections 2024-03-28T18:49:01.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:01.881+0000 7f6e59e00640 1 -- 172.21.15.119:0/3071986632 wait complete. 2024-03-28T18:49:01.899 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:01.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:49:01.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:01 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1077450659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:02.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 -- 172.21.15.119:0/3196138964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 msgr2=0x7f8478075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 --2- 172.21.15.119:0/3196138964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f8478075fb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f846c009cd0 tx=0x7f846c02c0e0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 -- 172.21.15.119:0/3196138964 shutdown_connections 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 --2- 172.21.15.119:0/3196138964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84781124e0 0x7f84781148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 --2- 172.21.15.119:0/3196138964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84780764f0 0x7f8478111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 --2- 172.21.15.119:0/3196138964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f8478075fb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 -- 172.21.15.119:0/3196138964 >> 172.21.15.119:0/3196138964 conn(0x7f84780ff350 msgr2=0x7f8478101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 -- 172.21.15.119:0/3196138964 shutdown_connections 2024-03-28T18:49:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.110+0000 7f8480400640 1 -- 172.21.15.119:0/3196138964 wait complete. 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.111+0000 7f8480400640 1 Processor -- start 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.111+0000 7f8480400640 1 -- start start 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84780764f0 0x7f847819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 0x7f84781a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478118040 con 0x7f84781124e0 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478117ec0 con 0x7f84780764f0 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f8480400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781181c0 con 0x7f8478075bd0 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37836/0 (socket says 172.21.15.119:37836) 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f847da00640 1 -- 172.21.15.119:0/728512088 learned_addr learned my addr 172.21.15.119:0/728512088 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f847d000640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84780764f0 0x7f847819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.112+0000 7f847e400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 0x7f84781a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 -- 172.21.15.119:0/728512088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84780764f0 msgr2=0x7f847819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84780764f0 0x7f847819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 -- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 msgr2=0x7f84781a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 0x7f84781a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 -- 172.21.15.119:0/728512088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84781a35b0 con 0x7f8478075bd0 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847e400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 0x7f84781a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:02.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f847da00640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f846c000f80 tx=0x7f846c032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f8476000640 1 -- 172.21.15.119:0/728512088 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c039040 con 0x7f8478075bd0 2024-03-28T18:49:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f8476000640 1 -- 172.21.15.119:0/728512088 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c044070 con 0x7f8478075bd0 2024-03-28T18:49:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f8476000640 1 -- 172.21.15.119:0/728512088 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0042a0 con 0x7f8478075bd0 2024-03-28T18:49:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84781a3840 con 0x7f8478075bd0 2024-03-28T18:49:02.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.113+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84781ab130 con 0x7f8478075bd0 2024-03-28T18:49:02.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 Processor -- start 2024-03-28T18:49:02.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 -- start start 2024-03-28T18:49:02.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8476000640 1 -- 172.21.15.119:0/728512088 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f846c003970 con 0x7f8478075bd0 2024-03-28T18:49:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5bbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5bc080 0x5601ea5c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 0x5601ea5c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea51f2c0 con 0x5601ea5bc080 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea4c7c90 con 0x5601ea5c4c00 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.118+0000 7f8483066740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea51b360 con 0x5601ea5bb6d0 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847e400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5bbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847da00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5bc080 0x5601ea5c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 0x5601ea5c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847e400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5bbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37840/0 (socket says 172.21.15.119:37840) 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847e400640 1 -- 172.21.15.119:0/3030684232 learned_addr learned my addr 172.21.15.119:0/3030684232 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 -- 172.21.15.119:0/3030684232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 msgr2=0x5601ea5bbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5bbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 -- 172.21.15.119:0/3030684232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5bc080 msgr2=0x5601ea5c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5bc080 0x5601ea5c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 -- 172.21.15.119:0/3030684232 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601ea4ec080 con 0x5601ea5c4c00 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f847d000640 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 0x5601ea5c7050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8464009bf0 tx=0x7f8464011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4bf1b77ea61c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f846aa00640 1 -- 172.21.15.119:0/3030684232 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464004050 con 0x5601ea5c4c00 2024-03-28T18:49:02.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f8476000640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 0x7f84600a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.119+0000 7f846aa00640 1 -- 172.21.15.119:0/3030684232 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8464005500 con 0x5601ea5c4c00 2024-03-28T18:49:02.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f846aa00640 1 -- 172.21.15.119:0/3030684232 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8464004e40 con 0x5601ea5c4c00 2024-03-28T18:49:02.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8476000640 1 -- 172.21.15.119:0/728512088 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f846c0f1280 con 0x7f8478075bd0 2024-03-28T18:49:02.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f847e400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 0x7f84600a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 -- 172.21.15.119:0/3030684232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 msgr2=0x5601ea5c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 0x5601ea5c7050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8464009bf0 tx=0x7f8464011810 comp rx=0 tx=0).stop 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 -- 172.21.15.119:0/3030684232 shutdown_connections 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5c4c00 0x5601ea5c7050 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5bc080 0x5601ea5c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 --2- 172.21.15.119:0/3030684232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5bbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 -- 172.21.15.119:0/3030684232 >> 172.21.15.119:0/3030684232 conn(0x5601ea5b5da0 msgr2=0x5601ea5b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f8483066740 1 -- 172.21.15.119:0/3030684232 shutdown_connections 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.120+0000 7f847e400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 0x7f84600a4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8470009d50 tx=0x7f8470005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.121+0000 7f8483066740 1 -- 172.21.15.119:0/3030684232 wait complete. 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.121+0000 7f8483066740 1 -- start start 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.121+0000 7f8483066740 1 Processor -- start 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea5c7b70 con 0x5601ea5c4c00 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea4db0d0 con 0x5601ea5bc080 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ea5c7cf0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f8483066740 10 client.0 _ll_register_callbacks cb 0x5601ea4dfe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847d000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37844/0 (socket says 172.21.15.119:37844) 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 -- 172.21.15.119:0/2749011613 learned_addr learned my addr 172.21.15.119:0/2749011613 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847e400640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 -- 172.21.15.119:0/2749011613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 msgr2=0x5601ea5c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 -- 172.21.15.119:0/2749011613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 msgr2=0x5601ea5c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847da00640 1 -- 172.21.15.119:0/2749011613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601ea2333f0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.122+0000 7f847e400640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f847da00640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f846c0eeae0 tx=0x7f846c0b9cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea2228b328b6dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0be120 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f846c032590 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8464005500 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f846c0bda20 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f8483066740 10 client.26848 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601ea2fd7d0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8464005500 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f846c03c030 con 0x5601ea5bb6d0 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f8483066740 10 client.26848 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.123+0000 7f8483066740 10 client.26848 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f8483066740 20 client.26848 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea5d4950 0x5601ea5d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f8483066740 4 client.26848 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:02.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601ea52c800 con 0x5601ea5d4950 2024-03-28T18:49:02.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f847d000640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea5d4950 0x5601ea5d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f847d000640 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea5d4950 0x5601ea5d6d80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1caa9e2e984425a5 server_cookie=5693d0b02a8603ed in_seq=0 out_seq=0 2024-03-28T18:49:02.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.124+0000 7f845f400640 10 client.26848 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f845f400640 1 -- 172.21.15.119:0/2749011613 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5601ea52c800 con 0x5601ea5d4950 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f845f400640 10 client.26848 handle_command_reply: tid=0 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 client.26848 shutdown 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 msgr2=0x5601ea5c4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f846c0eeae0 tx=0x7f846c0b9cc0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 shutdown_connections 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea5d4950 0x5601ea5d6d80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f845f400640 0 client.26848 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:02.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.125+0000 7f8483066740 1 --2- 172.21.15.119:0/2749011613 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c4570 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.126+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 >> 172.21.15.119:0/2749011613 conn(0x5601ea5b5da0 msgr2=0x5601ea5b7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.126+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 shutdown_connections 2024-03-28T18:49:02.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.126+0000 7f8483066740 1 -- 172.21.15.119:0/2749011613 wait complete. 2024-03-28T18:49:02.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.126+0000 7f8483066740 20 client.26848 trim_cache size 0 max 16384 2024-03-28T18:49:02.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 Processor -- start 2024-03-28T18:49:02.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 -- start start 2024-03-28T18:49:02.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b3970 con 0x5601ea5c4c00 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b52f0 con 0x5601ea5bc080 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.139+0000 7f8483066740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b3fd0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37860/0 (socket says 172.21.15.119:37860) 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 -- 172.21.15.119:0/102096428 learned_addr learned my addr 172.21.15.119:0/102096428 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847d000640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847e400640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 -- 172.21.15.119:0/102096428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 msgr2=0x5601ea5c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 -- 172.21.15.119:0/102096428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 msgr2=0x5601ea5beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 -- 172.21.15.119:0/102096428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601ea5c2510 con 0x5601ea5bb6d0 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847e400640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5beb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f847da00640 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f846c0bd9b0 tx=0x7f846c0bbf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148f0826a284655e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f846a000640 1 -- 172.21.15.119:0/102096428 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0bd7f0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.140+0000 7f846a000640 1 -- 172.21.15.119:0/102096428 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c0be120 con 0x5601ea5bb6d0 2024-03-28T18:49:02.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f846a000640 1 -- 172.21.15.119:0/102096428 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0c5ca0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 -- 172.21.15.119:0/102096428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 msgr2=0x5601ea5c3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f846c0bd9b0 tx=0x7f846c0bbf30 comp rx=0 tx=0).stop 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 -- 172.21.15.119:0/102096428 shutdown_connections 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5beb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 --2- 172.21.15.119:0/102096428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5c3fc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 -- 172.21.15.119:0/102096428 >> 172.21.15.119:0/102096428 conn(0x5601ea5b5da0 msgr2=0x5601ea5b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 -- 172.21.15.119:0/102096428 shutdown_connections 2024-03-28T18:49:02.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.141+0000 7f8483066740 1 -- 172.21.15.119:0/102096428 wait complete. 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 -- start start 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 Processor -- start 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5cae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b4c90 con 0x5601ea5c4c00 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b4630 con 0x5601ea5bc080 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781b1b30 con 0x5601ea5bb6d0 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f8483066740 10 client.0 _ll_register_callbacks cb 0x7f846c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f847d000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f847da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37864/0 (socket says 172.21.15.119:37864) 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.142+0000 7f847da00640 1 -- 172.21.15.119:0/3330817627 learned_addr learned my addr 172.21.15.119:0/3330817627 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847e400640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 -- 172.21.15.119:0/3330817627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 msgr2=0x5601ea5cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 -- 172.21.15.119:0/3330817627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 msgr2=0x5601ea5cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 -- 172.21.15.119:0/3330817627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601ea5cb660 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847e400640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f847da00640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f846c0b9e80 tx=0x7f846c0f4bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9619f813ad76574b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f845f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0bd7f0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f846c0c5eb0 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f846c0be120 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f846c0c1b70 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f8483066740 10 client.26854 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.143+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601ea5cb660 con 0x5601ea5bb6d0 2024-03-28T18:49:02.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f846c0be120 con 0x5601ea5bb6d0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f846c0bce70 con 0x5601ea5bb6d0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 10 client.26854 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 10 client.26854 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 20 client.26854 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea6310d0 0x5601ea6334c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 4 client.26854 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5601ea526b50 con 0x5601ea6310d0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.144+0000 7f847d000640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea6310d0 0x5601ea6334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f847d000640 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea6310d0 0x5601ea6334c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4dcd803fa608e742 server_cookie=d4886e0bbfc35df7 in_seq=0 out_seq=0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f845f400640 10 client.26854 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f845f400640 1 -- 172.21.15.119:0/3330817627 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5601ea526b50 con 0x5601ea6310d0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f845f400640 10 client.26854 handle_command_reply: tid=0 2024-03-28T18:49:02.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 client.26854 shutdown 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 msgr2=0x5601ea5cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f846c0b9e80 tx=0x7f846c0f4bf0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 shutdown_connections 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5601ea6310d0 0x5601ea6334c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601ea5c4c00 0x5601ea5cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5601ea5bc080 0x5601ea5cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f845f400640 0 client.26854 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 --2- 172.21.15.119:0/3330817627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5601ea5bb6d0 0x5601ea5cc910 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 >> 172.21.15.119:0/3330817627 conn(0x5601ea5b5da0 msgr2=0x5601ea5bc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 shutdown_connections 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 1 -- 172.21.15.119:0/3330817627 wait complete. 2024-03-28T18:49:02.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.145+0000 7f8483066740 20 client.26854 trim_cache size 0 max 16384 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 msgr2=0x7f84600a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 0x7f84600a4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8470009d50 tx=0x7f8470005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 msgr2=0x7f847819e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f846c000f80 tx=0x7f846c032460 comp rx=0 tx=0).stop 2024-03-28T18:49:02.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 shutdown_connections 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84600a29f0 0x7f84600a4eb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84781124e0 0x7f84781a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84780764f0 0x7f847819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 --2- 172.21.15.119:0/728512088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8478075bd0 0x7f847819e610 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 >> 172.21.15.119:0/728512088 conn(0x7f84780ff350 msgr2=0x7f8478101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 shutdown_connections 2024-03-28T18:49:02.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.146+0000 7f8480400640 1 -- 172.21.15.119:0/728512088 wait complete. 2024-03-28T18:49:02.165 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:02.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 -- 172.21.15.119:0/1077393475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c10def0 msgr2=0x7f428c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 --2- 172.21.15.119:0/1077393475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c10def0 0x7f428c1102e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4284009be0 tx=0x7f428402bf80 comp rx=0 tx=0).stop 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 -- 172.21.15.119:0/1077393475 shutdown_connections 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 --2- 172.21.15.119:0/1077393475 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c10def0 0x7f428c1102e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 --2- 172.21.15.119:0/1077393475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c074370 0x7f428c10d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 --2- 172.21.15.119:0/1077393475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c073a50 0x7f428c073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 -- 172.21.15.119:0/1077393475 >> 172.21.15.119:0/1077393475 conn(0x7f428c078aa0 msgr2=0x7f428c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 -- 172.21.15.119:0/1077393475 shutdown_connections 2024-03-28T18:49:02.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.378+0000 7f4293600640 1 -- 172.21.15.119:0/1077393475 wait complete. 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 Processor -- start 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 -- start start 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c074370 0x7f428c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c10def0 0x7f428c19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c113c60 con 0x7f428c074370 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c113ae0 con 0x7f428c073a50 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.379+0000 7f4293600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c113de0 con 0x7f428c10def0 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:32816/0 (socket says 172.21.15.119:32816) 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 -- 172.21.15.119:0/2809216729 learned_addr learned my addr 172.21.15.119:0/2809216729 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f428be00640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c074370 0x7f428c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4291600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c10def0 0x7f428c19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 -- 172.21.15.119:0/2809216729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c10def0 msgr2=0x7f428c19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c10def0 0x7f428c19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 -- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c074370 msgr2=0x7f428c19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c074370 0x7f428c19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 -- 172.21.15.119:0/2809216729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c19f130 con 0x7f428c073a50 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.380+0000 7f4290c00640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f428000d950 tx=0x7f428000dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.381+0000 7f4289600640 1 -- 172.21.15.119:0/2809216729 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280017070 con 0x7f428c073a50 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.381+0000 7f4289600640 1 -- 172.21.15.119:0/2809216729 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f428000de80 con 0x7f428c073a50 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.381+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c19f420 con 0x7f428c073a50 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.381+0000 7f4289600640 1 -- 172.21.15.119:0/2809216729 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000f620 con 0x7f428c073a50 2024-03-28T18:49:02.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.381+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c1a6cb0 con 0x7f428c073a50 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 Processor -- start 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f4289600640 1 -- 172.21.15.119:0/2809216729 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4280009c30 con 0x7f428c073a50 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 -- start start 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf535ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf536080 0x555abf53e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf4992c0 con 0x555abf5356d0 2024-03-28T18:49:02.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf441c90 con 0x555abf536080 2024-03-28T18:49:02.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.384+0000 7f429639b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf495360 con 0x555abf53ec00 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f4291600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf535ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37892/0 (socket says 172.21.15.119:37892) 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 -- 172.21.15.119:0/2699248346 learned_addr learned my addr 172.21.15.119:0/2699248346 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f4290c00640 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf536080 0x555abf53e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 -- 172.21.15.119:0/2699248346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf536080 msgr2=0x555abf53e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf536080 0x555abf53e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 -- 172.21.15.119:0/2699248346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 msgr2=0x555abf535ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf535ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 -- 172.21.15.119:0/2699248346 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555abf466080 con 0x555abf53ec00 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.385+0000 7f428be00640 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4278009c40 tx=0x7f4278011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189c398d8e9c664c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f427dc00640 1 -- 172.21.15.119:0/2699248346 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278004230 con 0x555abf53ec00 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f427dc00640 1 -- 172.21.15.119:0/2699248346 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42780043d0 con 0x555abf53ec00 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f427dc00640 1 -- 172.21.15.119:0/2699248346 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278004e50 con 0x555abf53ec00 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f4289600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 0x7f42740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f429639b740 1 -- 172.21.15.119:0/2699248346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 msgr2=0x555abf541050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f429639b740 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4278009c40 tx=0x7f4278011810 comp rx=0 tx=0).stop 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.386+0000 7f4289600640 1 -- 172.21.15.119:0/2809216729 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f42800cea30 con 0x7f428c073a50 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 -- 172.21.15.119:0/2699248346 shutdown_connections 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf541050 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf536080 0x555abf53e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 --2- 172.21.15.119:0/2699248346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf535ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 -- 172.21.15.119:0/2699248346 >> 172.21.15.119:0/2699248346 conn(0x555abf52fda0 msgr2=0x555abf530180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f4291600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 0x7f42740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 -- 172.21.15.119:0/2699248346 shutdown_connections 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 -- 172.21.15.119:0/2699248346 wait complete. 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 -- start start 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f4291600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 0x7f42740a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f4284009280 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.387+0000 7f429639b740 1 Processor -- start 2024-03-28T18:49:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 0x555abf53b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf53bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf541b70 con 0x555abf5356d0 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf4550d0 con 0x555abf53ec00 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555abf541cf0 con 0x555abf536080 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f429639b740 10 client.0 _ll_register_callbacks cb 0x555abf459e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f4290c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f428be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 0x555abf53b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f4290c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46308/0 (socket says 172.21.15.119:46308) 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f4290c00640 1 -- 172.21.15.119:0/3452482205 learned_addr learned my addr 172.21.15.119:0/3452482205 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.388+0000 7f4291600640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf53bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 -- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 msgr2=0x555abf53bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf53bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 -- 172.21.15.119:0/3452482205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 msgr2=0x555abf53e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 -- 172.21.15.119:0/3452482205 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555abf1ad3f0 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4290c00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4291600640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf53bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f428be00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 0x555abf53b820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4278009c40 tx=0x7f4278005220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9c218f4e4e35124 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4272000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278028020 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4278024420 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42780043d0 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4278033050 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.389+0000 7f429639b740 10 client.26866 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555abf2777d0 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42780043d0 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4278018070 con 0x555abf536080 2024-03-28T18:49:02.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 10 client.26866 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 10 client.26866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 20 client.26866 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf54e990 0x555abf550d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 4 client.26866 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555abf4a6800 con 0x555abf54e990 2024-03-28T18:49:02.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.390+0000 7f4290c00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf54e990 0x555abf550d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.391+0000 7f4290c00640 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf54e990 0x555abf550d80 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a202e8ca5210f37 server_cookie=d60acd6b6640cf9e in_seq=0 out_seq=0 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.391+0000 7f4272000640 10 client.26866 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f4272000640 1 -- 172.21.15.119:0/3452482205 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7f4280079e50 con 0x555abf54e990 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f4272000640 10 client.26866 handle_command_reply: tid=0 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 client.26866 shutdown 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 msgr2=0x555abf53b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 0x555abf53b820 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4278009c40 tx=0x7f4278005220 comp rx=0 tx=0).stop 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 shutdown_connections 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf54e990 0x555abf550d80 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf53bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf536080 0x555abf53b820 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 --2- 172.21.15.119:0/3452482205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf5356d0 0x555abf53e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f4272000640 0 client.26866 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 >> 172.21.15.119:0/3452482205 conn(0x555abf52fda0 msgr2=0x555abf531af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 shutdown_connections 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.392+0000 7f429639b740 1 -- 172.21.15.119:0/3452482205 wait complete. 2024-03-28T18:49:02.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.393+0000 7f429639b740 20 client.26866 trim_cache size 0 max 16384 2024-03-28T18:49:02.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 Processor -- start 2024-03-28T18:49:02.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 -- start start 2024-03-28T18:49:02.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 0x555abf531130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf53e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1af500 con 0x555abf536080 2024-03-28T18:49:02.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1b0e80 con 0x555abf5356d0 2024-03-28T18:49:02.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.406+0000 7f429639b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1afb60 con 0x555abf53ec00 2024-03-28T18:49:02.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f4290c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 0x555abf531130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46316/0 (socket says 172.21.15.119:46316) 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 -- 172.21.15.119:0/2894833353 learned_addr learned my addr 172.21.15.119:0/2894833353 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f4291600640 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf53e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 -- 172.21.15.119:0/2894833353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 msgr2=0x555abf53e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf53e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 -- 172.21.15.119:0/2894833353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 msgr2=0x555abf531130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 0x555abf531130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 -- 172.21.15.119:0/2894833353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555abf537400 con 0x555abf536080 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f4290c00640 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 0x555abf531130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.407+0000 7f428be00640 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4278011850 tx=0x7f42780048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3170b355f6598edf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.408+0000 7f427d200640 1 -- 172.21.15.119:0/2894833353 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278028040 con 0x555abf536080 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.408+0000 7f427d200640 1 -- 172.21.15.119:0/2894833353 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f427801f040 con 0x555abf536080 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.408+0000 7f427d200640 1 -- 172.21.15.119:0/2894833353 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4278005290 con 0x555abf536080 2024-03-28T18:49:02.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.408+0000 7f429639b740 1 -- 172.21.15.119:0/2894833353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 msgr2=0x555abf53df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.408+0000 7f429639b740 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4278011850 tx=0x7f42780048d0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 -- 172.21.15.119:0/2894833353 shutdown_connections 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf53ec00 0x555abf53e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf53df50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 --2- 172.21.15.119:0/2894833353 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf5356d0 0x555abf531130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 -- 172.21.15.119:0/2894833353 >> 172.21.15.119:0/2894833353 conn(0x555abf52fda0 msgr2=0x555abf536cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 -- 172.21.15.119:0/2894833353 shutdown_connections 2024-03-28T18:49:02.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 -- 172.21.15.119:0/2894833353 wait complete. 2024-03-28T18:49:02.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 -- start start 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.409+0000 7f429639b740 1 Processor -- start 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf542a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf542f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1b0820 con 0x555abf536080 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1b01c0 con 0x555abf53ec00 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1ad6c0 con 0x555abf5356d0 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f429639b740 10 client.0 _ll_register_callbacks cb 0x7f4284009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f428be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf542a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37910/0 (socket says 172.21.15.119:37910) 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 -- 172.21.15.119:0/2373519410 learned_addr learned my addr 172.21.15.119:0/2373519410 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4291600640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf542f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 -- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 msgr2=0x555abf542f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf542f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 -- 172.21.15.119:0/2373519410 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 msgr2=0x555abf542a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf542a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 -- 172.21.15.119:0/2373519410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555abf5a91f0 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4290c00640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f42800cc720 tx=0x7f4280000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ccaff7ed398a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.410+0000 7f4272000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280020040 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f42800cc3a0 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f427801f040 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f428009b730 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 10 client.26872 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555abf5aa4d0 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f427801f040 con 0x555abf5356d0 2024-03-28T18:49:02.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4280095ba0 con 0x555abf5356d0 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 10 client.26872 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 10 client.26872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 20 client.26872 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf5aab30 0x555abf5acf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 4 client.26872 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.411+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555abf53c0f0 con 0x555abf5aab30 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f428be00640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf5aab30 0x555abf5acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f428be00640 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf5aab30 0x555abf5acf50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4608cfaec4cefb server_cookie=5e5678ea84e05aa6 in_seq=0 out_seq=0 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f4272000640 10 client.26872 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f4272000640 1 -- 172.21.15.119:0/2373519410 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555abf53c0f0 con 0x555abf5aab30 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f4272000640 10 client.26872 handle_command_reply: tid=0 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f429639b740 1 client.26872 shutdown 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 msgr2=0x555abf542500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.412+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f42800cc720 tx=0x7f4280000bc0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 shutdown_connections 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x555abf5aab30 0x555abf5acf50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555abf53ec00 0x555abf542f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555abf536080 0x555abf542a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 --2- 172.21.15.119:0/2373519410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555abf5356d0 0x555abf542500 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f4272000640 0 client.26872 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 >> 172.21.15.119:0/2373519410 conn(0x555abf52fda0 msgr2=0x555abf536cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 shutdown_connections 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 1 -- 172.21.15.119:0/2373519410 wait complete. 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f429639b740 20 client.26872 trim_cache size 0 max 16384 2024-03-28T18:49:02.418 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 msgr2=0x7f42740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 0x7f42740a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f4284009280 comp rx=0 tx=0).stop 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 msgr2=0x7f428c19a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.413+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f428000d950 tx=0x7f428000dd50 comp rx=0 tx=0).stop 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 shutdown_connections 2024-03-28T18:49:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42740a29f0 0x7f42740a4eb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f428c10def0 0x7f428c19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f428c074370 0x7f428c19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 --2- 172.21.15.119:0/2809216729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f428c073a50 0x7f428c19a140 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 >> 172.21.15.119:0/2809216729 conn(0x7f428c078aa0 msgr2=0x7f428c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 shutdown_connections 2024-03-28T18:49:02.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:02.414+0000 7f4293600640 1 -- 172.21.15.119:0/2809216729 wait complete. 2024-03-28T18:49:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:02 smithi119 ceph-mon[31813]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:02 smithi196 ceph-mon[36300]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:02 smithi131 ceph-mon[36065]: pgmap v152: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:03.432 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:49:03.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:49:03.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:49:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 -- 172.21.15.119:0/3724717622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 msgr2=0x7f4698103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 --2- 172.21.15.119:0/3724717622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f4698103cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4680009930 tx=0x7f468002bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 -- 172.21.15.119:0/3724717622 shutdown_connections 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 --2- 172.21.15.119:0/3724717622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698104200 0x7f4698110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 --2- 172.21.15.119:0/3724717622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f4698103cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 --2- 172.21.15.119:0/3724717622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698109850 0x7f4698109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 -- 172.21.15.119:0/3724717622 >> 172.21.15.119:0/3724717622 conn(0x7f46980ff310 msgr2=0x7f4698101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 -- 172.21.15.119:0/3724717622 shutdown_connections 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.648+0000 7f469ea00640 1 -- 172.21.15.119:0/3724717622 wait complete. 2024-03-28T18:49:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.649+0000 7f469ea00640 1 Processor -- start 2024-03-28T18:49:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 -- start start 2024-03-28T18:49:03.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698104200 0x7f469819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698109850 0x7f469819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698113c70 con 0x7f4698104200 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698113af0 con 0x7f4698109850 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ea00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4698113df0 con 0x7f4698103840 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f469ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698109850 0x7f469819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37932/0 (socket says 172.21.15.119:37932) 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.650+0000 7f4697e00640 1 -- 172.21.15.119:0/1647426683 learned_addr learned my addr 172.21.15.119:0/1647426683 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f468fe00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698104200 0x7f469819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 -- 172.21.15.119:0/1647426683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698109850 msgr2=0x7f469819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698109850 0x7f469819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 -- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698104200 msgr2=0x7f469819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698104200 0x7f469819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 -- 172.21.15.119:0/1647426683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469819f180 con 0x7f4698103840 2024-03-28T18:49:03.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4697e00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f468800ea60 tx=0x7f468800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4695600640 1 -- 172.21.15.119:0/1647426683 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468800cd20 con 0x7f4698103840 2024-03-28T18:49:03.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4695600640 1 -- 172.21.15.119:0/1647426683 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46880035f0 con 0x7f4698103840 2024-03-28T18:49:03.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469819f470 con 0x7f4698103840 2024-03-28T18:49:03.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.651+0000 7f4695600640 1 -- 172.21.15.119:0/1647426683 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688010690 con 0x7f4698103840 2024-03-28T18:49:03.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.652+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46981a6d00 con 0x7f4698103840 2024-03-28T18:49:03.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.654+0000 7f4695600640 1 -- 172.21.15.119:0/1647426683 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f46880037a0 con 0x7f4698103840 2024-03-28T18:49:03.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 Processor -- start 2024-03-28T18:49:03.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 -- start start 2024-03-28T18:49:03.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x56493886b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938874040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938876970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649387cfeb0 con 0x564938874580 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x564938778a60 con 0x56493886b9a0 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.655+0000 7f46a165f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649387cbf50 con 0x56493886b080 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f469ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x56493886b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938874040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f469ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x56493886b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37944/0 (socket says 172.21.15.119:37944) 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f469ca00640 1 -- 172.21.15.119:0/3041921270 learned_addr learned my addr 172.21.15.119:0/3041921270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f468fe00640 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938876970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4695600640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 0x7f467c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4695600640 1 -- 172.21.15.119:0/1647426683 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f46880cf110 con 0x7f4698103840 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 -- 172.21.15.119:0/3041921270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 msgr2=0x56493886b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x56493886b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 -- 172.21.15.119:0/3041921270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 msgr2=0x564938876970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938876970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 -- 172.21.15.119:0/3041921270 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649385ac7e0 con 0x56493886b9a0 2024-03-28T18:49:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f469ca00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 0x7f467c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.656+0000 7f4697e00640 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938874040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f46880cc7a0 tx=0x7f4688003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f84ce78920bfbf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f468d200640 1 -- 172.21.15.119:0/3041921270 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688097950 con 0x56493886b9a0 2024-03-28T18:49:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f468d200640 1 -- 172.21.15.119:0/3041921270 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f468809bdb0 con 0x56493886b9a0 2024-03-28T18:49:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f468d200640 1 -- 172.21.15.119:0/3041921270 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468809b010 con 0x56493886b9a0 2024-03-28T18:49:03.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 -- 172.21.15.119:0/3041921270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 msgr2=0x564938874040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938874040 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f46880cc7a0 tx=0x7f4688003f30 comp rx=0 tx=0).stop 2024-03-28T18:49:03.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f469ca00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 0x7f467c0a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4690003bf0 tx=0x7f4690004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 -- 172.21.15.119:0/3041921270 shutdown_connections 2024-03-28T18:49:03.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938876970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938874040 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 --2- 172.21.15.119:0/3041921270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x56493886b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.657+0000 7f46a165f740 1 -- 172.21.15.119:0/3041921270 >> 172.21.15.119:0/3041921270 conn(0x5649388657b0 msgr2=0x564938865b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.658+0000 7f46a165f740 1 -- 172.21.15.119:0/3041921270 shutdown_connections 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.658+0000 7f46a165f740 1 -- 172.21.15.119:0/3041921270 wait complete. 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.658+0000 7f46a165f740 1 -- start start 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.658+0000 7f46a165f740 1 Processor -- start 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x5649388711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938871710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649388772b0 con 0x564938874580 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493878beb0 con 0x56493886b9a0 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x564938877430 con 0x56493886b080 2024-03-28T18:49:03.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f46a165f740 10 client.0 _ll_register_callbacks cb 0x564938780150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f468fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x5649388711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f469ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938871710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f468fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x5649388711d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:32892/0 (socket says 172.21.15.119:32892) 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f468fe00640 1 -- 172.21.15.119:0/2192661137 learned_addr learned my addr 172.21.15.119:0/2192661137 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 -- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 msgr2=0x5649388711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x5649388711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 -- 172.21.15.119:0/2192661137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 msgr2=0x564938871710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938871710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 -- 172.21.15.119:0/2192661137 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56493860cf70 con 0x56493886b080 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.659+0000 7f4697e00640 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873e40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4688098080 tx=0x7f4688097f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf7dbb70b14d84c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468809b1f0 con 0x56493886b080 2024-03-28T18:49:03.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f468809ab00 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f468809bdb0 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f468809b480 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 10 client.26884 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5649384a4170 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f468809bdb0 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f46880b42d0 con 0x56493886b080 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 10 client.26884 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 10 client.26884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 20 client.26884 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564938884360 0x564938886750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 4 client.26884 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.660+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649387dd3f0 con 0x564938884360 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.661+0000 7f468fe00640 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564938884360 0x564938886750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.661+0000 7f468fe00640 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564938884360 0x564938886750 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f9026fee3a12b55 server_cookie=78e3202bab1f94f7 in_seq=0 out_seq=0 2024-03-28T18:49:03.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.661+0000 7f4685600640 10 client.26884 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f4685600640 1 -- 172.21.15.119:0/2192661137 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5649387dd3f0 con 0x564938884360 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f4685600640 10 client.26884 handle_command_reply: tid=0 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 client.26884 shutdown 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 msgr2=0x564938873e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873e40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4688098080 tx=0x7f4688097f50 comp rx=0 tx=0).stop 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 shutdown_connections 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x564938884360 0x564938886750 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x564938871710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x5649388711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 --2- 172.21.15.119:0/2192661137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873e40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f4685600640 0 client.26884 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 >> 172.21.15.119:0/2192661137 conn(0x5649388657b0 msgr2=0x564938867500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 shutdown_connections 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.662+0000 7f46a165f740 1 -- 172.21.15.119:0/2192661137 wait complete. 2024-03-28T18:49:03.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.663+0000 7f46a165f740 20 client.26884 trim_cache size 0 max 16384 2024-03-28T18:49:03.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 Processor -- start 2024-03-28T18:49:03.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 -- start start 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938873e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x56493886e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981af4c0 con 0x564938874580 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981b0e40 con 0x56493886b9a0 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.680+0000 7f46a165f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981afb20 con 0x56493886b080 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f469ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x56493886e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37968/0 (socket says 172.21.15.119:37968) 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 -- 172.21.15.119:0/1332721761 learned_addr learned my addr 172.21.15.119:0/1332721761 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f468fe00640 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938873e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 -- 172.21.15.119:0/1332721761 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 msgr2=0x564938873e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938873e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 -- 172.21.15.119:0/1332721761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 msgr2=0x56493886e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x56493886e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 -- 172.21.15.119:0/1332721761 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649388701e0 con 0x56493886b080 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f468fe00640 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938873e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.681+0000 7f4697e00640 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f468800c6a0 tx=0x7f4688097950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d016ad06c11bfef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f4687e00640 1 -- 172.21.15.119:0/1332721761 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468809b1f0 con 0x56493886b080 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f4687e00640 1 -- 172.21.15.119:0/1332721761 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f468809bdb0 con 0x56493886b080 2024-03-28T18:49:03.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f4687e00640 1 -- 172.21.15.119:0/1332721761 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46880ccc30 con 0x56493886b080 2024-03-28T18:49:03.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 -- 172.21.15.119:0/1332721761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 msgr2=0x564938873900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f468800c6a0 tx=0x7f4688097950 comp rx=0 tx=0).stop 2024-03-28T18:49:03.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 -- 172.21.15.119:0/1332721761 shutdown_connections 2024-03-28T18:49:03.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564938874580 0x56493886e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56493886b9a0 0x564938873e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 --2- 172.21.15.119:0/1332721761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938873900 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 -- 172.21.15.119:0/1332721761 >> 172.21.15.119:0/1332721761 conn(0x5649388657b0 msgr2=0x56493886c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 -- 172.21.15.119:0/1332721761 shutdown_connections 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.682+0000 7f46a165f740 1 -- 172.21.15.119:0/1332721761 wait complete. 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 -- start start 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 Processor -- start 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56493886b9a0 0x564938888530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564938874580 0x5649388e80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981b07e0 con 0x56493886b9a0 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981b0180 con 0x564938874580 2024-03-28T18:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46981ad680 con 0x56493886b080 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.683+0000 7f46a165f740 10 client.0 _ll_register_callbacks cb 0x7f46900093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f468fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56493886b9a0 0x564938888530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f469ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564938874580 0x5649388e80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37972/0 (socket says 172.21.15.119:37972) 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 -- 172.21.15.119:0/1977998142 learned_addr learned my addr 172.21.15.119:0/1977998142 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 -- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564938874580 msgr2=0x5649388e80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564938874580 0x5649388e80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 -- 172.21.15.119:0/1977998142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56493886b9a0 msgr2=0x564938888530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56493886b9a0 0x564938888530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 -- 172.21.15.119:0/1977998142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649388e85f0 con 0x56493886b080 2024-03-28T18:49:03.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4697e00640 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f468809b3b0 tx=0x7f46880cf390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aceb83f1809b76df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4685600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468809b010 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4688097700 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.684+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f468809bdb0 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46880ccc30 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 10 client.26890 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5649388e85f0 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f468809bdb0 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f468809a570 con 0x56493886b080 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 10 client.26890 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 10 client.26890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 20 client.26890 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5649388e99c0 0x5649388ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 4 client.26890 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5649387d7740 con 0x5649388e99c0 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.685+0000 7f468fe00640 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5649388e99c0 0x5649388ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.686+0000 7f468fe00640 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5649388e99c0 0x5649388ebdb0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96c0cd663cd9394f server_cookie=9ebad57f7d4463b7 in_seq=0 out_seq=0 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.686+0000 7f4685600640 10 client.26890 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f4685600640 1 -- 172.21.15.119:0/1977998142 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5649387d7740 con 0x5649388e99c0 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f4685600640 10 client.26890 handle_command_reply: tid=0 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 client.26890 shutdown 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 msgr2=0x564938887ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f468809b3b0 tx=0x7f46880cf390 comp rx=0 tx=0).stop 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 shutdown_connections 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5649388e99c0 0x5649388ebdb0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x564938874580 0x5649388e80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56493886b9a0 0x564938888530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 --2- 172.21.15.119:0/1977998142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56493886b080 0x564938887ff0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f4685600640 0 client.26890 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 >> 172.21.15.119:0/1977998142 conn(0x5649388657b0 msgr2=0x564938863fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 shutdown_connections 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 1 -- 172.21.15.119:0/1977998142 wait complete. 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.687+0000 7f46a165f740 20 client.26890 trim_cache size 0 max 16384 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "01714888-fcc8-41dc-9860-4f0aef69045b", 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:49:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:03.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 msgr2=0x7f467c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 0x7f467c0a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4690003bf0 tx=0x7f4690004a40 comp rx=0 tx=0).stop 2024-03-28T18:49:03.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 msgr2=0x7f469819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f468800ea60 tx=0x7f468800ee60 comp rx=0 tx=0).stop 2024-03-28T18:49:03.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 shutdown_connections 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f467c0a29f0 0x7f467c0a4eb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4698109850 0x7f469819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4698104200 0x7f469819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 --2- 172.21.15.119:0/1647426683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4698103840 0x7f469819a190 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 >> 172.21.15.119:0/1647426683 conn(0x7f46980ff310 msgr2=0x7f46980ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 shutdown_connections 2024-03-28T18:49:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.688+0000 7f469ea00640 1 -- 172.21.15.119:0/1647426683 wait complete. 2024-03-28T18:49:03.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:49:03.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:49:03.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 -- 172.21.15.119:0/2880245408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830109850 msgr2=0x7fe830109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 --2- 172.21.15.119:0/2880245408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830109850 0x7fe830109c30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe820009860 tx=0x7fe82002bb70 comp rx=0 tx=0).stop 2024-03-28T18:49:03.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 -- 172.21.15.119:0/2880245408 shutdown_connections 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 --2- 172.21.15.119:0/2880245408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830104200 0x7fe830110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 --2- 172.21.15.119:0/2880245408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830103840 0x7fe830103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 --2- 172.21.15.119:0/2880245408 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830109850 0x7fe830109c30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 -- 172.21.15.119:0/2880245408 >> 172.21.15.119:0/2880245408 conn(0x7fe8300ff2f0 msgr2=0x7fe830101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 -- 172.21.15.119:0/2880245408 shutdown_connections 2024-03-28T18:49:03.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.919+0000 7fe837600640 1 -- 172.21.15.119:0/2880245408 wait complete. 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.920+0000 7fe837600640 1 Processor -- start 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.920+0000 7fe837600640 1 -- start start 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830104200 0x7fe83019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 0x7fe83019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113c60 con 0x7fe830109850 2024-03-28T18:49:03.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113ae0 con 0x7fe830104200 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe837600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830113de0 con 0x7fe830103840 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830104200 0x7fe83019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37982/0 (socket says 172.21.15.119:37982) 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 -- 172.21.15.119:0/479678440 learned_addr learned my addr 172.21.15.119:0/479678440 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe835600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 0x7fe83019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 -- 172.21.15.119:0/479678440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830104200 msgr2=0x7fe83019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830104200 0x7fe83019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 -- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 msgr2=0x7fe83019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 0x7fe83019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe834c00640 1 -- 172.21.15.119:0/479678440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83019f130 con 0x7fe830103840 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.921+0000 7fe835600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 0x7fe83019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe834c00640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe820037040 tx=0x7fe8200323b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe82d600640 1 -- 172.21.15.119:0/479678440 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820041020 con 0x7fe830103840 2024-03-28T18:49:03.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83019f3c0 con 0x7fe830103840 2024-03-28T18:49:03.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe82d600640 1 -- 172.21.15.119:0/479678440 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe82002c9f0 con 0x7fe830103840 2024-03-28T18:49:03.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe82d600640 1 -- 172.21.15.119:0/479678440 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820039040 con 0x7fe830103840 2024-03-28T18:49:03.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.922+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8301a6cb0 con 0x7fe830103840 2024-03-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 Processor -- start 2024-03-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe82d600640 1 -- 172.21.15.119:0/479678440 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe820044020 con 0x7fe830103840 2024-03-28T18:49:03.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 -- start start 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 0x56018edce950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edcf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edda680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018ed33250 con 0x56018edcfc00 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018ecdba30 con 0x56018edcef20 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.925+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018ed2f2f0 con 0x56018edd8290 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe835600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 0x56018edce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe834c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edcf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe835600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 0x56018edce950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46408/0 (socket says 172.21.15.119:46408) 2024-03-28T18:49:03.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edda680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe835600640 1 -- 172.21.15.119:0/2916646051 learned_addr learned my addr 172.21.15.119:0/2916646051 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 -- 172.21.15.119:0/2916646051 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 msgr2=0x56018edcf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edcf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 -- 172.21.15.119:0/2916646051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 msgr2=0x56018edce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 0x56018edce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 -- 172.21.15.119:0/2916646051 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018ecc6690 con 0x56018edd8290 2024-03-28T18:49:03.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe82fe00640 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edda680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005d00 tx=0x7fe81c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d9c3df602e13852 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.926+0000 7fe825c00640 1 -- 172.21.15.119:0/2916646051 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c01c480 con 0x56018edd8290 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe82d600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 0x7fe8140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe825c00640 1 -- 172.21.15.119:0/2916646051 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c01cb00 con 0x56018edd8290 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe825c00640 1 -- 172.21.15.119:0/2916646051 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c011c50 con 0x56018edd8290 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe82d600640 1 -- 172.21.15.119:0/479678440 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe8200f07e0 con 0x7fe830103840 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe835600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 0x7fe8140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 -- 172.21.15.119:0/2916646051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 msgr2=0x56018edda680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edda680 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005d00 tx=0x7fe81c0107a0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 -- 172.21.15.119:0/2916646051 shutdown_connections 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edda680 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edcf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/2916646051 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcfc00 0x56018edce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 -- 172.21.15.119:0/2916646051 >> 172.21.15.119:0/2916646051 conn(0x56018ed96200 msgr2=0x56018edc9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe83a1f2740 1 -- 172.21.15.119:0/2916646051 shutdown_connections 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.927+0000 7fe835600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 0x7fe8140a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe828009fd0 tx=0x7fe828011040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.928+0000 7fe83a1f2740 1 -- 172.21.15.119:0/2916646051 wait complete. 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.928+0000 7fe83a1f2740 1 -- start start 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.928+0000 7fe83a1f2740 1 Processor -- start 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.928+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.928+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 0x56018edd4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edd51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018eddb100 con 0x56018edcef20 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018eceee80 con 0x56018edcfc00 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56018eddb280 con 0x56018edd8290 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe83a1f2740 10 client.0 _ll_register_callbacks cb 0x56018ed01130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe834c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 0x56018edd4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edd51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 0x56018edd4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:32958/0 (socket says 172.21.15.119:32958) 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe834c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46424/0 (socket says 172.21.15.119:46424) 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe82fe00640 1 -- 172.21.15.119:0/3829108884 learned_addr learned my addr 172.21.15.119:0/3829108884 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 -- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 msgr2=0x56018edd4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 0x56018edd4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 -- 172.21.15.119:0/3829108884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 msgr2=0x56018edd78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 -- 172.21.15.119:0/3829108884 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56018eb117d0 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe834c00640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe835600640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edd51b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe828005fd0 tx=0x7fe828004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a82ce13394d34c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.929+0000 7fe81a000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828029030 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe82801bd10 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81c01cb00 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe828021040 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe83a1f2740 10 client.26902 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56018ea473f0 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c01cb00 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe828024030 con 0x56018edd8290 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe83a1f2740 10 client.26902 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.930+0000 7fe83a1f2740 10 client.26902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:03.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe83a1f2740 20 client.26902 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ede8190 0x56018edea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe83a1f2740 4 client.26902 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56018ed40790 con 0x56018ede8190 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe834c00640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ede8190 0x56018edea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe834c00640 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ede8190 0x56018edea5c0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=292fba74d43630b8 server_cookie=c67e5a3408b187e in_seq=0 out_seq=0 2024-03-28T18:49:03.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.931+0000 7fe81a000640 10 client.26902 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe81a000640 1 -- 172.21.15.119:0/3829108884 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56018ed40790 con 0x56018ede8190 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe81a000640 10 client.26902 handle_command_reply: tid=0 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 client.26902 shutdown 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 msgr2=0x56018edd51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edd51b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe828005fd0 tx=0x7fe828004ab0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 shutdown_connections 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ede8190 0x56018edea5c0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edd8290 0x56018edd51b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcfc00 0x56018edd4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/3829108884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.932+0000 7fe81a000640 0 client.26902 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.933+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 >> 172.21.15.119:0/3829108884 conn(0x56018ed96200 msgr2=0x56018edcb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.933+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 shutdown_connections 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.933+0000 7fe83a1f2740 1 -- 172.21.15.119:0/3829108884 wait complete. 2024-03-28T18:49:03.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.933+0000 7fe83a1f2740 20 client.26902 trim_cache size 0 max 16384 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 Processor -- start 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 -- start start 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edd8290 0x56018edd9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301af540 con 0x56018edcef20 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301b0ec0 con 0x56018edd8290 2024-03-28T18:49:03.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.950+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301afba0 con 0x56018edcfc00 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe834c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:38014/0 (socket says 172.21.15.119:38014) 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 -- 172.21.15.119:0/1557885273 learned_addr learned my addr 172.21.15.119:0/1557885273 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe835600640 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edd8290 0x56018edd9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 -- 172.21.15.119:0/1557885273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edd8290 msgr2=0x56018edd9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edd8290 0x56018edd9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 -- 172.21.15.119:0/1557885273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 msgr2=0x56018edd73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 -- 172.21.15.119:0/1557885273 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56018ea070f0 con 0x56018edcfc00 2024-03-28T18:49:03.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe834c00640 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.951+0000 7fe82fe00640 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005fd0 tx=0x7fe81c011820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f29f13eb893bd4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe825200640 1 -- 172.21.15.119:0/1557885273 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00bd20 con 0x56018edcfc00 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe825200640 1 -- 172.21.15.119:0/1557885273 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c0118e0 con 0x56018edcfc00 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe825200640 1 -- 172.21.15.119:0/1557885273 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0052a0 con 0x56018edcfc00 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 -- 172.21.15.119:0/1557885273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 msgr2=0x56018edd78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005fd0 tx=0x7fe81c011820 comp rx=0 tx=0).stop 2024-03-28T18:49:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 -- 172.21.15.119:0/1557885273 shutdown_connections 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edd8290 0x56018edd9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd78e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/1557885273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edcef20 0x56018edd73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.952+0000 7fe83a1f2740 1 -- 172.21.15.119:0/1557885273 >> 172.21.15.119:0/1557885273 conn(0x56018ed96200 msgr2=0x56018edc94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.953+0000 7fe83a1f2740 1 -- 172.21.15.119:0/1557885273 shutdown_connections 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.953+0000 7fe83a1f2740 1 -- 172.21.15.119:0/1557885273 wait complete. 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.953+0000 7fe83a1f2740 1 -- start start 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.953+0000 7fe83a1f2740 1 Processor -- start 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edd8290 0x56018edd1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301b0860 con 0x56018edd8290 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301b0200 con 0x56018edcef20 2024-03-28T18:49:03.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301ad700 con 0x56018edcfc00 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe83a1f2740 10 client.0 _ll_register_callbacks cb 0x7fe828009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe835600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edd8290 0x56018edd1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:32986/0 (socket says 172.21.15.119:32986) 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 -- 172.21.15.119:0/149438642 learned_addr learned my addr 172.21.15.119:0/149438642 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 -- 172.21.15.119:0/149438642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 msgr2=0x56018edd1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 -- 172.21.15.119:0/149438642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edd8290 msgr2=0x56018edd1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edd8290 0x56018edd1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 -- 172.21.15.119:0/149438642 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56018ee42c70 con 0x56018edcef20 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe834c00640 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe8200b9860 tx=0x7fe8200f6530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd3c3051c0236120 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.954+0000 7fe81a000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8200bc7f0 con 0x56018edcef20 2024-03-28T18:49:03.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81c0118e0 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe8200baf30 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe820034910 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 10 client.26930 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56018ee43fc0 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe81c0118e0 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe82003c030 con 0x56018edcef20 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 10 client.26930 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 10 client.26930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 20 client.26930 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ee44690 0x56018ee46a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 4 client.26930 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.955+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56018ed3aae0 con 0x56018ee44690 2024-03-28T18:49:03.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe82fe00640 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ee44690 0x56018ee46a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe82fe00640 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ee44690 0x56018ee46a80 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=858988aefab5641b server_cookie=d39bbb165bc4018d in_seq=0 out_seq=0 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe81a000640 10 client.26930 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe81a000640 1 -- 172.21.15.119:0/149438642 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56018ed3aae0 con 0x56018ee44690 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe81a000640 10 client.26930 handle_command_reply: tid=0 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe83a1f2740 1 client.26930 shutdown 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 msgr2=0x56018edd1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.956+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe8200b9860 tx=0x7fe8200f6530 comp rx=0 tx=0).stop 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 shutdown_connections 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x56018ee44690 0x56018ee46a80 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56018edd8290 0x56018edd1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56018edcfc00 0x56018edd1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 --2- 172.21.15.119:0/149438642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56018edcef20 0x56018edd1560 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 >> 172.21.15.119:0/149438642 conn(0x56018ed96200 msgr2=0x56018edd3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe81a000640 0 client.26930 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:03.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 shutdown_connections 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 1 -- 172.21.15.119:0/149438642 wait complete. 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe83a1f2740 20 client.26930 trim_cache size 0 max 16384 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "01714888-fcc8-41dc-9860-4f0aef69045b": { 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "01714888-fcc8-41dc-9860-4f0aef69045b", 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:49:03.964 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:49:03.965 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:03.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 msgr2=0x7fe8140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 0x7fe8140a4e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe828009fd0 tx=0x7fe828011040 comp rx=0 tx=0).stop 2024-03-28T18:49:03.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 msgr2=0x7fe83019a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:03.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.957+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe820037040 tx=0x7fe8200323b0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 shutdown_connections 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8140a2970 0x7fe8140a4e30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe830109850 0x7fe83019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe830104200 0x7fe83019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 --2- 172.21.15.119:0/479678440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe830103840 0x7fe83019a140 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 >> 172.21.15.119:0/479678440 conn(0x7fe8300ff2f0 msgr2=0x7fe8300ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 shutdown_connections 2024-03-28T18:49:03.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:03.958+0000 7fe837600640 1 -- 172.21.15.119:0/479678440 wait complete. 2024-03-28T18:49:03.976 INFO:tasks.cephfs.filesystem:scrub status for tag:01714888-fcc8-41dc-9860-4f0aef69045b - {'path': '/', 'tag': '01714888-fcc8-41dc-9860-4f0aef69045b', 'options': 'recursive,force'} 2024-03-28T18:49:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:04 smithi119 ceph-mon[31813]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:04.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:04 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:49:04.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:04 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:04.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:04 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:49:04.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:04 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:04 smithi196 ceph-mon[36300]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:04 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:49:04.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:04 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:04.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:04 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:49:04.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:04 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:04 smithi131 ceph-mon[36065]: pgmap v153: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:04 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:49:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:04 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:04.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:04 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:49:04.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:04 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:06.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:06 smithi119 ceph-mon[31813]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:06 smithi196 ceph-mon[36300]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:06 smithi131 ceph-mon[36065]: pgmap v154: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:08.735 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:08 smithi196 ceph-mon[36300]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:08.735 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:08 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:49:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:08 smithi119 ceph-mon[31813]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:08 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:49:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:08 smithi131 ceph-mon[36065]: pgmap v155: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:08 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:49:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:10 smithi119 ceph-mon[31813]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:10 smithi196 ceph-mon[36300]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:10 smithi131 ceph-mon[36065]: pgmap v156: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:12 smithi119 ceph-mon[31813]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:12.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:12 smithi196 ceph-mon[36300]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:12.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:12 smithi131 ceph-mon[36065]: pgmap v157: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:14 smithi119 ceph-mon[31813]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:14 smithi196 ceph-mon[36300]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:14 smithi131 ceph-mon[36065]: pgmap v158: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:49:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:16 smithi119 ceph-mon[31813]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:16 smithi196 ceph-mon[36300]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:16 smithi131 ceph-mon[36065]: pgmap v159: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:49:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:18 smithi119 ceph-mon[31813]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:49:18.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:18 smithi196 ceph-mon[36300]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:49:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:18 smithi131 ceph-mon[36065]: pgmap v160: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:49:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:19 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:19.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:19 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:19.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:19 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:49:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:20 smithi119 ceph-mon[31813]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:49:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:20 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:20 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:20.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:20 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:49:20.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:20 smithi196 ceph-mon[36300]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:49:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:20 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:20 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:20 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:20 smithi131 ceph-mon[36065]: pgmap v161: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:20 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:49:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:20 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:49:20.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:20 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:49:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:22 smithi119 ceph-mon[31813]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:22 smithi196 ceph-mon[36300]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:22 smithi131 ceph-mon[36065]: pgmap v162: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:24.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:24 smithi119 ceph-mon[31813]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:24 smithi196 ceph-mon[36300]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:24 smithi131 ceph-mon[36065]: pgmap v163: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:26 smithi119 ceph-mon[31813]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:26.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:26 smithi196 ceph-mon[36300]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:26.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:26 smithi131 ceph-mon[36065]: pgmap v164: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:28 smithi119 ceph-mon[31813]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:28 smithi196 ceph-mon[36300]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:28 smithi131 ceph-mon[36065]: pgmap v165: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:49:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:30 smithi119 ceph-mon[31813]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:30.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:30 smithi196 ceph-mon[36300]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:30.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:30 smithi131 ceph-mon[36065]: pgmap v166: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:32 smithi119 ceph-mon[31813]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:32.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:32 smithi196 ceph-mon[36300]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:32.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:32 smithi131 ceph-mon[36065]: pgmap v167: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:33.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:33 smithi119 ceph-mon[31813]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:33.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:33 smithi196 ceph-mon[36300]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:33.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:33 smithi131 ceph-mon[36065]: pgmap v168: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:33.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:49:33.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:49:34.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 -- 172.21.15.119:0/2502802028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c107640 msgr2=0x7fde8c107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 --2- 172.21.15.119:0/2502802028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c107640 0x7fde8c107a20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fde80009900 tx=0x7fde8002bc40 comp rx=0 tx=0).stop 2024-03-28T18:49:34.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 -- 172.21.15.119:0/2502802028 shutdown_connections 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 --2- 172.21.15.119:0/2502802028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101ff0 0x7fde8c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 --2- 172.21.15.119:0/2502802028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c101630 0x7fde8c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 --2- 172.21.15.119:0/2502802028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c107640 0x7fde8c107a20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 -- 172.21.15.119:0/2502802028 >> 172.21.15.119:0/2502802028 conn(0x7fde8c078aa0 msgr2=0x7fde8c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 -- 172.21.15.119:0/2502802028 shutdown_connections 2024-03-28T18:49:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.200+0000 7fde93e00640 1 -- 172.21.15.119:0/2502802028 wait complete. 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.201+0000 7fde93e00640 1 Processor -- start 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.201+0000 7fde93e00640 1 -- start start 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 0x7fde8c073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c107640 0x7fde8c074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c111a60 con 0x7fde8c107640 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1118e0 con 0x7fde8c101ff0 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde93e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c111be0 con 0x7fde8c101630 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde91400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde91400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:57976/0 (socket says 172.21.15.119:57976) 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde91400640 1 -- 172.21.15.119:0/2696436382 learned_addr learned my addr 172.21.15.119:0/2696436382 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde90a00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 0x7fde8c073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.202+0000 7fde91e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c107640 0x7fde8c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 -- 172.21.15.119:0/2696436382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 msgr2=0x7fde8c073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 0x7fde8c073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 -- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c107640 msgr2=0x7fde8c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c107640 0x7fde8c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 -- 172.21.15.119:0/2696436382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c1a63a0 con 0x7fde8c101630 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde90a00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 0x7fde8c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde91400640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fde80005ab0 tx=0x7fde800042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde8a000640 1 -- 172.21.15.119:0/2696436382 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80041070 con 0x7fde8c101630 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde8a000640 1 -- 172.21.15.119:0/2696436382 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde8002ccd0 con 0x7fde8c101630 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c1a6630 con 0x7fde8c101630 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde8a000640 1 -- 172.21.15.119:0/2696436382 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80003d10 con 0x7fde8c101630 2024-03-28T18:49:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.203+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c1a6b20 con 0x7fde8c101630 2024-03-28T18:49:34.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 Processor -- start 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde8a000640 1 -- 172.21.15.119:0/2696436382 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fde80044070 con 0x7fde8c101630 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 -- start start 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a8238483a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823853680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8237ac250 con 0x55a823851290 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a823754a30 con 0x55a823848c00 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde96b72740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8237a82f0 con 0x55a823847f20 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde91e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.207+0000 7fde91400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a8238483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48580/0 (socket says 172.21.15.119:48580) 2024-03-28T18:49:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 -- 172.21.15.119:0/2279666303 learned_addr learned my addr 172.21.15.119:0/2279666303 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde90a00640 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823853680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 -- 172.21.15.119:0/2279666303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 msgr2=0x55a8238483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a8238483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 -- 172.21.15.119:0/2279666303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 msgr2=0x55a823853680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823853680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 -- 172.21.15.119:0/2279666303 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a82373f690 con 0x55a823848c00 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91400640 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a8238483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.208+0000 7fde91e00640 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f50 tx=0x7fde840118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea1e68c9bc6d2907 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde7e600640 1 -- 172.21.15.119:0/2279666303 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84004d20 con 0x55a823848c00 2024-03-28T18:49:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde7e600640 1 -- 172.21.15.119:0/2279666303 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde84004050 con 0x55a823848c00 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde7e600640 1 -- 172.21.15.119:0/2279666303 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde84005280 con 0x55a823848c00 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde8a000640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 0x7fde740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde91400640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 0x7fde740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde8a000640 1 -- 172.21.15.119:0/2696436382 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fde800f0790 con 0x7fde8c101630 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde96b72740 1 -- 172.21.15.119:0/2279666303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 msgr2=0x55a823847950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde96b72740 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fde84009f50 tx=0x7fde840118d0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.209+0000 7fde91400640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 0x7fde740a4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fde800ee040 tx=0x7fde800f6550 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 -- 172.21.15.119:0/2279666303 shutdown_connections 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823853680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a8238483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 --2- 172.21.15.119:0/2279666303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823848c00 0x55a823847950 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 -- 172.21.15.119:0/2279666303 >> 172.21.15.119:0/2279666303 conn(0x55a82380f200 msgr2=0x55a823842970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 -- 172.21.15.119:0/2279666303 shutdown_connections 2024-03-28T18:49:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 -- 172.21.15.119:0/2279666303 wait complete. 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 -- start start 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.210+0000 7fde96b72740 1 Processor -- start 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a823850940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a82384e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a823854100 con 0x55a823848c00 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a823767e80 con 0x55a823851290 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a823854280 con 0x55a823847f20 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde96b72740 10 client.0 _ll_register_callbacks cb 0x55a82377a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde91e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a823850940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde90a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde90a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34990/0 (socket says 172.21.15.119:34990) 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde90a00640 1 -- 172.21.15.119:0/233344019 learned_addr learned my addr 172.21.15.119:0/233344019 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.211+0000 7fde91e00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a82384e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 -- 172.21.15.119:0/233344019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 msgr2=0x55a823850940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a823850940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 -- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 msgr2=0x55a82384e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a82384e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 -- 172.21.15.119:0/233344019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a82358a7d0 con 0x55a823848c00 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde91e00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a82384e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde90a00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fde78011780 tx=0x7fde78011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6bee5c282ea735 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde72a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78010ba0 con 0x55a823848c00 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde84004050 con 0x55a823848c00 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.212+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fde78010d40 con 0x55a823848c00 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde78004020 con 0x55a823848c00 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 10 client.16719 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a8234c03f0 con 0x55a823848c00 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde84004050 con 0x55a823848c00 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fde7801a410 con 0x55a823848c00 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 10 client.16719 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 10 client.16719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 20 client.16719 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a823861160 0x55a823863580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 4 client.16719 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.213+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8237b9790 con 0x55a823861160 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.214+0000 7fde91e00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a823861160 0x55a823863580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.214+0000 7fde91e00640 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a823861160 0x55a823863580 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45f59f79e99cd5a0 server_cookie=d6d618d4a89b996e in_seq=0 out_seq=0 2024-03-28T18:49:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.214+0000 7fde72a00640 10 client.16719 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:34.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde72a00640 1 -- 172.21.15.119:0/233344019 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a8237b9790 con 0x55a823861160 2024-03-28T18:49:34.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde72a00640 10 client.16719 handle_command_reply: tid=0 2024-03-28T18:49:34.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 client.16719 shutdown 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 msgr2=0x55a82384dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fde78011780 tx=0x7fde78011c40 comp rx=0 tx=0).stop 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 shutdown_connections 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a823861160 0x55a823863580 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a82384e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823848c00 0x55a82384dcd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde72a00640 0 client.16719 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 --2- 172.21.15.119:0/233344019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823847f20 0x55a823850940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.215+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 >> 172.21.15.119:0/233344019 conn(0x55a82380f200 msgr2=0x55a8238446c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.216+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 shutdown_connections 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.216+0000 7fde96b72740 1 -- 172.21.15.119:0/233344019 wait complete. 2024-03-28T18:49:34.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.216+0000 7fde96b72740 20 client.16719 trim_cache size 0 max 16384 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 Processor -- start 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 -- start start 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 0x55a823850400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a823850940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a823852680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1af500 con 0x55a823847f20 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1b0e80 con 0x55a823851290 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde96b72740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1afb60 con 0x55a823848c00 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde91e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 0x55a823850400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde90a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a823850940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde91e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 0x55a823850400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34992/0 (socket says 172.21.15.119:34992) 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.234+0000 7fde91e00640 1 -- 172.21.15.119:0/2799778063 learned_addr learned my addr 172.21.15.119:0/2799778063 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde91e00640 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a823852680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 -- 172.21.15.119:0/2799778063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 msgr2=0x55a823852680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a823852680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 -- 172.21.15.119:0/2799778063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 msgr2=0x55a823850400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 0x55a823850400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 -- 172.21.15.119:0/2799778063 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8234800f0 con 0x55a823848c00 2024-03-28T18:49:34.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde90a00640 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a823850940 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fde7801a880 tx=0x7fde780042c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d97e9daaa94b841 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde7dc00640 1 -- 172.21.15.119:0/2799778063 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78000e60 con 0x55a823848c00 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde7dc00640 1 -- 172.21.15.119:0/2799778063 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde780043e0 con 0x55a823848c00 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.235+0000 7fde7dc00640 1 -- 172.21.15.119:0/2799778063 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78010ba0 con 0x55a823848c00 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 -- 172.21.15.119:0/2799778063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 msgr2=0x55a823850940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a823850940 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fde7801a880 tx=0x7fde780042c0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 -- 172.21.15.119:0/2799778063 shutdown_connections 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823851290 0x55a823852680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a823850940 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 --2- 172.21.15.119:0/2799778063 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823847f20 0x55a823850400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 -- 172.21.15.119:0/2799778063 >> 172.21.15.119:0/2799778063 conn(0x55a82380f200 msgr2=0x55a8238424a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 -- 172.21.15.119:0/2799778063 shutdown_connections 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.236+0000 7fde96b72740 1 -- 172.21.15.119:0/2799778063 wait complete. 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 -- start start 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 Processor -- start 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823847f20 0x55a82384ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823856a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1b0820 con 0x55a823851290 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1b01c0 con 0x55a823847f20 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c1ad6c0 con 0x55a823848c00 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.237+0000 7fde96b72740 10 client.0 _ll_register_callbacks cb 0x7fde7800bbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:34.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58004/0 (socket says 172.21.15.119:58004) 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 -- 172.21.15.119:0/2219684636 learned_addr learned my addr 172.21.15.119:0/2219684636 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde91e00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823847f20 0x55a82384ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde91e00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823856a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 -- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823847f20 msgr2=0x55a82384ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823847f20 0x55a82384ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 -- 172.21.15.119:0/2219684636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 msgr2=0x55a823856a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823856a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 -- 172.21.15.119:0/2219684636 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a823857200 con 0x55a823848c00 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde90a00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fde78004390 tx=0x7fde7800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0141e3a114c22c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.238+0000 7fde72a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78037060 con 0x55a823848c00 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde780043e0 con 0x55a823848c00 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fde7800b390 con 0x55a823848c00 2024-03-28T18:49:34.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde7801aa70 con 0x55a823848c00 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 10 client.26926 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a8238bda40 con 0x55a823848c00 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde780043e0 con 0x55a823848c00 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fde780035f0 con 0x55a823848c00 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 10 client.26926 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 10 client.26926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 20 client.26926 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a8238bdfd0 0x55a8238c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 4 client.26926 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.239+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a8237b3ae0 con 0x55a8238bdfd0 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde91e00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a8238bdfd0 0x55a8238c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde91e00640 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a8238bdfd0 0x55a8238c03f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d010bdf6b612c82 server_cookie=1a5863a15888cddc in_seq=0 out_seq=0 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde72a00640 10 client.26926 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde72a00640 1 -- 172.21.15.119:0/2219684636 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a8237b3ae0 con 0x55a8238bdfd0 2024-03-28T18:49:34.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde72a00640 10 client.26926 handle_command_reply: tid=0 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.240+0000 7fde96b72740 1 client.26926 shutdown 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 msgr2=0x55a8238564e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fde78004390 tx=0x7fde7800bab0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 shutdown_connections 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55a8238bdfd0 0x55a8238c03f0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a823851290 0x55a823856a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a823848c00 0x55a8238564e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 --2- 172.21.15.119:0/2219684636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a823847f20 0x55a82384ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde72a00640 0 client.26926 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 >> 172.21.15.119:0/2219684636 conn(0x55a82380f200 msgr2=0x55a823843d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 shutdown_connections 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 1 -- 172.21.15.119:0/2219684636 wait complete. 2024-03-28T18:49:34.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.241+0000 7fde96b72740 20 client.26926 trim_cache size 0 max 16384 2024-03-28T18:49:34.246 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:34.246 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:49:34.246 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:49:34.246 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 msgr2=0x7fde740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 0x7fde740a4e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fde800ee040 tx=0x7fde800f6550 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 msgr2=0x7fde8c0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fde80005ab0 tx=0x7fde800042f0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 shutdown_connections 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fde740a29a0 0x7fde740a4e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde8c107640 0x7fde8c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fde8c101ff0 0x7fde8c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 --2- 172.21.15.119:0/2696436382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde8c101630 0x7fde8c0739a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 >> 172.21.15.119:0/2696436382 conn(0x7fde8c078aa0 msgr2=0x7fde8c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 shutdown_connections 2024-03-28T18:49:34.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.242+0000 7fde93e00640 1 -- 172.21.15.119:0/2696436382 wait complete. 2024-03-28T18:49:34.261 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:49:34.262 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 -- 172.21.15.119:0/1373265072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 msgr2=0x7fe7bc100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 --2- 172.21.15.119:0/1373265072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc100a20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009930 tx=0x7fe7a402bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 -- 172.21.15.119:0/1373265072 shutdown_connections 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 --2- 172.21.15.119:0/1373265072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 0x7fe7bc10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 --2- 172.21.15.119:0/1373265072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc100a20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 --2- 172.21.15.119:0/1373265072 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 -- 172.21.15.119:0/1373265072 >> 172.21.15.119:0/1373265072 conn(0x7fe7bc0776a0 msgr2=0x7fe7bc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 -- 172.21.15.119:0/1373265072 shutdown_connections 2024-03-28T18:49:34.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.467+0000 7fe7c1800640 1 -- 172.21.15.119:0/1373265072 wait complete. 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 Processor -- start 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 -- start start 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 0x7fe7bc102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc1109b0 con 0x7fe7bc100f60 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc110830 con 0x7fe7bc106730 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.468+0000 7fe7c1800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc110b30 con 0x7fe7bc1005a0 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58028/0 (socket says 172.21.15.119:58028) 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 -- 172.21.15.119:0/3980716716 learned_addr learned my addr 172.21.15.119:0/3980716716 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7bb400640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7b2000640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 0x7fe7bc102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 -- 172.21.15.119:0/3980716716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 msgr2=0x7fe7bc102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 -- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 msgr2=0x7fe7bc102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 0x7fe7bc102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 -- 172.21.15.119:0/3980716716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc1a64c0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7bb400640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7baa00640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00e9f0 tx=0x7fe7ac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac00cd80 con 0x7fe7bc1005a0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7ac010910 con 0x7fe7bc1005a0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.469+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc1a67b0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.470+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac018990 con 0x7fe7bc1005a0 2024-03-28T18:49:34.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.470+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc1a6cd0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.471+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc068490 con 0x7fe7bc1005a0 2024-03-28T18:49:34.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.472+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe7ac010ac0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.474+0000 7fe7b3e00640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 0x7fe7a00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.474+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe7ac0cf1f0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.474+0000 7fe7b2000640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 0x7fe7a00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.475+0000 7fe7b2000640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 0x7fe7a00a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009900 tx=0x7fe7a402df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.482+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe7ac09b840 con 0x7fe7bc1005a0 2024-03-28T18:49:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.722+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7bc1a6f30 con 0x7fe7bc1005a0 2024-03-28T18:49:34.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.724+0000 7fe7b3e00640 1 -- 172.21.15.119:0/3980716716 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fe7ac09f8e0 con 0x7fe7bc1005a0 2024-03-28T18:49:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:49:34.725 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:49:34.725 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:49:34.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 msgr2=0x7fe7a00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 0x7fe7a00a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009900 tx=0x7fe7a402df90 comp rx=0 tx=0).stop 2024-03-28T18:49:34.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 msgr2=0x7fe7bc102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac00e9f0 tx=0x7fe7ac00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 shutdown_connections 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe7a00a2970 0x7fe7a00a4e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe7bc106730 0x7fe7bc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc100f60 0x7fe7bc102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 --2- 172.21.15.119:0/3980716716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7bc1005a0 0x7fe7bc102230 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 >> 172.21.15.119:0/3980716716 conn(0x7fe7bc0776a0 msgr2=0x7fe7bc0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 shutdown_connections 2024-03-28T18:49:34.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.727+0000 7fe7c1800640 1 -- 172.21.15.119:0/3980716716 wait complete. 2024-03-28T18:49:34.747 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:34.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:49:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 -- 172.21.15.119:0/26267476 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101530 msgr2=0x7f30581019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 --2- 172.21.15.119:0/26267476 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101530 0x7f30581019b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3040009930 tx=0x7f304002bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 -- 172.21.15.119:0/26267476 shutdown_connections 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 --2- 172.21.15.119:0/26267476 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3058101ef0 0x7f305810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 --2- 172.21.15.119:0/26267476 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101530 0x7f30581019b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 --2- 172.21.15.119:0/26267476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30581075f0 0x7f30581079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 -- 172.21.15.119:0/26267476 >> 172.21.15.119:0/26267476 conn(0x7f30580fd000 msgr2=0x7f30580ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 -- 172.21.15.119:0/26267476 shutdown_connections 2024-03-28T18:49:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.956+0000 7f3057e00640 1 -- 172.21.15.119:0/26267476 wait complete. 2024-03-28T18:49:34.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.957+0000 7f3057e00640 1 Processor -- start 2024-03-28T18:49:34.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.957+0000 7f3057e00640 1 -- start start 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3058101530 0x7f30580739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058111900 con 0x7f3058101530 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058111780 con 0x7f3058101ef0 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058111a80 con 0x7f30581075f0 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3056000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3056a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3058101530 0x7f30580739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58044/0 (socket says 172.21.15.119:58044) 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3056000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48660/0 (socket says 172.21.15.119:48660) 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.958+0000 7f3057400640 1 -- 172.21.15.119:0/188454141 learned_addr learned my addr 172.21.15.119:0/188454141 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 -- 172.21.15.119:0/188454141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 msgr2=0x7f3058074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 -- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3058101530 msgr2=0x7f30580739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3058101530 0x7f30580739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 -- 172.21.15.119:0/188454141 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30581a63c0 con 0x7f3058101ef0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3057400640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3056000640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3040036040 tx=0x7f3040005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f304f400640 1 -- 172.21.15.119:0/188454141 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040043020 con 0x7f3058101ef0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f304f400640 1 -- 172.21.15.119:0/188454141 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3040005000 con 0x7f3058101ef0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30581a6650 con 0x7f3058101ef0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f304f400640 1 -- 172.21.15.119:0/188454141 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040003a80 con 0x7f3058101ef0 2024-03-28T18:49:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.959+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581a6bc0 con 0x7f3058101ef0 2024-03-28T18:49:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.962+0000 7f304f400640 1 -- 172.21.15.119:0/188454141 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3040046020 con 0x7f3058101ef0 2024-03-28T18:49:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.962+0000 7f30609cb740 1 Processor -- start 2024-03-28T18:49:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 -- start start 2024-03-28T18:49:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f84de0 0x55cfb7f87230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7edf2c0 con 0x55cfb7f7c260 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7e87c90 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f30609cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7edb360 con 0x55cfb7f84de0 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f3057400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f3056000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f84de0 0x55cfb7f87230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f3056a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f3057400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48676/0 (socket says 172.21.15.119:48676) 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.963+0000 7f3057400640 1 -- 172.21.15.119:0/3789729676 learned_addr learned my addr 172.21.15.119:0/3789729676 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 -- 172.21.15.119:0/3789729676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f84de0 msgr2=0x55cfb7f87230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f84de0 0x55cfb7f87230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 -- 172.21.15.119:0/3789729676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 msgr2=0x55cfb7f84810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 -- 172.21.15.119:0/3789729676 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfb7eac080 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3056a00640 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3057400640 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f305000e830 tx=0x7f305000ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d23ffa7383aaa67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f304f400640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 0x7f303c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3047e00640 1 -- 172.21.15.119:0/3789729676 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050004d20 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3047e00640 1 -- 172.21.15.119:0/3789729676 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3050005020 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3047e00640 1 -- 172.21.15.119:0/3789729676 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305001a600 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f304f400640 1 -- 172.21.15.119:0/188454141 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f30400f0c30 con 0x7f3058101ef0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.964+0000 7f3056a00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 0x7f303c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- 172.21.15.119:0/3789729676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 msgr2=0x55cfb7f7bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f305000e830 tx=0x7f305000ecf0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- 172.21.15.119:0/3789729676 shutdown_connections 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f84de0 0x55cfb7f87230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 --2- 172.21.15.119:0/3789729676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f7bc90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- 172.21.15.119:0/3789729676 >> 172.21.15.119:0/3789729676 conn(0x55cfb7f75f80 msgr2=0x55cfb7f76360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f3056a00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 0x7f303c0a4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55cfb7f76ff0 tx=0x7f3048005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- 172.21.15.119:0/3789729676 shutdown_connections 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- 172.21.15.119:0/3789729676 wait complete. 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.965+0000 7f30609cb740 1 -- start start 2024-03-28T18:49:34.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 Processor -- start 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f84760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f82100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7f87d50 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7e9b0d0 con 0x55cfb7f84de0 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfb7f87ed0 con 0x55cfb7f7c260 2024-03-28T18:49:34.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f30609cb740 10 client.0 _ll_register_callbacks cb 0x55cfb7e9fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f3056000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f3056a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f84760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f3056000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58066/0 (socket says 172.21.15.119:58066) 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f3056000640 1 -- 172.21.15.119:0/1238715018 learned_addr learned my addr 172.21.15.119:0/1238715018 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.966+0000 7f3057400640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f82100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 -- 172.21.15.119:0/1238715018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 msgr2=0x55cfb7f82100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f82100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 -- 172.21.15.119:0/1238715018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 msgr2=0x55cfb7f84760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f84760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 -- 172.21.15.119:0/1238715018 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfb7bf33f0 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3056000640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f30400bd270 tx=0x7f3040009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e7eeb5a233e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3044c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400bcc40 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f30400dd0b0 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3050005020 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30400bd9e0 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.967+0000 7f30609cb740 10 client.26944 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cfb7cbd7d0 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3050005020 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f30400ee540 con 0x55cfb7f7c260 2024-03-28T18:49:34.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 10 client.26944 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 10 client.26944 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 20 client.26944 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7f94bc0 0x55cfb7f96fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 4 client.26944 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cfb7eec800 con 0x55cfb7f94bc0 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.968+0000 7f3056a00640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7f94bc0 0x55cfb7f96fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.969+0000 7f3044c00640 10 client.26944 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.969+0000 7f3056a00640 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7f94bc0 0x55cfb7f96fb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2d2cab716aa554f2 server_cookie=31d8c0355303a665 in_seq=0 out_seq=0 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f3044c00640 1 -- 172.21.15.119:0/1238715018 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cfb7eec800 con 0x55cfb7f94bc0 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f3044c00640 10 client.26944 handle_command_reply: tid=0 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 client.26944 shutdown 2024-03-28T18:49:34.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 msgr2=0x55cfb7f81af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f30400bd270 tx=0x7f3040009620 comp rx=0 tx=0).stop 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 shutdown_connections 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7f94bc0 0x55cfb7f96fb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f82100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f81af0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f3044c00640 0 client.26944 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 --2- 172.21.15.119:0/1238715018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f84760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 >> 172.21.15.119:0/1238715018 conn(0x55cfb7f75f80 msgr2=0x55cfb7f77cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 shutdown_connections 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 1 -- 172.21.15.119:0/1238715018 wait complete. 2024-03-28T18:49:34.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.970+0000 7f30609cb740 20 client.26944 trim_cache size 0 max 16384 2024-03-28T18:49:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 Processor -- start 2024-03-28T18:49:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 -- start start 2024-03-28T18:49:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f753a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f84760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581af410 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581b0d90 con 0x55cfb7f84de0 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f30609cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581afa70 con 0x55cfb7f7c260 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f3056a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f3056000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f3057400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f84760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f3056000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58076/0 (socket says 172.21.15.119:58076) 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.984+0000 7f3056000640 1 -- 172.21.15.119:0/3844576194 learned_addr learned my addr 172.21.15.119:0/3844576194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 -- 172.21.15.119:0/3844576194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 msgr2=0x55cfb7f84760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f84760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 -- 172.21.15.119:0/3844576194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 msgr2=0x55cfb7f753a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f753a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 -- 172.21.15.119:0/3844576194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfb7f97320 con 0x55cfb7f7c260 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056a00640 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f753a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3056000640 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3040006010 tx=0x7f3040000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9747d2defbad782e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3047400640 1 -- 172.21.15.119:0/3844576194 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400bcc40 con 0x55cfb7f7c260 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3047400640 1 -- 172.21.15.119:0/3844576194 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30400b7e50 con 0x55cfb7f7c260 2024-03-28T18:49:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.985+0000 7f3047400640 1 -- 172.21.15.119:0/3844576194 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400bd0b0 con 0x55cfb7f7c260 2024-03-28T18:49:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- 172.21.15.119:0/3844576194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 msgr2=0x55cfb7f84220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3040006010 tx=0x7f3040000bc0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- 172.21.15.119:0/3844576194 shutdown_connections 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f84de0 0x55cfb7f84760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7c260 0x55cfb7f84220 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 --2- 172.21.15.119:0/3844576194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f753a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- 172.21.15.119:0/3844576194 >> 172.21.15.119:0/3844576194 conn(0x55cfb7f75f80 msgr2=0x55cfb7f77b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- 172.21.15.119:0/3844576194 shutdown_connections 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- 172.21.15.119:0/3844576194 wait complete. 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.986+0000 7f30609cb740 1 -- start start 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 Processor -- start 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 0x55cfb7f8b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f84de0 0x55cfb7f8b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581b0730 con 0x55cfb7f84de0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581b00d0 con 0x55cfb7f7c260 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581ad5d0 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f30609cb740 10 client.0 _ll_register_callbacks cb 0x7f3040005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f3056a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f3056a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58084/0 (socket says 172.21.15.119:58084) 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f3056a00640 1 -- 172.21.15.119:0/1033986136 learned_addr learned my addr 172.21.15.119:0/1033986136 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f3057400640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f84de0 0x55cfb7f8b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.987+0000 7f3056000640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 0x55cfb7f8b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 -- 172.21.15.119:0/1033986136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 msgr2=0x55cfb7f8b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 0x55cfb7f8b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 -- 172.21.15.119:0/1033986136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f84de0 msgr2=0x55cfb7f8b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f84de0 0x55cfb7f8b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 -- 172.21.15.119:0/1033986136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfb7f8c1a0 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056000640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 0x55cfb7f8b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3056a00640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3048012e20 tx=0x7f30480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=598a3b211e389ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3044c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048004480 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30400b7e50 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.988+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3048003ac0 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3048003cc0 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 10 client.26950 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cfb7ff1550 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30400b7e50 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f304801f690 con 0x55cfb7f7b8b0 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 10 client.26950 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 10 client.26950 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 20 client.26950 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7ff1c00 0x55cfb7ff4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 4 client.26950 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cfb7ee6b50 con 0x55cfb7ff1c00 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.989+0000 7f3056000640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7ff1c00 0x55cfb7ff4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f3056000640 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7ff1c00 0x55cfb7ff4020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c1fca7345c9b4880 server_cookie=9e7f23895ff2105e in_seq=0 out_seq=0 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f3044c00640 10 client.26950 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f3044c00640 1 -- 172.21.15.119:0/1033986136 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cfb7ee6b50 con 0x55cfb7ff1c00 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f3044c00640 10 client.26950 handle_command_reply: tid=0 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 client.26950 shutdown 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 msgr2=0x55cfb7f8cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3048012e20 tx=0x7f30480046c0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 shutdown_connections 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55cfb7ff1c00 0x55cfb7ff4020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cfb7f84de0 0x55cfb7f8b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55cfb7f7c260 0x55cfb7f8b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 --2- 172.21.15.119:0/1033986136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cfb7f7b8b0 0x55cfb7f8cfb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f3044c00640 0 client.26950 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 >> 172.21.15.119:0/1033986136 conn(0x55cfb7f75f80 msgr2=0x55cfb7f7c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 shutdown_connections 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 1 -- 172.21.15.119:0/1033986136 wait complete. 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.990+0000 7f30609cb740 20 client.26950 trim_cache size 0 max 16384 2024-03-28T18:49:34.996 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 msgr2=0x7f303c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 0x7f303c0a4eb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55cfb7f76ff0 tx=0x7f3048005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 msgr2=0x7f3058073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3040036040 tx=0x7f3040005e80 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 shutdown_connections 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f303c0a29f0 0x7f303c0a4eb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30581075f0 0x7f3058074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3058101ef0 0x7f3058073ee0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 --2- 172.21.15.119:0/188454141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3058101530 0x7f30580739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 >> 172.21.15.119:0/188454141 conn(0x7f30580fd000 msgr2=0x7f30580fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 shutdown_connections 2024-03-28T18:49:34.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:34.991+0000 7f3057e00640 1 -- 172.21.15.119:0/188454141 wait complete. 2024-03-28T18:49:35.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3980716716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:35.009 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:35.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:49:35.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3980716716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:35.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3980716716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 -- 172.21.15.119:0/3102055849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8104200 msgr2=0x7f7fb8110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 --2- 172.21.15.119:0/3102055849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8104200 0x7f7fb8110710 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009900 tx=0x7f7fb002bc40 comp rx=0 tx=0).stop 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 -- 172.21.15.119:0/3102055849 shutdown_connections 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 --2- 172.21.15.119:0/3102055849 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8104200 0x7f7fb8110710 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 --2- 172.21.15.119:0/3102055849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 --2- 172.21.15.119:0/3102055849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8109850 0x7f7fb8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 -- 172.21.15.119:0/3102055849 >> 172.21.15.119:0/3102055849 conn(0x7f7fb80ff2f0 msgr2=0x7f7fb8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 -- 172.21.15.119:0/3102055849 shutdown_connections 2024-03-28T18:49:35.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.212+0000 7f7fbce00640 1 -- 172.21.15.119:0/3102055849 wait complete. 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 Processor -- start 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 -- start start 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 0x7f7fb819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113cb0 con 0x7f7fb8103840 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113b30 con 0x7f7fb8109850 2024-03-28T18:49:35.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.213+0000 7f7fbce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8113e30 con 0x7f7fb8104200 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 0x7f7fb819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48712/0 (socket says 172.21.15.119:48712) 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb6800640 1 -- 172.21.15.119:0/393780803 learned_addr learned my addr 172.21.15.119:0/393780803 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 -- 172.21.15.119:0/393780803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 msgr2=0x7f7fb819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 -- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 msgr2=0x7f7fb819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 -- 172.21.15.119:0/393780803 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb819f180 con 0x7f7fb8104200 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb6800640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.214+0000 7f7fb5400640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 0x7f7fb819a6d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e9f0 tx=0x7f7fa400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.215+0000 7f7faea00640 1 -- 172.21.15.119:0/393780803 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400ccc0 con 0x7f7fb8104200 2024-03-28T18:49:35.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.215+0000 7f7faea00640 1 -- 172.21.15.119:0/393780803 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa4010910 con 0x7f7fb8104200 2024-03-28T18:49:35.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.215+0000 7f7faea00640 1 -- 172.21.15.119:0/393780803 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4018960 con 0x7f7fb8104200 2024-03-28T18:49:35.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.215+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb819f470 con 0x7f7fb8104200 2024-03-28T18:49:35.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.215+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81a6d00 con 0x7f7fb8104200 2024-03-28T18:49:35.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 Processor -- start 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 -- start start 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x555733462630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555733462c00 0x555733465050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557333bd2c0 con 0x55573345a080 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555733365c90 con 0x555733462c00 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557333b9360 con 0x5557334596d0 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.218+0000 7f7faea00640 1 -- 172.21.15.119:0/393780803 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f7fa4010ac0 con 0x7f7fb8104200 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58124/0 (socket says 172.21.15.119:58124) 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 -- 172.21.15.119:0/2607271398 learned_addr learned my addr 172.21.15.119:0/2607271398 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x555733462630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb5400640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555733462c00 0x555733465050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 -- 172.21.15.119:0/2607271398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555733462c00 msgr2=0x555733465050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555733462c00 0x555733465050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 -- 172.21.15.119:0/2607271398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 msgr2=0x555733462630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x555733462630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 -- 172.21.15.119:0/2607271398 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55573338a080 con 0x5557334596d0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x555733462630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fb6800640 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005e90 tx=0x7f7fb0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3ed665d7e053bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.219+0000 7f7fa3400640 1 -- 172.21.15.119:0/2607271398 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0004a70 con 0x5557334596d0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fa3400640 1 -- 172.21.15.119:0/2607271398 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fb0004c10 con 0x5557334596d0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fa3400640 1 -- 172.21.15.119:0/2607271398 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0005440 con 0x5557334596d0 2024-03-28T18:49:35.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7faea00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 0x7f7f9c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7faea00640 1 -- 172.21.15.119:0/393780803 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f7fa40cf180 con 0x7f7fb8104200 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 -- 172.21.15.119:0/2607271398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 msgr2=0x555733459ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005e90 tx=0x7f7fb0009560 comp rx=0 tx=0).stop 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 0x7f7f9c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 -- 172.21.15.119:0/2607271398 shutdown_connections 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555733462c00 0x555733465050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x555733462630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/2607271398 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733459ab0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 -- 172.21.15.119:0/2607271398 >> 172.21.15.119:0/2607271398 conn(0x555733453da0 msgr2=0x555733454180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.220+0000 7f7fbfa08740 1 -- 172.21.15.119:0/2607271398 shutdown_connections 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.221+0000 7f7fbfa08740 1 -- 172.21.15.119:0/2607271398 wait complete. 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.221+0000 7f7fbfa08740 1 -- start start 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.221+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 0x7f7f9c0a4eb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x555733454e10 tx=0x7f7fa8009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.221+0000 7f7fbfa08740 1 Processor -- start 2024-03-28T18:49:35.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555733465b10 con 0x555733462c00 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557333790d0 con 0x5557334596d0 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555733465c90 con 0x55573345a080 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fbfa08740 10 client.0 _ll_register_callbacks cb 0x55573337de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58126/0 (socket says 172.21.15.119:58126) 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48716/0 (socket says 172.21.15.119:48716) 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 -- 172.21.15.119:0/482054389 learned_addr learned my addr 172.21.15.119:0/482054389 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb6800640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 -- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 msgr2=0x555733462500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 -- 172.21.15.119:0/482054389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 msgr2=0x55573345fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5400640 1 -- 172.21.15.119:0/482054389 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557330d13f0 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.222+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fb5400640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e9c0 tx=0x7f7fa409b640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33bbdf22177d9e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa409b870 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f7fa40d2b60 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fb0004c10 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fa409f910 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 10 client.26962 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55573319b7d0 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fb0004c10 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7fa409acc0 con 0x55573345a080 2024-03-28T18:49:35.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 10 client.26962 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 10 client.26962 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 20 client.26962 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334729a0 0x555733474d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 4 client.26962 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.223+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557333ca800 con 0x5557334729a0 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.224+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334729a0 0x555733474d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.224+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334729a0 0x555733474d90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7dcdc503294d86cd server_cookie=90a223a7471f64d2 in_seq=0 out_seq=0 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.224+0000 7f7f97e00640 10 client.26962 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.225+0000 7f7f97e00640 1 -- 172.21.15.119:0/482054389 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5557333ca800 con 0x5557334729a0 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.225+0000 7f7f97e00640 10 client.26962 handle_command_reply: tid=0 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.225+0000 7f7fbfa08740 1 client.26962 shutdown 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.225+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 msgr2=0x55573345f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.225+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e9c0 tx=0x7f7fa409b640 comp rx=0 tx=0).stop 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 shutdown_connections 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334729a0 0x555733474d90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55573345a080 0x55573345f890 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/482054389 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x555733462500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 >> 172.21.15.119:0/482054389 conn(0x555733453da0 msgr2=0x555733455af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7f97e00640 0 client.26962 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 shutdown_connections 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 1 -- 172.21.15.119:0/482054389 wait complete. 2024-03-28T18:49:35.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.226+0000 7f7fbfa08740 20 client.26962 trim_cache size 0 max 16384 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.239+0000 7f7fbfa08740 1 Processor -- start 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.239+0000 7f7fbfa08740 1 -- start start 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81af4b0 con 0x555733462c00 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81b0e30 con 0x55573345a080 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81afb10 con 0x5557334596d0 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58138/0 (socket says 172.21.15.119:58138) 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5e00640 1 -- 172.21.15.119:0/869009842 learned_addr learned my addr 172.21.15.119:0/869009842 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48724/0 (socket says 172.21.15.119:48724) 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb6800640 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 -- 172.21.15.119:0/869009842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 msgr2=0x555733461f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 -- 172.21.15.119:0/869009842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 msgr2=0x55573345caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5400640 1 -- 172.21.15.119:0/869009842 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555733458760 con 0x55573345a080 2024-03-28T18:49:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.240+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fb5400640 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400cf10 tx=0x7f7fa409b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0efff5c950f7ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fa2a00640 1 -- 172.21.15.119:0/869009842 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa409b3b0 con 0x55573345a080 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fa2a00640 1 -- 172.21.15.119:0/869009842 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa40a3f60 con 0x55573345a080 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fa2a00640 1 -- 172.21.15.119:0/869009842 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40afba0 con 0x55573345a080 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 -- 172.21.15.119:0/869009842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 msgr2=0x555733462490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400cf10 tx=0x7f7fa409b770 comp rx=0 tx=0).stop 2024-03-28T18:49:35.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 -- 172.21.15.119:0/869009842 shutdown_connections 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555733462c00 0x55573345caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55573345a080 0x555733462490 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/869009842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5557334596d0 0x555733461f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.241+0000 7f7fbfa08740 1 -- 172.21.15.119:0/869009842 >> 172.21.15.119:0/869009842 conn(0x555733453da0 msgr2=0x555733455a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.242+0000 7f7fbfa08740 1 -- 172.21.15.119:0/869009842 shutdown_connections 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.242+0000 7f7fbfa08740 1 -- 172.21.15.119:0/869009842 wait complete. 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.242+0000 7f7fbfa08740 1 -- start start 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.242+0000 7f7fbfa08740 1 Processor -- start 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x55573346aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 0x555733468b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81b07d0 con 0x55573345a080 2024-03-28T18:49:35.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81b0170 con 0x5557334596d0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81ad670 con 0x555733462c00 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fbfa08740 10 client.0 _ll_register_callbacks cb 0x7f7fa400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x55573346aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 0x555733468b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48726/0 (socket says 172.21.15.119:48726) 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 -- 172.21.15.119:0/737459578 learned_addr learned my addr 172.21.15.119:0/737459578 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 -- 172.21.15.119:0/737459578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 msgr2=0x555733468b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 0x555733468b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 -- 172.21.15.119:0/737459578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 msgr2=0x55573346aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x55573346aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5e00640 1 -- 172.21.15.119:0/737459578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555733469330 con 0x5557334596d0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb6800640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 0x555733468b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.243+0000 7f7fb5400640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x55573346aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7fb5e00640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8013010 tx=0x7f7fa8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=922dee054215ff94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7f97e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa802f070 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fa40a3f60 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f7fa8004600 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fa803b050 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7fbfa08740 10 client.26990 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.244+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5557334cec80 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa40a3f60 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7fa801f5c0 con 0x5557334596d0 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 10 client.26990 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 10 client.26990 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 20 client.26990 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334cf360 0x5557334d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 4 client.26990 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5557333c4b50 con 0x5557334cf360 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fb5400640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334cf360 0x5557334d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7fb5400640 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334cf360 0x5557334d1780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ed257275b668fbda server_cookie=269eaa34daf6bb96 in_seq=0 out_seq=0 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.245+0000 7f7f97e00640 10 client.26990 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7f97e00640 1 -- 172.21.15.119:0/737459578 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5557333c4b50 con 0x5557334cf360 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7f97e00640 10 client.26990 handle_command_reply: tid=0 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 client.26990 shutdown 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 msgr2=0x55573346a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8013010 tx=0x7f7fa8004a40 comp rx=0 tx=0).stop 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 shutdown_connections 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555733462c00 0x555733468b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5557334cf360 0x5557334d1780 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55573345a080 0x55573346aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 --2- 172.21.15.119:0/737459578 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5557334596d0 0x55573346a660 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7f97e00640 0 client.26990 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:49:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 >> 172.21.15.119:0/737459578 conn(0x555733453da0 msgr2=0x555733452620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 shutdown_connections 2024-03-28T18:49:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 1 -- 172.21.15.119:0/737459578 wait complete. 2024-03-28T18:49:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.246+0000 7f7fbfa08740 20 client.26990 trim_cache size 0 max 16384 2024-03-28T18:49:35.251 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 msgr2=0x7f7f9c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 0x7f7f9c0a4eb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x555733454e10 tx=0x7f7fa8009500 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 msgr2=0x7f7fb819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 0x7f7fb819a6d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400e9f0 tx=0x7f7fa400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 shutdown_connections 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7f9c0a29f0 0x7f7f9c0a4eb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7fb8109850 0x7f7fb819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fb8104200 0x7f7fb819a6d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 --2- 172.21.15.119:0/393780803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fb8103840 0x7f7fb819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 >> 172.21.15.119:0/393780803 conn(0x7f7fb80ff2f0 msgr2=0x7f7fb80ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 shutdown_connections 2024-03-28T18:49:35.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.247+0000 7f7fbce00640 1 -- 172.21.15.119:0/393780803 wait complete. 2024-03-28T18:49:35.264 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:35.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 -- 172.21.15.119:0/636595940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede010cf80 msgr2=0x7fede0114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 --2- 172.21.15.119:0/636595940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede010cf80 0x7fede0114b50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009ca0 tx=0x7fedd802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 -- 172.21.15.119:0/636595940 shutdown_connections 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 --2- 172.21.15.119:0/636595940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede010cf80 0x7fede0114b50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 --2- 172.21.15.119:0/636595940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede0108860 0x7fede010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.476+0000 7fede6800640 1 --2- 172.21.15.119:0/636595940 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede0107eb0 0x7fede0108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.477+0000 7fede6800640 1 -- 172.21.15.119:0/636595940 >> 172.21.15.119:0/636595940 conn(0x7fede0075be0 msgr2=0x7fede0078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.477+0000 7fede6800640 1 -- 172.21.15.119:0/636595940 shutdown_connections 2024-03-28T18:49:35.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.477+0000 7fede6800640 1 -- 172.21.15.119:0/636595940 wait complete. 2024-03-28T18:49:35.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 Processor -- start 2024-03-28T18:49:35.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 -- start start 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede010cf80 0x7fede019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 0x7fede01a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0118030 con 0x7fede010cf80 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.478+0000 7fede6800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0117eb0 con 0x7fede01a2df0 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7fede6800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01181b0 con 0x7fede0107eb0 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58166/0 (socket says 172.21.15.119:58166) 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 -- 172.21.15.119:0/2091500172 learned_addr learned my addr 172.21.15.119:0/2091500172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddfe00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 0x7fede01a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddea00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede010cf80 0x7fede019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 -- 172.21.15.119:0/2091500172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 msgr2=0x7fede01a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 0x7fede01a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 -- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede010cf80 msgr2=0x7fede019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede010cf80 0x7fede019ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 -- 172.21.15.119:0/2091500172 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede01a38e0 con 0x7fede0107eb0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddfe00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 0x7fede01a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.479+0000 7feddf400640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd000e9f0 tx=0x7fedd000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.480+0000 7fedd7e00640 1 -- 172.21.15.119:0/2091500172 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd000cd80 con 0x7fede0107eb0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.480+0000 7fedd7e00640 1 -- 172.21.15.119:0/2091500172 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedd0010910 con 0x7fede0107eb0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.480+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede01a3bd0 con 0x7fede0107eb0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.480+0000 7fedd7e00640 1 -- 172.21.15.119:0/2091500172 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0018990 con 0x7fede0107eb0 2024-03-28T18:49:35.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.480+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede01ab410 con 0x7fede0107eb0 2024-03-28T18:49:35.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.483+0000 7fedd7e00640 1 -- 172.21.15.119:0/2091500172 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fedd0010ac0 con 0x7fede0107eb0 2024-03-28T18:49:35.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.483+0000 7fede9590740 1 Processor -- start 2024-03-28T18:49:35.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 -- start start 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c37422eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c374237630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3741922c0 con 0x55c37422e6d0 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c37413ac90 con 0x55c374237c00 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7fede9590740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c37418e360 con 0x55c37422f080 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c374237630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c374237630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58170/0 (socket says 172.21.15.119:58170) 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddf400640 1 -- 172.21.15.119:0/278888560 learned_addr learned my addr 172.21.15.119:0/278888560 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48758/0 (socket says 172.21.15.119:48758) 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.484+0000 7feddfe00640 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c37422eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 -- 172.21.15.119:0/278888560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 msgr2=0x55c374237630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c374237630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 -- 172.21.15.119:0/278888560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 msgr2=0x55c37422eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c37422eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 -- 172.21.15.119:0/278888560 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c37415f080 con 0x55c374237c00 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7feddea00640 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fede006acf0 tx=0x7fedcc0103d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d842cebab6048e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7fedd7e00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 0x7fedc40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.485+0000 7fedd4c00640 1 -- 172.21.15.119:0/278888560 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc010e90 con 0x55c374237c00 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7fedd4c00640 1 -- 172.21.15.119:0/278888560 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedcc0035f0 con 0x55c374237c00 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7fedd4c00640 1 -- 172.21.15.119:0/278888560 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0116a0 con 0x55c374237c00 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7fedd7e00640 1 -- 172.21.15.119:0/2091500172 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fedd00cf1e0 con 0x7fede0107eb0 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7feddfe00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 0x7fedc40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7fede9590740 1 -- 172.21.15.119:0/278888560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 msgr2=0x55c37423a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7fede9590740 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fede006acf0 tx=0x7fedcc0103d0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.486+0000 7feddfe00640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 0x7fedc40a4eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fedd8005e30 tx=0x7fedd8005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 -- 172.21.15.119:0/278888560 shutdown_connections 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c374237c00 0x55c37423a050 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c374237630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 --2- 172.21.15.119:0/278888560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c37422eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 -- 172.21.15.119:0/278888560 >> 172.21.15.119:0/278888560 conn(0x55c374228da0 msgr2=0x55c374229180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 -- 172.21.15.119:0/278888560 shutdown_connections 2024-03-28T18:49:35.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 -- 172.21.15.119:0/278888560 wait complete. 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 -- start start 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.487+0000 7fede9590740 1 Processor -- start 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c3742348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c374234f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c37423ab70 con 0x55c374237c00 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c37414e0d0 con 0x55c37422e6d0 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c37423acf0 con 0x55c37422f080 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7fede9590740 10 client.0 _ll_register_callbacks cb 0x55c374152e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48770/0 (socket says 172.21.15.119:48770) 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 -- 172.21.15.119:0/3177639144 learned_addr learned my addr 172.21.15.119:0/3177639144 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddea00640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c3742348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddfe00640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c374234f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 -- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 msgr2=0x55c3742348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c3742348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 -- 172.21.15.119:0/3177639144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 msgr2=0x55c374234f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c374234f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddf400640 1 -- 172.21.15.119:0/3177639144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c373ea63f0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.488+0000 7feddfe00640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c374234f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7feddf400640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fedd009b7c0 tx=0x7fedd0097b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bef5e9b909d570 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd009b830 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fedd00ccaf0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedcc0035f0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedd00cccf0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 10 client.27002 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c373f707d0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedcc0035f0 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fedd00afb60 con 0x55c37422e6d0 2024-03-28T18:49:35.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 10 client.27002 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 10 client.27002 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 20 client.27002 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c374247b20 0x55c374249f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 4 client.27002 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.489+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c37419f800 con 0x55c374247b20 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.490+0000 7feddea00640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c374247b20 0x55c374249f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.490+0000 7feddea00640 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c374247b20 0x55c374249f10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c69eee63bdf12f7c server_cookie=d8b3fef9bf85110a in_seq=0 out_seq=0 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.490+0000 7fedc9600640 10 client.27002 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fedc9600640 1 -- 172.21.15.119:0/3177639144 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c37419f800 con 0x55c374247b20 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fedc9600640 10 client.27002 handle_command_reply: tid=0 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 client.27002 shutdown 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 msgr2=0x55c374237560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fedd009b7c0 tx=0x7fedd0097b90 comp rx=0 tx=0).stop 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 shutdown_connections 2024-03-28T18:49:35.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c374247b20 0x55c374249f10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c374234f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c3742348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 --2- 172.21.15.119:0/3177639144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c374237560 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fedc9600640 0 client.27002 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 >> 172.21.15.119:0/3177639144 conn(0x55c374228da0 msgr2=0x55c37422aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 shutdown_connections 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.491+0000 7fede9590740 1 -- 172.21.15.119:0/3177639144 wait complete. 2024-03-28T18:49:35.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.492+0000 7fede9590740 20 client.27002 trim_cache size 0 max 16384 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 Processor -- start 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 -- start start 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c374237020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c374237c00 0x55c374231b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b3a70 con 0x55c37422e6d0 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b53f0 con 0x55c37422f080 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7fede9590740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b40d0 con 0x55c374237c00 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.505+0000 7feddf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c374237020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c374237c00 0x55c374231b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48784/0 (socket says 172.21.15.119:48784) 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 -- 172.21.15.119:0/2050532541 learned_addr learned my addr 172.21.15.119:0/2050532541 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c374237020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35168/0 (socket says 172.21.15.119:35168) 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 -- 172.21.15.119:0/2050532541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c374237c00 msgr2=0x55c374231b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c374237c00 0x55c374231b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 -- 172.21.15.119:0/2050532541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 msgr2=0x55c374237020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c374237020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 -- 172.21.15.119:0/2050532541 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c37424a570 con 0x55c37422f080 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7feddea00640 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00ba90 tx=0x7fedcc0045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=331fa5b9f8f4cd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7fedcbe00640 1 -- 172.21.15.119:0/2050532541 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc010e60 con 0x55c37422f080 2024-03-28T18:49:35.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7fedcbe00640 1 -- 172.21.15.119:0/2050532541 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedcc02b030 con 0x55c37422f080 2024-03-28T18:49:35.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.506+0000 7fedcbe00640 1 -- 172.21.15.119:0/2050532541 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc005500 con 0x55c37422f080 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 -- 172.21.15.119:0/2050532541 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 msgr2=0x55c374237560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00ba90 tx=0x7fedcc0045f0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 -- 172.21.15.119:0/2050532541 shutdown_connections 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c374237c00 0x55c374231b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422f080 0x55c374237560 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 --2- 172.21.15.119:0/2050532541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c37422e6d0 0x55c374237020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 -- 172.21.15.119:0/2050532541 >> 172.21.15.119:0/2050532541 conn(0x55c374228da0 msgr2=0x55c37422aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 -- 172.21.15.119:0/2050532541 shutdown_connections 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.507+0000 7fede9590740 1 -- 172.21.15.119:0/2050532541 wait complete. 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 -- start start 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 Processor -- start 2024-03-28T18:49:35.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c37423e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c37429c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b4d90 con 0x55c374237c00 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b4730 con 0x55c37422e6d0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01b1c30 con 0x55c37422f080 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7fede9590740 10 client.0 _ll_register_callbacks cb 0x7fedd00d2980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7feddf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7feddf400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48786/0 (socket says 172.21.15.119:48786) 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7feddf400640 1 -- 172.21.15.119:0/3774528846 learned_addr learned my addr 172.21.15.119:0/3774528846 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7feddea00640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c37423e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.508+0000 7feddfe00640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c37429c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 -- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 msgr2=0x55c37423e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c37423e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 -- 172.21.15.119:0/3774528846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 msgr2=0x55c37429c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c37429c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 -- 172.21.15.119:0/3774528846 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c37429c930 con 0x55c37422e6d0 2024-03-28T18:49:35.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddfe00640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c37429c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7feddf400640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fedd0099ae0 tx=0x7fedd0097c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6df0a26deb06ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fedc9600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd009b370 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fedd00a3ca0 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedcc02b030 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedd00a3f20 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fede9590740 10 client.27008 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.509+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c3742a3cb0 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedcc02b030 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fedd0097580 con 0x55c37422e6d0 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 10 client.27008 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 10 client.27008 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 20 client.27008 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c3742a4130 0x55c3742a6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 4 client.27008 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:35.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c374199b50 con 0x55c3742a4130 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.510+0000 7feddea00640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c3742a4130 0x55c3742a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7feddea00640 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c3742a4130 0x55c3742a6520 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=456e61340da702d9 server_cookie=f4f29183ee98698c in_seq=0 out_seq=0 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fedc9600640 10 client.27008 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fedc9600640 1 -- 172.21.15.119:0/3774528846 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c374199b50 con 0x55c3742a4130 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fedc9600640 10 client.27008 handle_command_reply: tid=0 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 client.27008 shutdown 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 msgr2=0x55c37423db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fedd0099ae0 tx=0x7fedd0097c00 comp rx=0 tx=0).stop 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 shutdown_connections 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c3742a4130 0x55c3742a6520 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c374237c00 0x55c37429c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c37422f080 0x55c37423e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 --2- 172.21.15.119:0/3774528846 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c37422e6d0 0x55c37423db80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fedc9600640 0 client.27008 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 >> 172.21.15.119:0/3774528846 conn(0x55c374228da0 msgr2=0x55c374227620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 shutdown_connections 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 1 -- 172.21.15.119:0/3774528846 wait complete. 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.511+0000 7fede9590740 20 client.27008 trim_cache size 0 max 16384 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 msgr2=0x7fedc40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 0x7fedc40a4eb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fedd8005e30 tx=0x7fedd8005d80 comp rx=0 tx=0).stop 2024-03-28T18:49:35.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 msgr2=0x7fede019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd000e9f0 tx=0x7fedd000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 shutdown_connections 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fedc40a29f0 0x7fedc40a4eb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fede01a2df0 0x7fede01a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fede010cf80 0x7fede019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 --2- 172.21.15.119:0/2091500172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0107eb0 0x7fede019e530 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 >> 172.21.15.119:0/2091500172 conn(0x7fede0075be0 msgr2=0x7fede0077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 shutdown_connections 2024-03-28T18:49:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.512+0000 7fede6800640 1 -- 172.21.15.119:0/2091500172 wait complete. 2024-03-28T18:49:35.531 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:35.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:49:35.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 -- 172.21.15.119:0/2453384133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824107640 msgr2=0x7f1824107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 --2- 172.21.15.119:0/2453384133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824107640 0x7f1824107a20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f181c009cd0 tx=0x7f181c02c0e0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 -- 172.21.15.119:0/2453384133 shutdown_connections 2024-03-28T18:49:35.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 --2- 172.21.15.119:0/2453384133 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 --2- 172.21.15.119:0/2453384133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824101630 0x7f1824101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 --2- 172.21.15.119:0/2453384133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824107640 0x7f1824107a20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 -- 172.21.15.119:0/2453384133 >> 172.21.15.119:0/2453384133 conn(0x7f1824078aa0 msgr2=0x7f1824078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 -- 172.21.15.119:0/2453384133 shutdown_connections 2024-03-28T18:49:35.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.743+0000 7f182b400640 1 -- 172.21.15.119:0/2453384133 wait complete. 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 Processor -- start 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 -- start start 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824107640 0x7f182419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824111a10 con 0x7f1824107640 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824111890 con 0x7f1824101ff0 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.744+0000 7f182b400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824111b90 con 0x7f1824101630 2024-03-28T18:49:35.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58190/0 (socket says 172.21.15.119:58190) 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 -- 172.21.15.119:0/4186664153 learned_addr learned my addr 172.21.15.119:0/4186664153 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1823e00640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1829400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824107640 0x7f182419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 -- 172.21.15.119:0/4186664153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 msgr2=0x7f182419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 -- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824107640 msgr2=0x7f182419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824107640 0x7f182419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 -- 172.21.15.119:0/4186664153 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182419f140 con 0x7f1824101630 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1823e00640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.745+0000 7f1828a00640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f181c000f80 tx=0x7f181c032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.746+0000 7f1821600640 1 -- 172.21.15.119:0/4186664153 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c039040 con 0x7f1824101630 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.746+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182419f3d0 con 0x7f1824101630 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.746+0000 7f1821600640 1 -- 172.21.15.119:0/4186664153 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f181c044070 con 0x7f1824101630 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.746+0000 7f1821600640 1 -- 172.21.15.119:0/4186664153 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c0042a0 con 0x7f1824101630 2024-03-28T18:49:35.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.746+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18241a6cc0 con 0x7f1824101630 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 Processor -- start 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 -- start start 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be25296d0 0x559be2529ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be252a080 0x559be2532630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f1821600640 1 -- 172.21.15.119:0/4186664153 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f181c032590 con 0x7f1824101630 2024-03-28T18:49:35.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be248d2c0 con 0x559be25296d0 2024-03-28T18:49:35.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be2435c90 con 0x559be252a080 2024-03-28T18:49:35.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.749+0000 7f182e1f3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be2489360 con 0x559be2532c00 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1829400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be25296d0 0x559be2529ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1828a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be252a080 0x559be2532630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58202/0 (socket says 172.21.15.119:58202) 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 -- 172.21.15.119:0/2550053951 learned_addr learned my addr 172.21.15.119:0/2550053951 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 -- 172.21.15.119:0/2550053951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be252a080 msgr2=0x559be2532630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be252a080 0x559be2532630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 -- 172.21.15.119:0/2550053951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be25296d0 msgr2=0x559be2529ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be25296d0 0x559be2529ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 -- 172.21.15.119:0/2550053951 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559be245a080 con 0x559be2532c00 2024-03-28T18:49:35.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1823e00640 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18140114f0 tx=0x7f18140119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4285f4b3c71f450 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1819c00640 1 -- 172.21.15.119:0/2550053951 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181401e070 con 0x559be2532c00 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1819c00640 1 -- 172.21.15.119:0/2550053951 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18140035f0 con 0x559be2532c00 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.750+0000 7f1819c00640 1 -- 172.21.15.119:0/2550053951 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181400f7c0 con 0x559be2532c00 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f1821600640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 0x7f18080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 -- 172.21.15.119:0/2550053951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 msgr2=0x559be2535050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f18140114f0 tx=0x7f18140119b0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 -- 172.21.15.119:0/2550053951 shutdown_connections 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2535050 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f1821600640 1 -- 172.21.15.119:0/4186664153 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f181c0f12a0 con 0x7f1824101630 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be252a080 0x559be2532630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2550053951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be25296d0 0x559be2529ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 -- 172.21.15.119:0/2550053951 >> 172.21.15.119:0/2550053951 conn(0x559be2523da0 msgr2=0x559be2524180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f1829400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 0x7f18080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 -- 172.21.15.119:0/2550053951 shutdown_connections 2024-03-28T18:49:35.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.751+0000 7f182e1f3740 1 -- 172.21.15.119:0/2550053951 wait complete. 2024-03-28T18:49:35.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.752+0000 7f182e1f3740 1 -- start start 2024-03-28T18:49:35.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.752+0000 7f1829400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 0x7f18080a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f180c009cf0 tx=0x7f180c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.752+0000 7f182e1f3740 1 Processor -- start 2024-03-28T18:49:35.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be2532520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be252f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be2535b70 con 0x559be252a080 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be24490d0 con 0x559be25296d0 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559be2535cf0 con 0x559be2532c00 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f182e1f3740 10 client.0 _ll_register_callbacks cb 0x559be244de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1828a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be2532520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58214/0 (socket says 172.21.15.119:58214) 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 -- 172.21.15.119:0/2084852765 learned_addr learned my addr 172.21.15.119:0/2084852765 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1823e00640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be252f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 -- 172.21.15.119:0/2084852765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 msgr2=0x559be2532520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be2532520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 -- 172.21.15.119:0/2084852765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 msgr2=0x559be252f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be252f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1829400640 1 -- 172.21.15.119:0/2084852765 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559be21a13f0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1828a00640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be2532520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.753+0000 7f1823e00640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be252f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1829400640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f180c004600 tx=0x7f180c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3594e46fadf012a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c01f630 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f180c01f7d0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18140035f0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f180c01f9d0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f182e1f3740 10 client.26998 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559be226b7d0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18140035f0 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.754+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f180c027070 con 0x559be2532c00 2024-03-28T18:49:35.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 10 client.26998 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 10 client.26998 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 20 client.26998 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be2542ae0 0x559be2544ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 4 client.26998 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559be249a800 con 0x559be2542ae0 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f1828a00640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be2542ae0 0x559be2544ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f1828a00640 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be2542ae0 0x559be2544ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3af6f774ec05269a server_cookie=c05704fda0b87f8f in_seq=0 out_seq=0 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.755+0000 7f1812000640 10 client.26998 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.756+0000 7f1812000640 1 -- 172.21.15.119:0/2084852765 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559be249a800 con 0x559be2542ae0 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.756+0000 7f1812000640 10 client.26998 handle_command_reply: tid=0 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.756+0000 7f182e1f3740 1 client.26998 shutdown 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 msgr2=0x559be252fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f180c004600 tx=0x7f180c004730 comp rx=0 tx=0).stop 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 shutdown_connections 2024-03-28T18:49:35.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be2542ae0 0x559be2544ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be252fe50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be252f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f1812000640 0 client.26998 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 --2- 172.21.15.119:0/2084852765 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be2532520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 >> 172.21.15.119:0/2084852765 conn(0x559be2523da0 msgr2=0x559be2525af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 shutdown_connections 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 1 -- 172.21.15.119:0/2084852765 wait complete. 2024-03-28T18:49:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.757+0000 7f182e1f3740 20 client.26998 trim_cache size 0 max 16384 2024-03-28T18:49:35.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.770+0000 7f182e1f3740 1 Processor -- start 2024-03-28T18:49:35.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 -- start start 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be2532520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241af570 con 0x559be252a080 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241b0ef0 con 0x559be2532c00 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f182e1f3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241afbd0 con 0x559be25296d0 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1828a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1829400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1828a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58222/0 (socket says 172.21.15.119:58222) 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1828a00640 1 -- 172.21.15.119:0/3588397230 learned_addr learned my addr 172.21.15.119:0/3588397230 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1829400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48838/0 (socket says 172.21.15.119:48838) 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.771+0000 7f1823e00640 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be2532520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 -- 172.21.15.119:0/3588397230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 msgr2=0x559be2531fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 -- 172.21.15.119:0/3588397230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 msgr2=0x559be2532520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be2532520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 -- 172.21.15.119:0/3588397230 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559be2524f40 con 0x559be2532c00 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1828a00640 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1829400640 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f180c019040 tx=0x7f180c0046b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42d38eed934c244c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1819200640 1 -- 172.21.15.119:0/3588397230 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c009a30 con 0x559be2532c00 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1819200640 1 -- 172.21.15.119:0/3588397230 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f180c01d3f0 con 0x559be2532c00 2024-03-28T18:49:35.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.772+0000 7f1819200640 1 -- 172.21.15.119:0/3588397230 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c01da90 con 0x559be2532c00 2024-03-28T18:49:35.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- 172.21.15.119:0/3588397230 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 msgr2=0x559be252cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f180c019040 tx=0x7f180c0046b0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- 172.21.15.119:0/3588397230 shutdown_connections 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be2532c00 0x559be252cb30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be2532520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 --2- 172.21.15.119:0/3588397230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be25296d0 0x559be2531fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- 172.21.15.119:0/3588397230 >> 172.21.15.119:0/3588397230 conn(0x559be2523da0 msgr2=0x559be2525a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- 172.21.15.119:0/3588397230 shutdown_connections 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- 172.21.15.119:0/3588397230 wait complete. 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 -- start start 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.773+0000 7f182e1f3740 1 Processor -- start 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be253ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2538b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241b0890 con 0x559be252a080 2024-03-28T18:49:35.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241b0230 con 0x559be25296d0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241ad730 con 0x559be2532c00 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f182e1f3740 10 client.0 _ll_register_callbacks cb 0x7f180c009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1829400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2538b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48848/0 (socket says 172.21.15.119:48848) 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1823e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be253ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 -- 172.21.15.119:0/1822521057 learned_addr learned my addr 172.21.15.119:0/1822521057 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 -- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 msgr2=0x559be2538b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2538b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 -- 172.21.15.119:0/1822521057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 msgr2=0x559be253ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.774+0000 7f1828a00640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be253ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1828a00640 1 -- 172.21.15.119:0/1822521057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559be2539200 con 0x559be25296d0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1829400640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2538b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1828a00640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ba360 tx=0x7f181c0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d651f1826249038d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1812000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c0bd340 con 0x559be25296d0 2024-03-28T18:49:35.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f181c0bbce0 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f180c01d3f0 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f181c0098e0 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f182e1f3740 10 client.27026 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559be259ee20 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.775+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f180c01d3f0 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f181c03c030 con 0x559be25296d0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 10 client.27026 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 10 client.27026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 20 client.27026 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be259f520 0x559be25a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 4 client.27026 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559be2494b50 con 0x559be259f520 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f1823e00640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be259f520 0x559be25a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.776+0000 7f1823e00640 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be259f520 0x559be25a1940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d5da4d5286caf0e server_cookie=66b8c47bbdf5afcf in_seq=0 out_seq=0 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f1812000640 10 client.27026 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:35.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f1812000640 1 -- 172.21.15.119:0/1822521057 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559be2494b50 con 0x559be259f520 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f1812000640 10 client.27026 handle_command_reply: tid=0 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 client.27026 shutdown 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 msgr2=0x559be253a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f181c0ba360 tx=0x7f181c0095e0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 shutdown_connections 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x559be259f520 0x559be25a1940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559be2532c00 0x559be2538b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559be252a080 0x559be253ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 --2- 172.21.15.119:0/1822521057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559be25296d0 0x559be253a5e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 >> 172.21.15.119:0/1822521057 conn(0x559be2523da0 msgr2=0x559be2525110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f1812000640 0 client.27026 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 shutdown_connections 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 1 -- 172.21.15.119:0/1822521057 wait complete. 2024-03-28T18:49:35.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.777+0000 7f182e1f3740 20 client.27026 trim_cache size 0 max 16384 2024-03-28T18:49:35.782 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 msgr2=0x7f18080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 0x7f18080a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f180c009cf0 tx=0x7f180c005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 msgr2=0x7f182419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f181c000f80 tx=0x7f181c032460 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 shutdown_connections 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f18080a2970 0x7f18080a4e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1824107640 0x7f182419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1824101ff0 0x7f182419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 --2- 172.21.15.119:0/4186664153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1824101630 0x7f182419a150 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 >> 172.21.15.119:0/4186664153 conn(0x7f1824078aa0 msgr2=0x7f182410cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 shutdown_connections 2024-03-28T18:49:35.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:35.778+0000 7f182b400640 1 -- 172.21.15.119:0/4186664153 wait complete. 2024-03-28T18:49:35.797 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:49:35.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 -- 172.21.15.119:0/1970744130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101ef0 msgr2=0x7f9cdc10e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1970744130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc10e3b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009cd0 tx=0x7f9cd402c0e0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 -- 172.21.15.119:0/1970744130 shutdown_connections 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1970744130 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc10e3b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1970744130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101530 0x7f9cdc1019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.007+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1970744130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 0x7f9cdc1079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.008+0000 7f9ce2400640 1 -- 172.21.15.119:0/1970744130 >> 172.21.15.119:0/1970744130 conn(0x7f9cdc0fd000 msgr2=0x7f9cdc0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.008+0000 7f9ce2400640 1 -- 172.21.15.119:0/1970744130 shutdown_connections 2024-03-28T18:49:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.008+0000 7f9ce2400640 1 -- 172.21.15.119:0/1970744130 wait complete. 2024-03-28T18:49:36.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.008+0000 7f9ce2400640 1 Processor -- start 2024-03-28T18:49:36.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 -- start start 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 0x7f9cdc074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1118c0 con 0x7f9cdc1075f0 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc111740 con 0x7f9cdc101530 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce2400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc111a40 con 0x7f9cdc101ef0 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce1000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce1000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48862/0 (socket says 172.21.15.119:48862) 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58242/0 (socket says 172.21.15.119:58242) 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce1000640 1 -- 172.21.15.119:0/1672865103 learned_addr learned my addr 172.21.15.119:0/1672865103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.009+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 0x7f9cdc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 -- 172.21.15.119:0/1672865103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 msgr2=0x7f9cdc073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 -- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 msgr2=0x7f9cdc074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 0x7f9cdc074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 -- 172.21.15.119:0/1672865103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc1a6260 con 0x7f9cdc101530 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce1000640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e9f0 tx=0x7f9ccc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9cd9600640 1 -- 172.21.15.119:0/1672865103 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00cd80 con 0x7f9cdc101530 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9cd9600640 1 -- 172.21.15.119:0/1672865103 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc010910 con 0x7f9cdc101530 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9cd9600640 1 -- 172.21.15.119:0/1672865103 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc018990 con 0x7f9cdc101530 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc1a6550 con 0x7f9cdc101530 2024-03-28T18:49:36.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.010+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc1a6af0 con 0x7f9cdc101530 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.013+0000 7f9cd9600640 1 -- 172.21.15.119:0/1672865103 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9ccc010ac0 con 0x7f9cdc101530 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 Processor -- start 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 -- start start 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41f74230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41ecc2c0 con 0x561c41f69260 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41e74c90 con 0x561c41f71de0 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41ec8360 con 0x561c41f688b0 2024-03-28T18:49:36.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58248/0 (socket says 172.21.15.119:58248) 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce1a00640 1 -- 172.21.15.119:0/750604144 learned_addr learned my addr 172.21.15.119:0/750604144 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9ce1000640 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.014+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41f74230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 -- 172.21.15.119:0/750604144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 msgr2=0x561c41f74230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41f74230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 -- 172.21.15.119:0/750604144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 msgr2=0x561c41f71810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 -- 172.21.15.119:0/750604144 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c41e99080 con 0x561c41f688b0 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4004af0 tx=0x7f9cd4005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6094dcce1806e8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9cd9600640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 0x7f9cc00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9cd2000640 1 -- 172.21.15.119:0/750604144 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4043070 con 0x561c41f688b0 2024-03-28T18:49:36.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9cd9600640 1 -- 172.21.15.119:0/1672865103 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9ccc0cf150 con 0x7f9cdc101530 2024-03-28T18:49:36.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9cd2000640 1 -- 172.21.15.119:0/750604144 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cd4032c70 con 0x561c41f688b0 2024-03-28T18:49:36.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9cd2000640 1 -- 172.21.15.119:0/750604144 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4039040 con 0x561c41f688b0 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.015+0000 7f9ce1000640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 0x7f9cc00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 -- 172.21.15.119:0/750604144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 msgr2=0x561c41f68c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce1000640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 0x7f9cc00a4eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x561c41f68380 tx=0x7f9ccc000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4004af0 tx=0x7f9cd4005f80 comp rx=0 tx=0).stop 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 -- 172.21.15.119:0/750604144 shutdown_connections 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41f74230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/750604144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f68c90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 -- 172.21.15.119:0/750604144 >> 172.21.15.119:0/750604144 conn(0x561c41f62f80 msgr2=0x561c41f63360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 -- 172.21.15.119:0/750604144 shutdown_connections 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.016+0000 7f9ce67b3740 1 -- 172.21.15.119:0/750604144 wait complete. 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.017+0000 7f9ce67b3740 1 -- start start 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.017+0000 7f9ce67b3740 1 Processor -- start 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.017+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f71750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.017+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 0x561c41f6f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41f74cf0 con 0x561c41f71de0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41e880d0 con 0x561c41f69260 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c41f74e70 con 0x561c41f688b0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce67b3740 10 client.0 _ll_register_callbacks cb 0x561c41e8ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48868/0 (socket says 172.21.15.119:48868) 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 -- 172.21.15.119:0/2722148600 learned_addr learned my addr 172.21.15.119:0/2722148600 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f71750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 0x561c41f6f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 -- 172.21.15.119:0/2722148600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 msgr2=0x561c41f71750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f71750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 -- 172.21.15.119:0/2722148600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 msgr2=0x561c41f6f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 0x561c41f6f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 -- 172.21.15.119:0/2722148600 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c41be03f0 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 0x561c41f6f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4012700 tx=0x7f9cc4012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6526ff183204c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.018+0000 7f9ccaa00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400f920 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cd4032c70 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9cc400fac0 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cc400fcc0 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ce67b3740 10 client.27038 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561c41caa7d0 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cd4032c70 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9cc401c420 con 0x561c41f69260 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ce67b3740 10 client.27038 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.019+0000 7f9ce67b3740 10 client.27038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce67b3740 20 client.27038 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41f81cf0 0x561c41f84110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce67b3740 4 client.27038 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:36.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c41ed9800 con 0x561c41f81cf0 2024-03-28T18:49:36.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41f81cf0 0x561c41f84110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41f81cf0 0x561c41f84110 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485ecc3dedec14c1 server_cookie=49364fd3f48ee7a in_seq=0 out_seq=0 2024-03-28T18:49:36.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.020+0000 7f9ccaa00640 10 client.27038 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.021+0000 7f9ccaa00640 1 -- 172.21.15.119:0/2722148600 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561c41ed9800 con 0x561c41f81cf0 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.021+0000 7f9ccaa00640 10 client.27038 handle_command_reply: tid=0 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 client.27038 shutdown 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 msgr2=0x561c41f6eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4012700 tx=0x7f9cc4012bc0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 shutdown_connections 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41f81cf0 0x561c41f84110 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f71de0 0x561c41f6f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f69260 0x561c41f6eae0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/2722148600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f71750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 >> 172.21.15.119:0/2722148600 conn(0x561c41f62f80 msgr2=0x561c41f64cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ccaa00640 0 client.27038 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:36.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 shutdown_connections 2024-03-28T18:49:36.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 1 -- 172.21.15.119:0/2722148600 wait complete. 2024-03-28T18:49:36.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.022+0000 7f9ce67b3740 20 client.27038 trim_cache size 0 max 16384 2024-03-28T18:49:36.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 Processor -- start 2024-03-28T18:49:36.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 -- start start 2024-03-28T18:49:36.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f71de0 0x561c41f71750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1af360 con 0x561c41f69260 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1b0ce0 con 0x561c41f688b0 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1af9c0 con 0x561c41f71de0 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce1a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35276/0 (socket says 172.21.15.119:35276) 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce1a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48870/0 (socket says 172.21.15.119:48870) 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9cdbe00640 1 -- 172.21.15.119:0/1404669240 learned_addr learned my addr 172.21.15.119:0/1404669240 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:36.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.036+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f71de0 0x561c41f71750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 -- 172.21.15.119:0/1404669240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f71de0 msgr2=0x561c41f71750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f71de0 0x561c41f71750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 -- 172.21.15.119:0/1404669240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 msgr2=0x561c41f623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 -- 172.21.15.119:0/1404669240 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c41f6f460 con 0x561c41f69260 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9cc401c8c0 tx=0x7f9cc400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dd83422a346ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cd1600640 1 -- 172.21.15.119:0/1404669240 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4009bc0 con 0x561c41f69260 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cd1600640 1 -- 172.21.15.119:0/1404669240 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cc401c420 con 0x561c41f69260 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.037+0000 7f9cd1600640 1 -- 172.21.15.119:0/1404669240 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40052a0 con 0x561c41f69260 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 -- 172.21.15.119:0/1404669240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 msgr2=0x561c41f71210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9cc401c8c0 tx=0x7f9cc400f790 comp rx=0 tx=0).stop 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 -- 172.21.15.119:0/1404669240 shutdown_connections 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f71de0 0x561c41f71750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f71210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/1404669240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f688b0 0x561c41f623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 -- 172.21.15.119:0/1404669240 >> 172.21.15.119:0/1404669240 conn(0x561c41f62f80 msgr2=0x561c41f64b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 -- 172.21.15.119:0/1404669240 shutdown_connections 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.038+0000 7f9ce67b3740 1 -- 172.21.15.119:0/1404669240 wait complete. 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 -- start start 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 Processor -- start 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f788f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41fdd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1b0680 con 0x561c41f69260 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1b0020 con 0x561c41f71de0 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1ad520 con 0x561c41f688b0 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.039+0000 7f9ce67b3740 10 client.0 _ll_register_callbacks cb 0x7f9cc4005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35286/0 (socket says 172.21.15.119:35286) 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9cdbe00640 1 -- 172.21.15.119:0/311339899 learned_addr learned my addr 172.21.15.119:0/311339899 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41fdd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 -- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 msgr2=0x561c41fdd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41fdd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 -- 172.21.15.119:0/311339899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 msgr2=0x561c41f78e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 -- 172.21.15.119:0/311339899 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c41fddb00 con 0x561c41f688b0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce1a00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f788f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009620 tx=0x7f9cd4041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e90c49c92752f049 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ccaa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4043040 con 0x561c41f688b0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cc401c420 con 0x561c41f688b0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.040+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9cd4032590 con 0x561c41f688b0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cd4039040 con 0x561c41f688b0 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 10 client.27022 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561c41fded30 con 0x561c41f688b0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cc401c420 con 0x561c41f688b0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9cd405d470 con 0x561c41f688b0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 10 client.27022 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 10 client.27022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 20 client.27022 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41fdefb0 0x561c41fe13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 4 client.27022 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c41ed3b50 con 0x561c41fdefb0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.041+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41fdefb0 0x561c41fe13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9cdbe00640 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41fdefb0 0x561c41fe13a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdc9f2553367f29 server_cookie=1d0013dcec303448 in_seq=0 out_seq=0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ccaa00640 10 client.27022 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ccaa00640 1 -- 172.21.15.119:0/311339899 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c41ed3b50 con 0x561c41fdefb0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ccaa00640 10 client.27022 handle_command_reply: tid=0 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 client.27022 shutdown 2024-03-28T18:49:36.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 msgr2=0x561c41f788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f788f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009620 tx=0x7f9cd4041040 comp rx=0 tx=0).stop 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 shutdown_connections 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561c41fdefb0 0x561c41fe13a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561c41f71de0 0x561c41fdd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561c41f69260 0x561c41f78e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 --2- 172.21.15.119:0/311339899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561c41f688b0 0x561c41f788f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 >> 172.21.15.119:0/311339899 conn(0x561c41f62f80 msgr2=0x561c41f69640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ccaa00640 0 client.27022 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.042+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 shutdown_connections 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce67b3740 1 -- 172.21.15.119:0/311339899 wait complete. 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce67b3740 20 client.27022 trim_cache size 0 max 16384 2024-03-28T18:49:36.047 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 msgr2=0x7f9cc00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 0x7f9cc00a4eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x561c41f68380 tx=0x7f9ccc000bc0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 msgr2=0x7f9cdc0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e9f0 tx=0x7f9ccc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 shutdown_connections 2024-03-28T18:49:36.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9cc00a29f0 0x7f9cc00a4eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9cdc1075f0 0x7f9cdc074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cdc101ef0 0x7f9cdc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 --2- 172.21.15.119:0/1672865103 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9cdc101530 0x7f9cdc0739a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.043+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 >> 172.21.15.119:0/1672865103 conn(0x7f9cdc0fd000 msgr2=0x7f9cdc0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.044+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 shutdown_connections 2024-03-28T18:49:36.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:36.044+0000 7f9ce2400640 1 -- 172.21.15.119:0/1672865103 wait complete. 2024-03-28T18:49:36.058 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:35 smithi119 ceph-mon[31813]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:36.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:35 smithi196 ceph-mon[36300]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:36.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:35 smithi131 ceph-mon[36065]: pgmap v169: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:37.060 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:49:37.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:49:37.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.291+0000 7fc493200640 1 -- 172.21.15.119:0/1949369553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101530 msgr2=0x7fc48c1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.291+0000 7fc493200640 1 --2- 172.21.15.119:0/1949369553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101530 0x7fc48c1019b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009930 tx=0x7fc47c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 -- 172.21.15.119:0/1949369553 shutdown_connections 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 --2- 172.21.15.119:0/1949369553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c101ef0 0x7fc48c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 --2- 172.21.15.119:0/1949369553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101530 0x7fc48c1019b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 --2- 172.21.15.119:0/1949369553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c1075f0 0x7fc48c1079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 -- 172.21.15.119:0/1949369553 >> 172.21.15.119:0/1949369553 conn(0x7fc48c0fd000 msgr2=0x7fc48c0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 -- 172.21.15.119:0/1949369553 shutdown_connections 2024-03-28T18:49:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.292+0000 7fc493200640 1 -- 172.21.15.119:0/1949369553 wait complete. 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 Processor -- start 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 -- start start 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 0x7fc48c0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c1075f0 0x7fc48c074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c111920 con 0x7fc48c101530 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1117a0 con 0x7fc48c1075f0 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc493200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c111aa0 con 0x7fc48c101ef0 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc491400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc491e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 0x7fc48c0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc491400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58286/0 (socket says 172.21.15.119:58286) 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.293+0000 7fc491400640 1 -- 172.21.15.119:0/1895187919 learned_addr learned my addr 172.21.15.119:0/1895187919 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc492800640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c1075f0 0x7fc48c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 -- 172.21.15.119:0/1895187919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c1075f0 msgr2=0x7fc48c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c1075f0 0x7fc48c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 -- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 msgr2=0x7fc48c0739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 0x7fc48c0739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 -- 172.21.15.119:0/1895187919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c1a62c0 con 0x7fc48c101ef0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491e00640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 0x7fc48c0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc491400640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009900 tx=0x7fc47c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc486a00640 1 -- 172.21.15.119:0/1895187919 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c043020 con 0x7fc48c101ef0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc486a00640 1 -- 172.21.15.119:0/1895187919 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc47c0054e0 con 0x7fc48c101ef0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc486a00640 1 -- 172.21.15.119:0/1895187919 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c003bb0 con 0x7fc48c101ef0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.294+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c1a6550 con 0x7fc48c101ef0 2024-03-28T18:49:37.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.295+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c1a6a50 con 0x7fc48c101ef0 2024-03-28T18:49:37.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 Processor -- start 2024-03-28T18:49:37.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 -- start start 2024-03-28T18:49:37.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc486a00640 1 -- 172.21.15.119:0/1895187919 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc47c046020 con 0x7fc48c101ef0 2024-03-28T18:49:37.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f4249250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f424bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f41a4eb0 con 0x5632f4249790 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f414da60 con 0x5632f4240bb0 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc497586740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f41a0f50 con 0x5632f4240290 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc492800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc492800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58294/0 (socket says 172.21.15.119:58294) 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc492800640 1 -- 172.21.15.119:0/872617243 learned_addr learned my addr 172.21.15.119:0/872617243 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc491400640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f424bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.298+0000 7fc491e00640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f4249250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 -- 172.21.15.119:0/872617243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 msgr2=0x5632f4249250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f4249250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 -- 172.21.15.119:0/872617243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 msgr2=0x5632f424bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f424bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 -- 172.21.15.119:0/872617243 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632f3f817e0 con 0x5632f4240290 2024-03-28T18:49:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc491400640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f424bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc492800640 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc488009fd0 tx=0x7fc4880118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4266c927cadc3f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc47b000640 1 -- 172.21.15.119:0/872617243 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488004d20 con 0x5632f4240290 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc47b000640 1 -- 172.21.15.119:0/872617243 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc488004050 con 0x5632f4240290 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc47b000640 1 -- 172.21.15.119:0/872617243 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488005220 con 0x5632f4240290 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.299+0000 7fc486a00640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 0x7fc4740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc486a00640 1 -- 172.21.15.119:0/1895187919 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc47c0f0e20 con 0x7fc48c101ef0 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc491e00640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 0x7fc4740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 -- 172.21.15.119:0/872617243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 msgr2=0x5632f4240670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc488009fd0 tx=0x7fc4880118d0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 -- 172.21.15.119:0/872617243 shutdown_connections 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f424bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f4249250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 --2- 172.21.15.119:0/872617243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f4240670 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 -- 172.21.15.119:0/872617243 >> 172.21.15.119:0/872617243 conn(0x5632f423a9c0 msgr2=0x5632f423ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 -- 172.21.15.119:0/872617243 shutdown_connections 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc497586740 1 -- 172.21.15.119:0/872617243 wait complete. 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.300+0000 7fc491e00640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 0x7fc4740a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5632f423fd60 tx=0x7fc480005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 -- start start 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 Processor -- start 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f4246460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42469a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f424c580 con 0x5632f4249790 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f4160eb0 con 0x5632f4240290 2024-03-28T18:49:37.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.301+0000 7fc497586740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632f424c700 con 0x5632f4240bb0 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc497586740 10 client.0 _ll_register_callbacks cb 0x5632f4155150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48912/0 (socket says 172.21.15.119:48912) 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f4246460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 -- 172.21.15.119:0/2683299514 learned_addr learned my addr 172.21.15.119:0/2683299514 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc492800640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 -- 172.21.15.119:0/2683299514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 msgr2=0x5632f4246460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f4246460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 -- 172.21.15.119:0/2683299514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 msgr2=0x5632f42469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 -- 172.21.15.119:0/2683299514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632f3fe1f70 con 0x5632f4240290 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491400640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f4246460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:37.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc491e00640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc48000fb90 tx=0x7fc4800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4939b23cb0e3ccb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.302+0000 7fc46f400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480005020 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc488004050 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc480018030 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4800051c0 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc497586740 10 client.27056 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632f3e79170 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc488004050 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.303+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc48001e790 con 0x5632f4240290 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 10 client.27056 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 10 client.27056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:37.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 20 client.27056 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f4259850 0x5632f425bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 4 client.27056 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632f41b23f0 con 0x5632f4259850 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc491400640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f4259850 0x5632f425bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc491400640 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f4259850 0x5632f425bc40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a19d27ca22ee6b0 server_cookie=3ba9d4eba70d10b8 in_seq=0 out_seq=0 2024-03-28T18:49:37.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.304+0000 7fc46f400640 10 client.27056 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc46f400640 1 -- 172.21.15.119:0/2683299514 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5632f41b23f0 con 0x5632f4259850 2024-03-28T18:49:37.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc46f400640 10 client.27056 handle_command_reply: tid=0 2024-03-28T18:49:37.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 client.27056 shutdown 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 msgr2=0x5632f42490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc48000fb90 tx=0x7fc4800046c0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 shutdown_connections 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f4259850 0x5632f425bc40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f4246460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 --2- 172.21.15.119:0/2683299514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240290 0x5632f42490d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc46f400640 0 client.27056 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 >> 172.21.15.119:0/2683299514 conn(0x5632f423a9c0 msgr2=0x5632f423c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 shutdown_connections 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 1 -- 172.21.15.119:0/2683299514 wait complete. 2024-03-28T18:49:37.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.305+0000 7fc497586740 20 client.27056 trim_cache size 0 max 16384 2024-03-28T18:49:37.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.323+0000 7fc497586740 1 Processor -- start 2024-03-28T18:49:37.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.323+0000 7fc497586740 1 -- start start 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f42490d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 0x5632f423c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1af390 con 0x5632f4240290 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1b0d10 con 0x5632f4249790 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc497586740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1af9f0 con 0x5632f4240bb0 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35328/0 (socket says 172.21.15.119:35328) 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 -- 172.21.15.119:0/1143553989 learned_addr learned my addr 172.21.15.119:0/1143553989 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491400640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f42490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc492800640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 0x5632f423c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 -- 172.21.15.119:0/1143553989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 msgr2=0x5632f42490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f42490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 -- 172.21.15.119:0/1143553989 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 msgr2=0x5632f423c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 0x5632f423c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491e00640 1 -- 172.21.15.119:0/1143553989 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632f424c5d0 con 0x5632f4240290 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc491400640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f42490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.324+0000 7fc492800640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 0x5632f423c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:37.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.325+0000 7fc491e00640 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc480000c00 tx=0x7fc48000fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1f0ee00399ad92 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.325+0000 7fc47a600640 1 -- 172.21.15.119:0/1143553989 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480004ea0 con 0x5632f4240290 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.325+0000 7fc47a600640 1 -- 172.21.15.119:0/1143553989 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc48001e700 con 0x5632f4240290 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.325+0000 7fc47a600640 1 -- 172.21.15.119:0/1143553989 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480004730 con 0x5632f4240290 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- 172.21.15.119:0/1143553989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 msgr2=0x5632f4248b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc480000c00 tx=0x7fc48000fa20 comp rx=0 tx=0).stop 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- 172.21.15.119:0/1143553989 shutdown_connections 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4249790 0x5632f423c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240bb0 0x5632f42490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 --2- 172.21.15.119:0/1143553989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4240290 0x5632f4248b90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- 172.21.15.119:0/1143553989 >> 172.21.15.119:0/1143553989 conn(0x5632f423a9c0 msgr2=0x5632f423bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- 172.21.15.119:0/1143553989 shutdown_connections 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- 172.21.15.119:0/1143553989 wait complete. 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 -- start start 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.326+0000 7fc497586740 1 Processor -- start 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1b06b0 con 0x5632f4249790 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1b0050 con 0x5632f4240bb0 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1ad550 con 0x5632f4240290 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc497586740 10 client.0 _ll_register_callbacks cb 0x7fc48000f4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:37.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc492800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58322/0 (socket says 172.21.15.119:58322) 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48930/0 (socket says 172.21.15.119:48930) 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 -- 172.21.15.119:0/3709318395 learned_addr learned my addr 172.21.15.119:0/3709318395 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 -- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 msgr2=0x5632f424e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 -- 172.21.15.119:0/3709318395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 msgr2=0x5632f42b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491e00640 1 -- 172.21.15.119:0/3709318395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632f42b5620 con 0x5632f4240290 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.327+0000 7fc491400640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc491e00640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc480009700 tx=0x7fc48001ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2a1f62203c856d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc46f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480040070 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc48001e700 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fc48001e040 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc48001e240 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc497586740 10 client.27040 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632f42b6800 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.328+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc48001e700 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc4800298f0 con 0x5632f4240290 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 10 client.27040 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 10 client.27040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 20 client.27040 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f42b6db0 0x5632f42b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 4 client.27040 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:49:37.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5632f41ac740 con 0x5632f42b6db0 2024-03-28T18:49:37.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc491400640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f42b6db0 0x5632f42b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc491400640 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f42b6db0 0x5632f42b91a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeaa7f46bc6da4d3 server_cookie=4899f102c2d5434a in_seq=0 out_seq=0 2024-03-28T18:49:37.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.329+0000 7fc46f400640 10 client.27040 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc46f400640 1 -- 172.21.15.119:0/3709318395 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5632f41ac740 con 0x5632f42b6db0 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc46f400640 10 client.27040 handle_command_reply: tid=0 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 client.27040 shutdown 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 msgr2=0x5632f424e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc480009700 tx=0x7fc48001ec40 comp rx=0 tx=0).stop 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 shutdown_connections 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5632f42b6db0 0x5632f42b91a0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5632f4249790 0x5632f42b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5632f4240bb0 0x5632f424e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 --2- 172.21.15.119:0/3709318395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5632f4240290 0x5632f424e150 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc46f400640 0 client.27040 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 >> 172.21.15.119:0/3709318395 conn(0x5632f423a9c0 msgr2=0x5632f423c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 shutdown_connections 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 1 -- 172.21.15.119:0/3709318395 wait complete. 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.331+0000 7fc497586740 20 client.27040 trim_cache size 0 max 16384 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:49:37.336 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "cba81602-c6e1-4118-96c6-934d0d1c5729", 2024-03-28T18:49:37.337 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:49:37.337 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:37.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 msgr2=0x7fc4740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 0x7fc4740a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5632f423fd60 tx=0x7fc480005f90 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 msgr2=0x7fc48c073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009900 tx=0x7fc47c004360 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 shutdown_connections 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4740a2970 0x7fc4740a4e30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc48c1075f0 0x7fc48c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc48c101ef0 0x7fc48c073ee0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 --2- 172.21.15.119:0/1895187919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc48c101530 0x7fc48c0739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 >> 172.21.15.119:0/1895187919 conn(0x7fc48c0fd000 msgr2=0x7fc48c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 shutdown_connections 2024-03-28T18:49:37.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.332+0000 7fc493200640 1 -- 172.21.15.119:0/1895187919 wait complete. 2024-03-28T18:49:37.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:49:37.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:49:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 -- 172.21.15.119:0/3826426412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8103840 msgr2=0x7f4ef8103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 --2- 172.21.15.119:0/3826426412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8103840 0x7f4ef8103cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009930 tx=0x7f4ee002bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 -- 172.21.15.119:0/3826426412 shutdown_connections 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 --2- 172.21.15.119:0/3826426412 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8104200 0x7f4ef8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 --2- 172.21.15.119:0/3826426412 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8103840 0x7f4ef8103cc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 --2- 172.21.15.119:0/3826426412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8109850 0x7f4ef8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 -- 172.21.15.119:0/3826426412 >> 172.21.15.119:0/3826426412 conn(0x7f4ef80ff2f0 msgr2=0x7f4ef8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 -- 172.21.15.119:0/3826426412 shutdown_connections 2024-03-28T18:49:37.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.553+0000 7f4efce00640 1 -- 172.21.15.119:0/3826426412 wait complete. 2024-03-28T18:49:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.554+0000 7f4efce00640 1 Processor -- start 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.554+0000 7f4efce00640 1 -- start start 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8104200 0x7f4ef819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8109850 0x7f4ef819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef8113c70 con 0x7f4ef8104200 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef8113af0 con 0x7f4ef8103840 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4efce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef8113df0 con 0x7f4ef8109850 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48936/0 (socket says 172.21.15.119:48936) 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 -- 172.21.15.119:0/3496165647 learned_addr learned my addr 172.21.15.119:0/3496165647 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4eed400640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8104200 0x7f4ef819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef6800640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8109850 0x7f4ef819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 -- 172.21.15.119:0/3496165647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8109850 msgr2=0x7f4ef819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8109850 0x7f4ef819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 -- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8104200 msgr2=0x7f4ef819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8104200 0x7f4ef819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.555+0000 7f4ef5e00640 1 -- 172.21.15.119:0/3496165647 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef819f180 con 0x7f4ef8103840 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4ef5e00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800ebc0 tx=0x7f4ee800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4eef400640 1 -- 172.21.15.119:0/3496165647 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8010630 con 0x7f4ef8103840 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef819f470 con 0x7f4ef8103840 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4eef400640 1 -- 172.21.15.119:0/3496165647 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee8010cb0 con 0x7f4ef8103840 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4eef400640 1 -- 172.21.15.119:0/3496165647 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8018bd0 con 0x7f4ef8103840 2024-03-28T18:49:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.556+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef81a6d00 con 0x7f4ef8103840 2024-03-28T18:49:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.559+0000 7f4eef400640 1 -- 172.21.15.119:0/3496165647 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4ee80107d0 con 0x7f4ef8103840 2024-03-28T18:49:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.559+0000 7f4effbd5740 1 Processor -- start 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.559+0000 7f4effbd5740 1 -- start start 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ed680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b898646250 con 0x55b8986eb290 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8985eea30 con 0x55b8986e1f20 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4effbd5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8986422f0 con 0x55b8986e2c00 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58352/0 (socket says 172.21.15.119:58352) 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 -- 172.21.15.119:0/1359949119 learned_addr learned my addr 172.21.15.119:0/1359949119 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4eed400640 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ed680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 -- 172.21.15.119:0/1359949119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 msgr2=0x55b8986e23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986e23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 -- 172.21.15.119:0/1359949119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 msgr2=0x55b8986ed680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ed680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4ef6800640 1 -- 172.21.15.119:0/1359949119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8985d9690 con 0x55b8986e2c00 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.560+0000 7f4eed400640 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ed680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4eef400640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 0x7f4edc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009ca0 tx=0x7f4ef0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf7c03252fb39aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4eef400640 1 -- 172.21.15.119:0/3496165647 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4ee80cf4c0 con 0x7f4ef8103840 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4ef5e00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 0x7f4edc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4ee7400640 1 -- 172.21.15.119:0/1359949119 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef001e070 con 0x55b8986e2c00 2024-03-28T18:49:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4ee7400640 1 -- 172.21.15.119:0/1359949119 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ef0009e50 con 0x55b8986e2c00 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.561+0000 7f4ee7400640 1 -- 172.21.15.119:0/1359949119 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000eac0 con 0x55b8986e2c00 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- 172.21.15.119:0/1359949119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 msgr2=0x55b8986e1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4ef5e00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 0x7f4edc0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55b8986e3150 tx=0x7f4ee80d2f70 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009ca0 tx=0x7f4ef0011810 comp rx=0 tx=0).stop 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- 172.21.15.119:0/1359949119 shutdown_connections 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ed680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986e23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1359949119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e1950 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- 172.21.15.119:0/1359949119 >> 172.21.15.119:0/1359949119 conn(0x55b8986a9200 msgr2=0x55b8986dc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- 172.21.15.119:0/1359949119 shutdown_connections 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- 172.21.15.119:0/1359949119 wait complete. 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.562+0000 7f4effbd5740 1 -- start start 2024-03-28T18:49:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 Processor -- start 2024-03-28T18:49:37.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 0x55b8986ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b8986e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8986ee100 con 0x55b8986e1f20 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b898601e80 con 0x55b8986eb290 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8986ee280 con 0x55b8986e2c00 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4effbd5740 10 client.0 _ll_register_callbacks cb 0x55b898614130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 0x55b8986ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4eed400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 0x55b8986ea910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35380/0 (socket says 172.21.15.119:35380) 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4eed400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58358/0 (socket says 172.21.15.119:58358) 2024-03-28T18:49:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.563+0000 7f4ef6800640 1 -- 172.21.15.119:0/1449107710 learned_addr learned my addr 172.21.15.119:0/1449107710 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b8986e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 -- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 msgr2=0x55b8986e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b8986e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 -- 172.21.15.119:0/1449107710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 msgr2=0x55b8986ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 0x55b8986ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 -- 172.21.15.119:0/1449107710 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8984247d0 con 0x55b8986e2c00 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4eed400640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0036040 tx=0x7f4ee0005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637579ce5d4b4dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4edbe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee002cb50 con 0x55b8986e2c00 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4ee00046c0 con 0x55b8986e2c00 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.564+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ef0009e50 con 0x55b8986e2c00 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee002ccf0 con 0x55b8986e2c00 2024-03-28T18:49:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 10 client.27052 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b89835a3f0 con 0x55b8986e2c00 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ef0009e50 con 0x55b8986e2c00 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4ee00035f0 con 0x55b8986e2c00 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 10 client.27052 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 10 client.27052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 20 client.27052 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b8986fb190 0x55b8986fd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 4 client.27052 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b898653790 con 0x55b8986fb190 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.565+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b8986fb190 0x55b8986fd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.566+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b8986fb190 0x55b8986fd580 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240a378b2560d772 server_cookie=297f75b3b8138e40 in_seq=0 out_seq=0 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.566+0000 7f4edbe00640 10 client.27052 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4edbe00640 1 -- 172.21.15.119:0/1449107710 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b898653790 con 0x55b8986fb190 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4edbe00640 10 client.27052 handle_command_reply: tid=0 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 client.27052 shutdown 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 msgr2=0x55b8986e7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0036040 tx=0x7f4ee0005db0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 shutdown_connections 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b8986fb190 0x55b8986fd580 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b8986e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986e7ca0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1449107710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e1f20 0x55b8986ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4edbe00640 0 client.27052 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 >> 172.21.15.119:0/1449107710 conn(0x55b8986a9200 msgr2=0x55b8986de6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 shutdown_connections 2024-03-28T18:49:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.567+0000 7f4effbd5740 1 -- 172.21.15.119:0/1449107710 wait complete. 2024-03-28T18:49:37.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.568+0000 7f4effbd5740 20 client.27052 trim_cache size 0 max 16384 2024-03-28T18:49:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 Processor -- start 2024-03-28T18:49:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 -- start start 2024-03-28T18:49:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ec680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81af580 con 0x55b8986eb290 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81b0f00 con 0x55b8986e1f20 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4effbd5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81afbe0 con 0x55b8986e2c00 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4eed400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4eed400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58364/0 (socket says 172.21.15.119:58364) 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4eed400640 1 -- 172.21.15.119:0/2908232847 learned_addr learned my addr 172.21.15.119:0/2908232847 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.585+0000 7f4ef6800640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4ef6800640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ec680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 -- 172.21.15.119:0/2908232847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 msgr2=0x55b8986ea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986ea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 -- 172.21.15.119:0/2908232847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 msgr2=0x55b8986ec680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ec680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 -- 172.21.15.119:0/2908232847 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b89831a0f0 con 0x55b8986e2c00 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4ef6800640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ec680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4eed400640 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0005c00 tx=0x7f4ee002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc2cbe054a2e8c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4ee6a00640 1 -- 172.21.15.119:0/2908232847 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0000e60 con 0x55b8986e2c00 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4ee6a00640 1 -- 172.21.15.119:0/2908232847 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee0009520 con 0x55b8986e2c00 2024-03-28T18:49:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.586+0000 7f4ee6a00640 1 -- 172.21.15.119:0/2908232847 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00050e0 con 0x55b8986e2c00 2024-03-28T18:49:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- 172.21.15.119:0/2908232847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 msgr2=0x55b8986ea910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0005c00 tx=0x7f4ee002bc30 comp rx=0 tx=0).stop 2024-03-28T18:49:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- 172.21.15.119:0/2908232847 shutdown_connections 2024-03-28T18:49:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986eb290 0x55b8986ec680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e2c00 0x55b8986ea910 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 --2- 172.21.15.119:0/2908232847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986e1f20 0x55b8986ea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- 172.21.15.119:0/2908232847 >> 172.21.15.119:0/2908232847 conn(0x55b8986a9200 msgr2=0x55b8986dc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- 172.21.15.119:0/2908232847 shutdown_connections 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- 172.21.15.119:0/2908232847 wait complete. 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.587+0000 7f4effbd5740 1 -- start start 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 Processor -- start 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e2c00 0x55b8986ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b898755970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81b08a0 con 0x55b8986e2c00 2024-03-28T18:49:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81b0240 con 0x55b8986eb290 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81ad740 con 0x55b8986e1f20 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4effbd5740 10 client.0 _ll_register_callbacks cb 0x7f4ef0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4ef6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58378/0 (socket says 172.21.15.119:58378) 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4ef6800640 1 -- 172.21.15.119:0/1856291591 learned_addr learned my addr 172.21.15.119:0/1856291591 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4eed400640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e2c00 0x55b8986ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.588+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b898755970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 -- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 msgr2=0x55b898755970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b898755970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 -- 172.21.15.119:0/1856291591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e2c00 msgr2=0x55b8986ef2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e2c00 0x55b8986ef2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 -- 172.21.15.119:0/1856291591 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b898755eb0 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4ef6800640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009ca0 tx=0x7f4ef0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeca634a97539a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4edbe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef001e040 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ee0009520 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f4ef00052e0 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ef0009e50 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4effbd5740 10 client.27058 fetch_fsmap learned FSMap version 28 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.589+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b898756ff0 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ee0009520 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4ef0024070 con 0x55b8986e1f20 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 10 client.27058 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 10 client.27058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:49:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 20 client.27058 populate_metadata read hostname 'smithi119' 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b898757460 0x55b898759880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 4 client.27058 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b89864dae0 con 0x55b898757460 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4eed400640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b898757460 0x55b898759880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4eed400640 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b898757460 0x55b898759880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44356bb70d911476 server_cookie=8382b087e41eb01b in_seq=0 out_seq=0 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.590+0000 7f4edbe00640 10 client.27058 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4edbe00640 1 -- 172.21.15.119:0/1856291591 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b89864dae0 con 0x55b898757460 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4edbe00640 10 client.27058 handle_command_reply: tid=0 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 client.27058 shutdown 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 msgr2=0x55b8986eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009ca0 tx=0x7f4ef0011810 comp rx=0 tx=0).stop 2024-03-28T18:49:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 shutdown_connections 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55b898757460 0x55b898759880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55b8986eb290 0x55b898755970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b8986e2c00 0x55b8986ef2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 --2- 172.21.15.119:0/1856291591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b8986e1f20 0x55b8986eed70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4edbe00640 0 client.27058 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 >> 172.21.15.119:0/1856291591 conn(0x55b8986a9200 msgr2=0x55b8986db200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 shutdown_connections 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 1 -- 172.21.15.119:0/1856291591 wait complete. 2024-03-28T18:49:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.591+0000 7f4effbd5740 20 client.27058 trim_cache size 0 max 16384 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "cba81602-c6e1-4118-96c6-934d0d1c5729": { 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "cba81602-c6e1-4118-96c6-934d0d1c5729", 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:49:37.599 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 msgr2=0x7f4edc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 0x7f4edc0a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55b8986e3150 tx=0x7f4ee80d2f70 comp rx=0 tx=0).stop 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 msgr2=0x7f4ef819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800ebc0 tx=0x7f4ee800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 shutdown_connections 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4edc0a2970 0x7f4edc0a4e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8109850 0x7f4ef819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ef8104200 0x7f4ef819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 --2- 172.21.15.119:0/3496165647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4ef8103840 0x7f4ef819a190 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:49:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 >> 172.21.15.119:0/3496165647 conn(0x7f4ef80ff2f0 msgr2=0x7f4ef80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:49:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 shutdown_connections 2024-03-28T18:49:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:49:37.592+0000 7f4efce00640 1 -- 172.21.15.119:0/3496165647 wait complete. 2024-03-28T18:49:37.608 INFO:tasks.cephfs.filesystem:scrub status for tag:cba81602-c6e1-4118-96c6-934d0d1c5729 - {'path': '/', 'tag': 'cba81602-c6e1-4118-96c6-934d0d1c5729', 'options': 'recursive,force'} 2024-03-28T18:49:37.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:37 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:49:37.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:37 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:37.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:37 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:49:37.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:37 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:37.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:37 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:49:37.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:37 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:37.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:37 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:49:37.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:37 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:37 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:49:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:37 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:37 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:49:37.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:37 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:49:38.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:38 smithi196 ceph-mon[36300]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:38.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:38 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:49:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:38 smithi131 ceph-mon[36065]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:38.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:38 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:49:38.721 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:38 smithi119 ceph-mon[31813]: pgmap v170: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:38.721 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:38 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:49:40.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:40 smithi196 ceph-mon[36300]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:40 smithi131 ceph-mon[36065]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:40 smithi119 ceph-mon[31813]: pgmap v171: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:42 smithi196 ceph-mon[36300]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:42 smithi131 ceph-mon[36065]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:42 smithi119 ceph-mon[31813]: pgmap v172: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:44.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:44 smithi196 ceph-mon[36300]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:44 smithi131 ceph-mon[36065]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:44 smithi119 ceph-mon[31813]: pgmap v173: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:46.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:46 smithi196 ceph-mon[36300]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:46.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:46 smithi131 ceph-mon[36065]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:46 smithi119 ceph-mon[31813]: pgmap v174: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:48 smithi196 ceph-mon[36300]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:48.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:48 smithi131 ceph-mon[36065]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:48 smithi119 ceph-mon[31813]: pgmap v175: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:49:50.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:50 smithi196 ceph-mon[36300]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:50.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:50 smithi131 ceph-mon[36065]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:50 smithi119 ceph-mon[31813]: pgmap v176: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:49:52.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:52 smithi131 ceph-mon[36065]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:52 smithi119 ceph-mon[31813]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:52 smithi196 ceph-mon[36300]: pgmap v177: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:54.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:54 smithi131 ceph-mon[36065]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:54 smithi119 ceph-mon[31813]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:54 smithi196 ceph-mon[36300]: pgmap v178: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:56 smithi119 ceph-mon[31813]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:56 smithi196 ceph-mon[36300]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:56 smithi131 ceph-mon[36065]: pgmap v179: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:49:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:49:58 smithi119 ceph-mon[31813]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:49:58 smithi196 ceph-mon[36300]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:49:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:49:58 smithi131 ceph-mon[36065]: pgmap v180: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:00 smithi119 ceph-mon[31813]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:00 smithi119 ceph-mon[31813]: overall HEALTH_OK 2024-03-28T18:50:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:00 smithi196 ceph-mon[36300]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:00.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:00 smithi196 ceph-mon[36300]: overall HEALTH_OK 2024-03-28T18:50:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:00 smithi131 ceph-mon[36065]: pgmap v181: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:00 smithi131 ceph-mon[36065]: overall HEALTH_OK 2024-03-28T18:50:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:02 smithi119 ceph-mon[31813]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:02 smithi196 ceph-mon[36300]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:02.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:02 smithi131 ceph-mon[36065]: pgmap v182: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:04.763 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:04 smithi196 ceph-mon[36300]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:04 smithi119 ceph-mon[31813]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:04 smithi131 ceph-mon[36065]: pgmap v183: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:06 smithi119 ceph-mon[31813]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:06 smithi196 ceph-mon[36300]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:06 smithi131 ceph-mon[36065]: pgmap v184: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:07.610 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:50:07.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.829+0000 7f0072e00640 1 -- 172.21.15.119:0/3735070850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 msgr2=0x7f006c111e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.829+0000 7f0072e00640 1 --2- 172.21.15.119:0/3735070850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c111e80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0058009930 tx=0x7f005802bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 -- 172.21.15.119:0/3735070850 shutdown_connections 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 --2- 172.21.15.119:0/3735070850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 0x7f006c1147b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 --2- 172.21.15.119:0/3735070850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c111e80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 --2- 172.21.15.119:0/3735070850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 -- 172.21.15.119:0/3735070850 >> 172.21.15.119:0/3735070850 conn(0x7f006c0ff1b0 msgr2=0x7f006c1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 -- 172.21.15.119:0/3735070850 shutdown_connections 2024-03-28T18:50:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.830+0000 7f0072e00640 1 -- 172.21.15.119:0/3735070850 wait complete. 2024-03-28T18:50:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.831+0000 7f0072e00640 1 Processor -- start 2024-03-28T18:50:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.831+0000 7f0072e00640 1 -- start start 2024-03-28T18:50:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.831+0000 7f0072e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c1a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 0x7f006c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c117f70 con 0x7f006c1123c0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c117df0 con 0x7f006c075bd0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1180f0 con 0x7f006c0764f0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38166/0 (socket says 172.21.15.119:38166) 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 -- 172.21.15.119:0/2031179084 learned_addr learned my addr 172.21.15.119:0/2031179084 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0072400640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 0x7f006c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071000640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c1a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 -- 172.21.15.119:0/2031179084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 msgr2=0x7f006c1a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c1a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 -- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 msgr2=0x7f006c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 0x7f006c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.832+0000 7f0071a00640 1 -- 172.21.15.119:0/2031179084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c19ed50 con 0x7f006c075bd0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f0071a00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f006000ea60 tx=0x7f006000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f006a000640 1 -- 172.21.15.119:0/2031179084 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f006000cd20 con 0x7f006c075bd0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f006a000640 1 -- 172.21.15.119:0/2031179084 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00600035f0 con 0x7f006c075bd0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c19f040 con 0x7f006c075bd0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f006a000640 1 -- 172.21.15.119:0/2031179084 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060010690 con 0x7f006c075bd0 2024-03-28T18:50:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.833+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c1ab2e0 con 0x7f006c075bd0 2024-03-28T18:50:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.837+0000 7f0077239740 1 Processor -- start 2024-03-28T18:50:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.837+0000 7f0077239740 1 -- start start 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f006a000640 1 -- 172.21.15.119:0/2031179084 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f00600037a0 con 0x7f006c075bd0 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf306850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf25f250 con 0x55c6cf304460 2024-03-28T18:50:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf207a30 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0077239740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf25b2f0 con 0x55c6cf2fb0f0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0072400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0072400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38176/0 (socket says 172.21.15.119:38176) 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0072400640 1 -- 172.21.15.119:0/1392225647 learned_addr learned my addr 172.21.15.119:0/1392225647 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0071a00640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0071000640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf306850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0072400640 1 -- 172.21.15.119:0/1392225647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 msgr2=0x55c6cf2fb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.838+0000 7f0072400640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f0072400640 1 -- 172.21.15.119:0/1392225647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 msgr2=0x55c6cf306850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f0072400640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf306850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f0072400640 1 -- 172.21.15.119:0/1392225647 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6cf1f2690 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f0071000640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf306850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f0072400640 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0064009f40 tx=0x7f00640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=182f359f2616792 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f005ea00640 1 -- 172.21.15.119:0/1392225647 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064004d20 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f005ea00640 1 -- 172.21.15.119:0/1392225647 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0064004050 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.839+0000 7f005ea00640 1 -- 172.21.15.119:0/1392225647 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064005280 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f006a000640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 0x7f00540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 -- 172.21.15.119:0/1392225647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 msgr2=0x55c6cf2fab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0064009f40 tx=0x7f00640118d0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f006a000640 1 -- 172.21.15.119:0/2031179084 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f00600cf1e0 con 0x7f006c075bd0 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0071a00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 0x7f00540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 -- 172.21.15.119:0/1392225647 shutdown_connections 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf306850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 --2- 172.21.15.119:0/1392225647 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fab20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 -- 172.21.15.119:0/1392225647 >> 172.21.15.119:0/1392225647 conn(0x55c6cf2c2200 msgr2=0x55c6cf2f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 -- 172.21.15.119:0/1392225647 shutdown_connections 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.840+0000 7f0077239740 1 -- 172.21.15.119:0/1392225647 wait complete. 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 -- start start 2024-03-28T18:50:07.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0071a00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 0x7f00540a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f00600cc980 tx=0x7f00600d2d30 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 Processor -- start 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf304460 0x55c6cf301450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf3072d0 con 0x55c6cf2fb0f0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf21ae80 con 0x55c6cf304460 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6cf307450 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.841+0000 7f0077239740 10 client.0 _ll_register_callbacks cb 0x55c6cf22d130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0072400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40058/0 (socket says 172.21.15.119:40058) 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 -- 172.21.15.119:0/29142167 learned_addr learned my addr 172.21.15.119:0/29142167 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0072400640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf304460 0x55c6cf301450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 -- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf304460 msgr2=0x55c6cf301450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf304460 0x55c6cf301450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 -- 172.21.15.119:0/29142167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf2fb0f0 msgr2=0x55c6cf303b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 -- 172.21.15.119:0/29142167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6cf03d7d0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0071000640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c00 tx=0x7f0058005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d70df9bd50d9965c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.842+0000 7f0053400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005802ca90 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f005802cc30 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0064004050 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00580046c0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 10 client.27070 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c6cef733f0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0064004050 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f00580035f0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 10 client.27070 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.843+0000 7f0077239740 20 client.27070 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3143a0 0x55c6cf316790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0077239740 4 client.27070 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6cf26c790 con 0x55c6cf3143a0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0072400640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3143a0 0x55c6cf316790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0053400640 10 client.27070 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.844+0000 7f0072400640 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3143a0 0x55c6cf316790 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf97c3e94b4f745 server_cookie=313c85f13409c077 in_seq=0 out_seq=0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.845+0000 7f0053400640 1 -- 172.21.15.119:0/29142167 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6cf26c790 con 0x55c6cf3143a0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.845+0000 7f0053400640 10 client.27070 handle_command_reply: tid=0 2024-03-28T18:50:07.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.845+0000 7f0077239740 1 client.27070 shutdown 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 msgr2=0x55c6cf300f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c00 tx=0x7f0058005db0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 shutdown_connections 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3143a0 0x55c6cf316790 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf304460 0x55c6cf301450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf300f10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 --2- 172.21.15.119:0/29142167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0053400640 0 client.27070 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 >> 172.21.15.119:0/29142167 conn(0x55c6cf2c2200 msgr2=0x55c6cf2f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 shutdown_connections 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 1 -- 172.21.15.119:0/29142167 wait complete. 2024-03-28T18:50:07.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.846+0000 7f0077239740 20 client.27070 trim_cache size 0 max 16384 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 Processor -- start 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 -- start start 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf305850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b3870 con 0x55c6cf304460 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b51f0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.863+0000 7f0077239740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b3ed0 con 0x55c6cf2fb0f0 2024-03-28T18:50:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38182/0 (socket says 172.21.15.119:38182) 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 -- 172.21.15.119:0/1422981242 learned_addr learned my addr 172.21.15.119:0/1422981242 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0072400640 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0072400640 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf305850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 -- 172.21.15.119:0/1422981242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 msgr2=0x55c6cf303640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 -- 172.21.15.119:0/1422981242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 msgr2=0x55c6cf305850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf305850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.864+0000 7f0071000640 1 -- 172.21.15.119:0/1422981242 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6cef330f0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f0071000640 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f005802c170 tx=0x7f005802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b94053ccf77ceb78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f005e000640 1 -- 172.21.15.119:0/1422981242 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058056070 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f005e000640 1 -- 172.21.15.119:0/1422981242 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0058009520 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f005e000640 1 -- 172.21.15.119:0/1422981242 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058004ee0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f0077239740 1 -- 172.21.15.119:0/1422981242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 msgr2=0x55c6cf303b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.865+0000 7f0077239740 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f005802c170 tx=0x7f005802bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 -- 172.21.15.119:0/1422981242 shutdown_connections 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf305850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf303b80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 --2- 172.21.15.119:0/1422981242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf303640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 -- 172.21.15.119:0/1422981242 >> 172.21.15.119:0/1422981242 conn(0x55c6cf2c2200 msgr2=0x55c6cf2f5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 -- 172.21.15.119:0/1422981242 shutdown_connections 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 -- 172.21.15.119:0/1422981242 wait complete. 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 -- start start 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.866+0000 7f0077239740 1 Processor -- start 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf2fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b4b90 con 0x55c6cf304460 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b4530 con 0x55c6cf2fb0f0 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1b1a30 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0077239740 10 client.0 _ll_register_callbacks cb 0x7f0060005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0072400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40090/0 (socket says 172.21.15.119:40090) 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 -- 172.21.15.119:0/2851188071 learned_addr learned my addr 172.21.15.119:0/2851188071 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0072400640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf2fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 -- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fb0f0 msgr2=0x55c6cf2fe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fe190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 -- 172.21.15.119:0/2851188071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 msgr2=0x55c6cf2fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf2fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.867+0000 7f0071000640 1 -- 172.21.15.119:0/2851188071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6cf304840 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0071000640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f00580092a0 tx=0x7f0058003830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d17734328c39dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0053400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058056070 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f0058004a60 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0058009520 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0058004ee0 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0077239740 10 client.27076 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.868+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c6cf36fe70 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0058009520 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f005804a860 con 0x55c6cf2fbdd0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 10 client.27076 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 10 client.27076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 20 client.27076 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3703d0 0x55c6cf3727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 4 client.27076 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c6cf266ae0 con 0x55c6cf3703d0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.869+0000 7f0072400640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3703d0 0x55c6cf3727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0072400640 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3703d0 0x55c6cf3727c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcc15fc683ec1847 server_cookie=cb7e2d9563d00ef0 in_seq=0 out_seq=0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0053400640 10 client.27076 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0053400640 1 -- 172.21.15.119:0/2851188071 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c6cf266ae0 con 0x55c6cf3703d0 2024-03-28T18:50:07.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0053400640 10 client.27076 handle_command_reply: tid=0 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 client.27076 shutdown 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 msgr2=0x55c6cf2fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f00580092a0 tx=0x7f0058003830 comp rx=0 tx=0).stop 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 shutdown_connections 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55c6cf3703d0 0x55c6cf3727c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c6cf304460 0x55c6cf2fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cf2fbdd0 0x55c6cf2fe6d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 --2- 172.21.15.119:0/2851188071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c6cf2fb0f0 0x55c6cf2fe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0053400640 0 client.27076 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 >> 172.21.15.119:0/2851188071 conn(0x55c6cf2c2200 msgr2=0x55c6cf2ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 shutdown_connections 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 1 -- 172.21.15.119:0/2851188071 wait complete. 2024-03-28T18:50:07.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.870+0000 7f0077239740 20 client.27076 trim_cache size 0 max 16384 2024-03-28T18:50:07.876 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:07.876 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:50:07.876 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:50:07.876 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:07.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 msgr2=0x7f00540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 0x7f00540a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f00600cc980 tx=0x7f00600d2d30 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 msgr2=0x7f006c1a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f006000ea60 tx=0x7f006000ee60 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 shutdown_connections 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f00540a2970 0x7f00540a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006c1123c0 0x7f006c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f006c0764f0 0x7f006c1a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 --2- 172.21.15.119:0/2031179084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f006c075bd0 0x7f006c1a4420 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 >> 172.21.15.119:0/2031179084 conn(0x7f006c0ff1b0 msgr2=0x7f006c1015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 shutdown_connections 2024-03-28T18:50:07.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:07.871+0000 7f0072e00640 1 -- 172.21.15.119:0/2031179084 wait complete. 2024-03-28T18:50:07.889 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:50:07.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 -- 172.21.15.119:0/1986671430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe96010dd50 msgr2=0x7fe960110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 --2- 172.21.15.119:0/1986671430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe96010dd50 0x7fe960110140 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe958009ca0 tx=0x7fe95802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 -- 172.21.15.119:0/1986671430 shutdown_connections 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 --2- 172.21.15.119:0/1986671430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe96010dd50 0x7fe960110140 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 --2- 172.21.15.119:0/1986671430 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe960068d40 0x7fe9600691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 --2- 172.21.15.119:0/1986671430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9601023b0 0x7fe960068800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 -- 172.21.15.119:0/1986671430 >> 172.21.15.119:0/1986671430 conn(0x7fe960075fb0 msgr2=0x7fe9600763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.100+0000 7fe964a00640 1 -- 172.21.15.119:0/1986671430 shutdown_connections 2024-03-28T18:50:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.101+0000 7fe964a00640 1 -- 172.21.15.119:0/1986671430 wait complete. 2024-03-28T18:50:08.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 Processor -- start 2024-03-28T18:50:08.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 -- start start 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 0x7fe96019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe96010dd50 0x7fe96019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960113aa0 con 0x7fe96010dd50 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960113920 con 0x7fe9601023b0 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe964a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960113c20 con 0x7fe960068d40 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe95e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe95e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40118/0 (socket says 172.21.15.119:40118) 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe95e800640 1 -- 172.21.15.119:0/1442207385 learned_addr learned my addr 172.21.15.119:0/1442207385 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.102+0000 7fe95f200640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe96010dd50 0x7fe96019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe955e00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 0x7fe96019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 -- 172.21.15.119:0/1442207385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 msgr2=0x7fe96019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 0x7fe96019a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 -- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe96010dd50 msgr2=0x7fe96019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe96010dd50 0x7fe96019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 -- 172.21.15.119:0/1442207385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96019ef70 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe955e00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 0x7fe96019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe95e800640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe95000d950 tx=0x7fe95000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe950017030 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe950004020 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96019f260 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9500047a0 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.103+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9601a6af0 con 0x7fe960068d40 2024-03-28T18:50:08.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.105+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9600683b0 con 0x7fe960068d40 2024-03-28T18:50:08.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.107+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe9500041d0 con 0x7fe960068d40 2024-03-28T18:50:08.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.108+0000 7fe957e00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 0x7fe9440a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.109+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe9500cef10 con 0x7fe960068d40 2024-03-28T18:50:08.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.109+0000 7fe955e00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 0x7fe9440a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.109+0000 7fe955e00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 0x7fe9440a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9600767b0 tx=0x7fe948005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.112+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe95009bbb0 con 0x7fe960068d40 2024-03-28T18:50:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.357+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe96019b2d0 con 0x7fe960068d40 2024-03-28T18:50:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.358+0000 7fe957e00640 1 -- 172.21.15.119:0/1442207385 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fe95009b4b0 con 0x7fe960068d40 2024-03-28T18:50:08.359 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:50:08.359 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:50:08.359 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 msgr2=0x7fe9440a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 0x7fe9440a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe9600767b0 tx=0x7fe948005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 msgr2=0x7fe960199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe95000d950 tx=0x7fe95000de10 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 shutdown_connections 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9440a29f0 0x7fe9440a4eb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe96010dd50 0x7fe96019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9601023b0 0x7fe96019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 --2- 172.21.15.119:0/1442207385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe960068d40 0x7fe960199f80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.361+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 >> 172.21.15.119:0/1442207385 conn(0x7fe960075fb0 msgr2=0x7fe9600ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.362+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 shutdown_connections 2024-03-28T18:50:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.362+0000 7fe964a00640 1 -- 172.21.15.119:0/1442207385 wait complete. 2024-03-28T18:50:08.380 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:08.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:50:08.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 -- 172.21.15.119:0/871749965 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 msgr2=0x7f54b8103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 --2- 172.21.15.119:0/871749965 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b8103cc0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009900 tx=0x7f54a002bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 -- 172.21.15.119:0/871749965 shutdown_connections 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 --2- 172.21.15.119:0/871749965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 0x7f54b8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 --2- 172.21.15.119:0/871749965 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b8103cc0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 --2- 172.21.15.119:0/871749965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 -- 172.21.15.119:0/871749965 >> 172.21.15.119:0/871749965 conn(0x7f54b80ff310 msgr2=0x7f54b8101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 -- 172.21.15.119:0/871749965 shutdown_connections 2024-03-28T18:50:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.594+0000 7f54be800640 1 -- 172.21.15.119:0/871749965 wait complete. 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.595+0000 7f54be800640 1 Processor -- start 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 -- start start 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 0x7f54b819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b8113cb0 con 0x7f54b8104200 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b8113b30 con 0x7f54b8103840 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54be800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b8113e30 con 0x7f54b8109850 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54b7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54b7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38238/0 (socket says 172.21.15.119:38238) 2024-03-28T18:50:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54b7400640 1 -- 172.21.15.119:0/1810874405 learned_addr learned my addr 172.21.15.119:0/1810874405 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.596+0000 7f54b6a00640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 0x7f54b819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 -- 172.21.15.119:0/1810874405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 msgr2=0x7f54b819ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b819ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 -- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 msgr2=0x7f54b819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 0x7f54b819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 -- 172.21.15.119:0/1810874405 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b819f1c0 con 0x7f54b8103840 2024-03-28T18:50:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54b7400640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54afe00640 1 -- 172.21.15.119:0/1810874405 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a800cd80 con 0x7f54b8103840 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54afe00640 1 -- 172.21.15.119:0/1810874405 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a8005500 con 0x7f54b8103840 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b819f4b0 con 0x7f54b8103840 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.597+0000 7f54afe00640 1 -- 172.21.15.119:0/1810874405 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8010690 con 0x7f54b8103840 2024-03-28T18:50:08.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.598+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b81a6d40 con 0x7f54b8103840 2024-03-28T18:50:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 Processor -- start 2024-03-28T18:50:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54afe00640 1 -- 172.21.15.119:0/1810874405 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f54a80046c0 con 0x7f54b8103840 2024-03-28T18:50:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 -- start start 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963a9080 0x5612963b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963b1c00 0x5612963b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56129630c2c0 con 0x5612963b1c00 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612962b4c90 con 0x5612963a9080 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.602+0000 7f54c15c7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561296308360 con 0x5612963a86d0 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963a9080 0x5612963b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40136/0 (socket says 172.21.15.119:40136) 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 -- 172.21.15.119:0/1448931802 learned_addr learned my addr 172.21.15.119:0/1448931802 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b6a00640 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963b1c00 0x5612963b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 -- 172.21.15.119:0/1448931802 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963a9080 msgr2=0x5612963b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963a9080 0x5612963b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 -- 172.21.15.119:0/1448931802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963b1c00 msgr2=0x5612963b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963b1c00 0x5612963b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 -- 172.21.15.119:0/1448931802 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612962d9080 con 0x5612963a86d0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54b00107e0 tx=0x7f54b0010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b11647b69725e117 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54afe00640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 0x7f549c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.603+0000 7f54acc00640 1 -- 172.21.15.119:0/1448931802 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0003b70 con 0x5612963a86d0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54acc00640 1 -- 172.21.15.119:0/1448931802 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54b000ee10 con 0x5612963a86d0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54acc00640 1 -- 172.21.15.119:0/1448931802 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0011440 con 0x5612963a86d0 2024-03-28T18:50:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54afe00640 1 -- 172.21.15.119:0/1810874405 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f54a80cf200 con 0x7f54b8103840 2024-03-28T18:50:08.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54b7400640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 0x7f549c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54b7400640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 0x7f549c0a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5612963a81a0 tx=0x7f54a80d2cb0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54c15c7740 1 -- 172.21.15.119:0/1448931802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 msgr2=0x5612963a8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.604+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54b00107e0 tx=0x7f54b0010ca0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 -- 172.21.15.119:0/1448931802 shutdown_connections 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963b1c00 0x5612963b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963a9080 0x5612963b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1448931802 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963a8ab0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 -- 172.21.15.119:0/1448931802 >> 172.21.15.119:0/1448931802 conn(0x5612963a2da0 msgr2=0x5612963a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 -- 172.21.15.119:0/1448931802 shutdown_connections 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 -- 172.21.15.119:0/1448931802 wait complete. 2024-03-28T18:50:08.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 -- start start 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.605+0000 7f54c15c7740 1 Processor -- start 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612963b4b70 con 0x5612963a9080 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612962c80d0 con 0x5612963b1c00 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612963b4cf0 con 0x5612963a86d0 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54c15c7740 10 client.0 _ll_register_callbacks cb 0x5612962cce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40150/0 (socket says 172.21.15.119:40150) 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b7e00640 1 -- 172.21.15.119:0/3912767011 learned_addr learned my addr 172.21.15.119:0/3912767011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b7e00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 -- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 msgr2=0x5612963b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 -- 172.21.15.119:0/3912767011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 msgr2=0x5612963aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.606+0000 7f54b6a00640 1 -- 172.21.15.119:0/3912767011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612960203f0 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54b6a00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963ae860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c170 tx=0x7f54a0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9771690fcf8df583 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54a5600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a002cab0 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f54a002cc50 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54b000ee10 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a00046c0 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54c15c7740 10 client.16890 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.607+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5612960ea7d0 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54b000ee10 con 0x5612963a9080 2024-03-28T18:50:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f54a0005380 con 0x5612963a9080 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 10 client.16890 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 10 client.16890 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 20 client.16890 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5612963c1a40 0x5612963c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 4 client.16890 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561296319800 con 0x5612963c1a40 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.608+0000 7f54b7e00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5612963c1a40 0x5612963c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.609+0000 7f54b7e00640 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5612963c1a40 0x5612963c3e30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0421d1ce805556f server_cookie=8735a8f78afbd447 in_seq=0 out_seq=0 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.609+0000 7f54a5600640 10 client.16890 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.609+0000 7f54a5600640 1 -- 172.21.15.119:0/3912767011 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561296319800 con 0x5612963c1a40 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.609+0000 7f54a5600640 10 client.16890 handle_command_reply: tid=0 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.609+0000 7f54c15c7740 1 client.16890 shutdown 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 msgr2=0x5612963ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963ae860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c170 tx=0x7f54a0004610 comp rx=0 tx=0).stop 2024-03-28T18:50:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 shutdown_connections 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x5612963c1a40 0x5612963c3e30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963ae860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 --2- 172.21.15.119:0/3912767011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54a5600640 0 client.16890 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 >> 172.21.15.119:0/3912767011 conn(0x5612963a2da0 msgr2=0x5612963a4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 shutdown_connections 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 1 -- 172.21.15.119:0/3912767011 wait complete. 2024-03-28T18:50:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.610+0000 7f54c15c7740 20 client.16890 trim_cache size 0 max 16384 2024-03-28T18:50:08.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 Processor -- start 2024-03-28T18:50:08.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 -- start start 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 0x5612963b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81af540 con 0x5612963a86d0 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81b0ec0 con 0x5612963b1c00 2024-03-28T18:50:08.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54c15c7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81afba0 con 0x5612963a9080 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 0x5612963b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40166/0 (socket says 172.21.15.119:40166) 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b6a00640 1 -- 172.21.15.119:0/1619065565 learned_addr learned my addr 172.21.15.119:0/1619065565 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 0x5612963b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38926/0 (socket says 172.21.15.119:38926) 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.624+0000 7f54b7e00640 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 -- 172.21.15.119:0/1619065565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 msgr2=0x5612963a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 -- 172.21.15.119:0/1619065565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 msgr2=0x5612963b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 0x5612963b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 -- 172.21.15.119:0/1619065565 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612963aa470 con 0x5612963a9080 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54b6a00640 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c240 tx=0x7f54a0005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db29d607c2b461a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54a7e00640 1 -- 172.21.15.119:0/1619065565 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0004a60 con 0x5612963a9080 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54a7e00640 1 -- 172.21.15.119:0/1619065565 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a0009520 con 0x5612963a9080 2024-03-28T18:50:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.625+0000 7f54a7e00640 1 -- 172.21.15.119:0/1619065565 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0005270 con 0x5612963a9080 2024-03-28T18:50:08.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- 172.21.15.119:0/1619065565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 msgr2=0x5612963b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f54a002c240 tx=0x7f54a0005b50 comp rx=0 tx=0).stop 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- 172.21.15.119:0/1619065565 shutdown_connections 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612963a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a9080 0x5612963b14d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 --2- 172.21.15.119:0/1619065565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a86d0 0x5612963b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- 172.21.15.119:0/1619065565 >> 172.21.15.119:0/1619065565 conn(0x5612963a2da0 msgr2=0x5612963a40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- 172.21.15.119:0/1619065565 shutdown_connections 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- 172.21.15.119:0/1619065565 wait complete. 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.626+0000 7f54c15c7740 1 -- start start 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 Processor -- start 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612964162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81b0860 con 0x5612963a9080 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81b0200 con 0x5612963b1c00 2024-03-28T18:50:08.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b81ad700 con 0x5612963a86d0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54c15c7740 10 client.0 _ll_register_callbacks cb 0x7f54a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40170/0 (socket says 172.21.15.119:40170) 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54b7e00640 1 -- 172.21.15.119:0/2927184666 learned_addr learned my addr 172.21.15.119:0/2927184666 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.627+0000 7f54b7e00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612964162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 -- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 msgr2=0x5612963b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 -- 172.21.15.119:0/2927184666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 msgr2=0x5612964162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612964162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 -- 172.21.15.119:0/2927184666 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612964168d0 con 0x5612963a9080 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b7e00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612964162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54b6a00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963b6090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5612963a82e0 tx=0x7f54a002cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e78c8d69db0740f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54a5600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:50:08.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0004860 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f54a0004a00 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.628+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54a0009520 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a0004c20 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 10 client.16896 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56129641dc50 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54a0009520 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f54a0003b20 con 0x5612963a9080 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 10 client.16896 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 10 client.16896 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 20 client.16896 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56129641e200 0x561296420620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 4 client.16896 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561296313b50 con 0x56129641e200 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.629+0000 7f54b7e00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56129641e200 0x561296420620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54b7e00640 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56129641e200 0x561296420620 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=df0cf6511bce5d58 server_cookie=111f429df2206ee3 in_seq=0 out_seq=0 2024-03-28T18:50:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54a5600640 10 client.16896 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54a5600640 1 -- 172.21.15.119:0/2927184666 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561296313b50 con 0x56129641e200 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54a5600640 10 client.16896 handle_command_reply: tid=0 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 client.16896 shutdown 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 msgr2=0x5612963b6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963b6090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5612963a82e0 tx=0x7f54a002cae0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 shutdown_connections 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56129641e200 0x561296420620 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5612963b1c00 0x5612964162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5612963a9080 0x5612963b6090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54a5600640 0 client.16896 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 --2- 172.21.15.119:0/2927184666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5612963a86d0 0x5612963b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.630+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 >> 172.21.15.119:0/2927184666 conn(0x5612963a2da0 msgr2=0x5612963a9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 shutdown_connections 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54c15c7740 1 -- 172.21.15.119:0/2927184666 wait complete. 2024-03-28T18:50:08.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54c15c7740 20 client.16896 trim_cache size 0 max 16384 2024-03-28T18:50:08.635 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 msgr2=0x7f549c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 0x7f549c0a4eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5612963a81a0 tx=0x7f54a80d2cb0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 msgr2=0x7f54b819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e9f0 tx=0x7f54a800edf0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 shutdown_connections 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f549c0a29f0 0x7f549c0a4eb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54b8109850 0x7f54b819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54b8104200 0x7f54b819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 --2- 172.21.15.119:0/1810874405 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f54b8103840 0x7f54b819a1d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.631+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 >> 172.21.15.119:0/1810874405 conn(0x7f54b80ff310 msgr2=0x7f54b80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.632+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 shutdown_connections 2024-03-28T18:50:08.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.632+0000 7f54be800640 1 -- 172.21.15.119:0/1810874405 wait complete. 2024-03-28T18:50:08.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:08 smithi119 ceph-mon[31813]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:08.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:08 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1442207385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:08.651 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:08.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:50:08.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:08 smithi196 ceph-mon[36300]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:08.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:08 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1442207385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.863+0000 7f3fe8800640 1 -- 172.21.15.119:0/2521851521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe00764f0 msgr2=0x7f3fe0111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.863+0000 7f3fe8800640 1 --2- 172.21.15.119:0/2521851521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe00764f0 0x7f3fe0111fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009900 tx=0x7f3fd002bc40 comp rx=0 tx=0).stop 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 -- 172.21.15.119:0/2521851521 shutdown_connections 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 --2- 172.21.15.119:0/2521851521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe01124e0 0x7f3fe01148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 --2- 172.21.15.119:0/2521851521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe00764f0 0x7f3fe0111fa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 --2- 172.21.15.119:0/2521851521 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe0075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 -- 172.21.15.119:0/2521851521 >> 172.21.15.119:0/2521851521 conn(0x7f3fe00ff2f0 msgr2=0x7f3fe0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 -- 172.21.15.119:0/2521851521 shutdown_connections 2024-03-28T18:50:08.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 -- 172.21.15.119:0/2521851521 wait complete. 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.864+0000 7f3fe8800640 1 Processor -- start 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 -- start start 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 0x7f3fe019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01124e0 0x7f3fe01a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0118090 con 0x7f3fe01124e0 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0117f10 con 0x7f3fe0075bd0 2024-03-28T18:50:08.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe8800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0118210 con 0x7f3fe00764f0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 0x7f3fe019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38286/0 (socket says 172.21.15.119:38286) 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe5e00640 1 -- 172.21.15.119:0/1592533199 learned_addr learned my addr 172.21.15.119:0/1592533199 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.865+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01124e0 0x7f3fe01a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 -- 172.21.15.119:0/1592533199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 msgr2=0x7f3fe019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 0x7f3fe019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 -- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01124e0 msgr2=0x7f3fe01a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01124e0 0x7f3fe01a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 -- 172.21.15.119:0/1592533199 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe01a3580 con 0x7f3fe0075bd0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5400640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 0x7f3fe019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe5e00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800ebc0 tx=0x7f3fd800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fd6a00640 1 -- 172.21.15.119:0/1592533199 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8010430 con 0x7f3fe0075bd0 2024-03-28T18:50:08.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fd6a00640 1 -- 172.21.15.119:0/1592533199 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8010ab0 con 0x7f3fe0075bd0 2024-03-28T18:50:08.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe01a3870 con 0x7f3fe0075bd0 2024-03-28T18:50:08.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fd6a00640 1 -- 172.21.15.119:0/1592533199 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8018b30 con 0x7f3fe0075bd0 2024-03-28T18:50:08.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.866+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe01ab100 con 0x7f3fe0075bd0 2024-03-28T18:50:08.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.869+0000 7f3feb59e740 1 Processor -- start 2024-03-28T18:50:08.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 -- start start 2024-03-28T18:50:08.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fd6a00640 1 -- 172.21.15.119:0/1592533199 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3fd8018cd0 con 0x7f3fe0075bd0 2024-03-28T18:50:08.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341d7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 0x55ef341da050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef341322c0 con 0x55ef341cf080 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef340dac90 con 0x55ef341ce6d0 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3feb59e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef3412e360 con 0x55ef341d7c00 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fe5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341d7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 0x55ef341da050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38290/0 (socket says 172.21.15.119:38290) 2024-03-28T18:50:08.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.870+0000 7f3fe6800640 1 -- 172.21.15.119:0/1733564502 learned_addr learned my addr 172.21.15.119:0/1733564502 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 -- 172.21.15.119:0/1733564502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 msgr2=0x55ef341da050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 0x55ef341da050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 -- 172.21.15.119:0/1733564502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 msgr2=0x55ef341d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341d7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 -- 172.21.15.119:0/1733564502 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef340ff080 con 0x55ef341ce6d0 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe5400640 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 0x55ef341da050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009bb0 tx=0x7f3fdc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13af26edd18d60a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fcb000640 1 -- 172.21.15.119:0/1733564502 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc004230 con 0x55ef341ce6d0 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fcb000640 1 -- 172.21.15.119:0/1733564502 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fdc009e20 con 0x55ef341ce6d0 2024-03-28T18:50:08.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fcb000640 1 -- 172.21.15.119:0/1733564502 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0197e0 con 0x55ef341ce6d0 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fd6a00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 0x7f3fcc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.871+0000 7f3fd6a00640 1 -- 172.21.15.119:0/1592533199 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3fd80cf5d0 con 0x7f3fe0075bd0 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3fe5e00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 0x7f3fcc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 -- 172.21.15.119:0/1733564502 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 msgr2=0x55ef341ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009bb0 tx=0x7f3fdc011810 comp rx=0 tx=0).stop 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 -- 172.21.15.119:0/1733564502 shutdown_connections 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341d7c00 0x55ef341da050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341d7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1733564502 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341ceab0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 -- 172.21.15.119:0/1733564502 >> 172.21.15.119:0/1733564502 conn(0x55ef341c8da0 msgr2=0x55ef341c9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 -- 172.21.15.119:0/1733564502 shutdown_connections 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3fe5e00640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 0x7f3fcc0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55ef341ce1a0 tx=0x7f3fd80d50e0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.872+0000 7f3feb59e740 1 -- 172.21.15.119:0/1733564502 wait complete. 2024-03-28T18:50:08.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 -- start start 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 Processor -- start 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341d7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341d4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef341dab70 con 0x55ef341d7c00 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef340ee0d0 con 0x55ef341ce6d0 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef341dacf0 con 0x55ef341cf080 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3feb59e740 10 client.0 _ll_register_callbacks cb 0x55ef340f2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.873+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:40204/0 (socket says 172.21.15.119:40204) 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 -- 172.21.15.119:0/1220568895 learned_addr learned my addr 172.21.15.119:0/1220568895 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341d7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341d4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 -- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 msgr2=0x55ef341d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341d7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 -- 172.21.15.119:0/1220568895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 msgr2=0x55ef341d4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341d4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 -- 172.21.15.119:0/1220568895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef33e463f0 con 0x55ef341cf080 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341d4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fe5400640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009900 tx=0x7f3fd002c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad3a30e9d9dca085 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fbf400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004380 con 0x55ef341cf080 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fdc009e20 con 0x55ef341cf080 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.874+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3fd002c8c0 con 0x55ef341cf080 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fd002cac0 con 0x55ef341cf080 2024-03-28T18:50:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 10 client.27112 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ef33f107d0 con 0x55ef341cf080 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fdc009e20 con 0x55ef341cf080 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3fd00037f0 con 0x55ef341cf080 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 10 client.27112 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 10 client.27112 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 20 client.27112 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef341e7ad0 0x55ef341e9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 4 client.27112 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef3413f800 con 0x55ef341e7ad0 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.875+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef341e7ad0 0x55ef341e9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.876+0000 7f3fe6800640 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef341e7ad0 0x55ef341e9ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f0fd5a544e792342 server_cookie=670f782fb4fabe30 in_seq=0 out_seq=0 2024-03-28T18:50:08.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.876+0000 7f3fbf400640 10 client.27112 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:08.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3fbf400640 1 -- 172.21.15.119:0/1220568895 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ef3413f800 con 0x55ef341e7ad0 2024-03-28T18:50:08.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3fbf400640 10 client.27112 handle_command_reply: tid=0 2024-03-28T18:50:08.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 client.27112 shutdown 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 msgr2=0x55ef341d48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009900 tx=0x7f3fd002c810 comp rx=0 tx=0).stop 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 shutdown_connections 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef341e7ad0 0x55ef341e9ef0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341d4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341cf080 0x55ef341d48c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 --2- 172.21.15.119:0/1220568895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341ce6d0 0x55ef341d7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 >> 172.21.15.119:0/1220568895 conn(0x55ef341c8da0 msgr2=0x55ef341caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3fbf400640 0 client.27112 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 shutdown_connections 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 1 -- 172.21.15.119:0/1220568895 wait complete. 2024-03-28T18:50:08.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.877+0000 7f3feb59e740 20 client.27112 trim_cache size 0 max 16384 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.890+0000 7f3feb59e740 1 Processor -- start 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 -- start start 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341d6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b39d0 con 0x55ef341d7c00 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b5350 con 0x55ef341cf080 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3feb59e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b4030 con 0x55ef341ce6d0 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341d6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53546/0 (socket says 172.21.15.119:53546) 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341d6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58262/0 (socket says 172.21.15.119:58262) 2024-03-28T18:50:08.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 -- 172.21.15.119:0/788690823 learned_addr learned my addr 172.21.15.119:0/788690823 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe6800640 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 -- 172.21.15.119:0/788690823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 msgr2=0x55ef341d6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341d6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 -- 172.21.15.119:0/788690823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 msgr2=0x55ef341c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.891+0000 7f3fe5400640 1 -- 172.21.15.119:0/788690823 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef341d5240 con 0x55ef341cf080 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3fe5400640 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3fd002c250 tx=0x7f3fd0005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a2e772a7877b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3fca600640 1 -- 172.21.15.119:0/788690823 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0000e60 con 0x55ef341cf080 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3fca600640 1 -- 172.21.15.119:0/788690823 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fd002cc60 con 0x55ef341cf080 2024-03-28T18:50:08.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3fca600640 1 -- 172.21.15.119:0/788690823 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004f60 con 0x55ef341cf080 2024-03-28T18:50:08.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3feb59e740 1 -- 172.21.15.119:0/788690823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 msgr2=0x55ef341d7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.892+0000 7f3feb59e740 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3fd002c250 tx=0x7f3fd0005c00 comp rx=0 tx=0).stop 2024-03-28T18:50:08.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 -- 172.21.15.119:0/788690823 shutdown_connections 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341d7c00 0x55ef341c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341cf080 0x55ef341d7530 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 --2- 172.21.15.119:0/788690823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341d6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 -- 172.21.15.119:0/788690823 >> 172.21.15.119:0/788690823 conn(0x55ef341c8da0 msgr2=0x55ef341c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 -- 172.21.15.119:0/788690823 shutdown_connections 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 -- 172.21.15.119:0/788690823 wait complete. 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 -- start start 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.893+0000 7f3feb59e740 1 Processor -- start 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341dfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341d7c00 0x55ef341de770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b4cf0 con 0x55ef341cf080 2024-03-28T18:50:08.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b4690 con 0x55ef341d7c00 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01b1b90 con 0x55ef341ce6d0 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3feb59e740 10 client.0 _ll_register_callbacks cb 0x7f3fd0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341dfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341dfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58272/0 (socket says 172.21.15.119:58272) 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe6800640 1 -- 172.21.15.119:0/2439420452 learned_addr learned my addr 172.21.15.119:0/2439420452 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe6800640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341d7c00 0x55ef341de770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 -- 172.21.15.119:0/2439420452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 msgr2=0x55ef341dfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341dfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 -- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341d7c00 msgr2=0x55ef341de770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341d7c00 0x55ef341de770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.894+0000 7f3fe5400640 1 -- 172.21.15.119:0/2439420452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef341def70 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fe5400640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341de160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55ef341da840 tx=0x7f3fd0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10c2a40a7269f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fbf400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0003db0 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3fd0004180 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fd002cc60 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fd00043c0 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3feb59e740 10 client.16911 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ef342444a0 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.895+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fd002cc60 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3fd0034630 con 0x55ef341cf080 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 10 client.16911 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 10 client.16911 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 20 client.16911 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:08.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef34244aa0 0x55ef34246e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 4 client.16911 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef34139b50 con 0x55ef34244aa0 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3fe6800640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef34244aa0 0x55ef34246e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3fbf400640 10 client.16911 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.896+0000 7f3fe6800640 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef34244aa0 0x55ef34246e90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ac66601238c3b93b server_cookie=fa7bf7c9d7a5d176 in_seq=0 out_seq=0 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3fbf400640 1 -- 172.21.15.119:0/2439420452 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef34139b50 con 0x55ef34244aa0 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3fbf400640 10 client.16911 handle_command_reply: tid=0 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 client.16911 shutdown 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 msgr2=0x55ef341de160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341de160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55ef341da840 tx=0x7f3fd0005d50 comp rx=0 tx=0).stop 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 shutdown_connections 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef341d7c00 0x55ef341de770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55ef34244aa0 0x55ef34246e90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef341cf080 0x55ef341de160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 --2- 172.21.15.119:0/2439420452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef341ce6d0 0x55ef341dfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3fbf400640 0 client.16911 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 >> 172.21.15.119:0/2439420452 conn(0x55ef341c8da0 msgr2=0x55ef341c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 shutdown_connections 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 1 -- 172.21.15.119:0/2439420452 wait complete. 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.897+0000 7f3feb59e740 20 client.16911 trim_cache size 0 max 16384 2024-03-28T18:50:08.902 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 msgr2=0x7f3fcc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 0x7f3fcc0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x55ef341ce1a0 tx=0x7f3fd80d50e0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 msgr2=0x7f3fe019e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800ebc0 tx=0x7f3fd800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 shutdown_connections 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3fcc0a2970 0x7f3fcc0a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01124e0 0x7f3fe01a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3fe00764f0 0x7f3fe019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 --2- 172.21.15.119:0/1592533199 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3fe0075bd0 0x7f3fe019e590 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:08.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 >> 172.21.15.119:0/1592533199 conn(0x7f3fe00ff2f0 msgr2=0x7f3fe01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:08.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 shutdown_connections 2024-03-28T18:50:08.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:08.898+0000 7f3fe8800640 1 -- 172.21.15.119:0/1592533199 wait complete. 2024-03-28T18:50:08.915 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:08.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:50:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:08 smithi131 ceph-mon[36065]: pgmap v185: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:08.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:08 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1442207385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:09.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 -- 172.21.15.119:0/3298487193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81341040c0 msgr2=0x7f81341105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 --2- 172.21.15.119:0/3298487193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81341040c0 0x7f81341105d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f812c009ca0 tx=0x7f812c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 -- 172.21.15.119:0/3298487193 shutdown_connections 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 --2- 172.21.15.119:0/3298487193 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f81341040c0 0x7f81341105d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 --2- 172.21.15.119:0/3298487193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134103700 0x7f8134103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 --2- 172.21.15.119:0/3298487193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8134109710 0x7f8134109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 -- 172.21.15.119:0/3298487193 >> 172.21.15.119:0/3298487193 conn(0x7f81340ff1b0 msgr2=0x7f81341015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 -- 172.21.15.119:0/3298487193 shutdown_connections 2024-03-28T18:50:09.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.114+0000 7f8133200640 1 -- 172.21.15.119:0/3298487193 wait complete. 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.115+0000 7f8133200640 1 Processor -- start 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.115+0000 7f8133200640 1 -- start start 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81341040c0 0x7f813419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134109710 0x7f813419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134113ae0 con 0x7f8134109710 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134113960 con 0x7f8134103700 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8133200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134113c60 con 0x7f81341040c0 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53560/0 (socket says 172.21.15.119:53560) 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81341040c0 0x7f813419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 -- 172.21.15.119:0/3140742250 learned_addr learned my addr 172.21.15.119:0/3140742250 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8132800640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134109710 0x7f813419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 -- 172.21.15.119:0/3140742250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81341040c0 msgr2=0x7f813419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81341040c0 0x7f813419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 -- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134109710 msgr2=0x7f813419e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134109710 0x7f813419e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.116+0000 7f8131e00640 1 -- 172.21.15.119:0/3140742250 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813419ef50 con 0x7f8134103700 2024-03-28T18:50:09.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f8131e00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f812400e9f0 tx=0x7f812400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f812aa00640 1 -- 172.21.15.119:0/3140742250 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812400cd80 con 0x7f8134103700 2024-03-28T18:50:09.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f812aa00640 1 -- 172.21.15.119:0/3140742250 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8124010910 con 0x7f8134103700 2024-03-28T18:50:09.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813419f240 con 0x7f8134103700 2024-03-28T18:50:09.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f812aa00640 1 -- 172.21.15.119:0/3140742250 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124018990 con 0x7f8134103700 2024-03-28T18:50:09.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.117+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81341a6ad0 con 0x7f8134103700 2024-03-28T18:50:09.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.121+0000 7f813b7ca740 1 Processor -- start 2024-03-28T18:50:09.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.121+0000 7f813b7ca740 1 -- start start 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f812aa00640 1 -- 172.21.15.119:0/3140742250 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8124010ac0 con 0x7f8134103700 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587360d42c0 con 0x558736171260 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873607cc90 con 0x5587361708b0 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f813b7ca740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587360d0360 con 0x558736179de0 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8131400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53570/0 (socket says 172.21.15.119:53570) 2024-03-28T18:50:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 -- 172.21.15.119:0/2803984536 learned_addr learned my addr 172.21.15.119:0/2803984536 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8131e00640 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 -- 172.21.15.119:0/2803984536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 msgr2=0x55873617c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 -- 172.21.15.119:0/2803984536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 msgr2=0x558736179810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.122+0000 7f8132800640 1 -- 172.21.15.119:0/2803984536 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587360a1080 con 0x5587361708b0 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.123+0000 7f8132800640 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f812c02c700 tx=0x7f812c005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e41651035a00f8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.123+0000 7f811f000640 1 -- 172.21.15.119:0/2803984536 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c043020 con 0x5587361708b0 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.123+0000 7f811f000640 1 -- 172.21.15.119:0/2803984536 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f812c032c90 con 0x5587361708b0 2024-03-28T18:50:09.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.123+0000 7f811f000640 1 -- 172.21.15.119:0/2803984536 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c039040 con 0x5587361708b0 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.123+0000 7f812aa00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 0x7f81180a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f812aa00640 1 -- 172.21.15.119:0/3140742250 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f81240cf0d0 con 0x7f8134103700 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 -- 172.21.15.119:0/2803984536 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 msgr2=0x558736170c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f812c02c700 tx=0x7f812c005f80 comp rx=0 tx=0).stop 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f8131e00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 0x7f81180a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 -- 172.21.15.119:0/2803984536 shutdown_connections 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 --2- 172.21.15.119:0/2803984536 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736170c90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 -- 172.21.15.119:0/2803984536 >> 172.21.15.119:0/2803984536 conn(0x55873616af80 msgr2=0x55873616b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 -- 172.21.15.119:0/2803984536 shutdown_connections 2024-03-28T18:50:09.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f813b7ca740 1 -- 172.21.15.119:0/2803984536 wait complete. 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.124+0000 7f8131e00640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 0x7f81180a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f81240cc870 tx=0x7f8124000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 -- start start 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 Processor -- start 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587361708b0 0x558736179640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 0x558736176f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873617ccf0 con 0x558736179de0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587360900d0 con 0x558736171260 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873617ce70 con 0x5587361708b0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.125+0000 7f813b7ca740 10 client.0 _ll_register_callbacks cb 0x558736094e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53582/0 (socket says 172.21.15.119:53582) 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 -- 172.21.15.119:0/950287729 learned_addr learned my addr 172.21.15.119:0/950287729 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8132800640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587361708b0 0x558736179640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8132800640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 0x558736176f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 -- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587361708b0 msgr2=0x558736179640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587361708b0 0x558736179640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 -- 172.21.15.119:0/950287729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 msgr2=0x558736176f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 0x558736176f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 -- 172.21.15.119:0/950287729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558735de83f0 con 0x558736171260 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8132800640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 0x558736176f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8131400640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81200126e0 tx=0x7f8120012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d2e3ed9b9b2fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8113400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8120012190 con 0x558736171260 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.126+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f812000f920 con 0x558736171260 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f812c032c90 con 0x558736171260 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f812000fb20 con 0x558736171260 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 10 client.27152 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558735eb27d0 con 0x558736171260 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f812c032c90 con 0x558736171260 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f812001c420 con 0x558736171260 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 10 client.27152 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 10 client.27152 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 20 client.27152 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x558736189c10 0x55873618c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 4 client.27152 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.127+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587360e1800 con 0x558736189c10 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.128+0000 7f8132800640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x558736189c10 0x55873618c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.128+0000 7f8132800640 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x558736189c10 0x55873618c000 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=913766bc5398887d server_cookie=edc5aa3fe145bc88 in_seq=0 out_seq=0 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.128+0000 7f8113400640 10 client.27152 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f8113400640 1 -- 172.21.15.119:0/950287729 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5587360e1800 con 0x558736189c10 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f8113400640 10 client.27152 handle_command_reply: tid=0 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 client.27152 shutdown 2024-03-28T18:50:09.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 msgr2=0x5587361769d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81200126e0 tx=0x7f8120012ba0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 shutdown_connections 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x558736189c10 0x55873618c000 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736179de0 0x558736176f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x5587361769d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 --2- 172.21.15.119:0/950287729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587361708b0 0x558736179640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f8113400640 0 client.27152 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.129+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 >> 172.21.15.119:0/950287729 conn(0x55873616af80 msgr2=0x55873616ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.130+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 shutdown_connections 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.130+0000 7f813b7ca740 1 -- 172.21.15.119:0/950287729 wait complete. 2024-03-28T18:50:09.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.130+0000 7f813b7ca740 20 client.27152 trim_cache size 0 max 16384 2024-03-28T18:50:09.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 Processor -- start 2024-03-28T18:50:09.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 -- start start 2024-03-28T18:50:09.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736179100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x558736173c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341af300 con 0x558736171260 2024-03-28T18:50:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341b0c80 con 0x5587361708b0 2024-03-28T18:50:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.145+0000 7f813b7ca740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341af960 con 0x558736179de0 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55262/0 (socket says 172.21.15.119:55262) 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8132800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736179100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 -- 172.21.15.119:0/151519824 learned_addr learned my addr 172.21.15.119:0/151519824 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8132800640 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x558736173c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 -- 172.21.15.119:0/151519824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 msgr2=0x558736173c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x558736173c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 -- 172.21.15.119:0/151519824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 msgr2=0x558736179100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736179100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8131400640 1 -- 172.21.15.119:0/151519824 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587361774f0 con 0x558736171260 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.146+0000 7f8132800640 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736179100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.147+0000 7f8131400640 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f812000f890 tx=0x7f812000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18aed38518c68e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.147+0000 7f811e600640 1 -- 172.21.15.119:0/151519824 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000fcc0 con 0x558736171260 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.147+0000 7f811e600640 1 -- 172.21.15.119:0/151519824 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f812001c420 con 0x558736171260 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.147+0000 7f811e600640 1 -- 172.21.15.119:0/151519824 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81200035f0 con 0x558736171260 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 -- 172.21.15.119:0/151519824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 msgr2=0x558736179640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f812000f890 tx=0x7f812000fab0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 -- 172.21.15.119:0/151519824 shutdown_connections 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x558736173c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558736171260 0x558736179640 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 --2- 172.21.15.119:0/151519824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5587361708b0 0x558736179100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 -- 172.21.15.119:0/151519824 >> 172.21.15.119:0/151519824 conn(0x55873616af80 msgr2=0x55873616a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 -- 172.21.15.119:0/151519824 shutdown_connections 2024-03-28T18:50:09.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.148+0000 7f813b7ca740 1 -- 172.21.15.119:0/151519824 wait complete. 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 -- start start 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 Processor -- start 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587361708b0 0x558736181b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341b0620 con 0x5587361708b0 2024-03-28T18:50:09.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341affc0 con 0x558736171260 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81341ad4c0 con 0x558736179de0 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f813b7ca740 10 client.0 _ll_register_callbacks cb 0x7f8120009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f8131400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f8131400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53592/0 (socket says 172.21.15.119:53592) 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f8131400640 1 -- 172.21.15.119:0/3981392203 learned_addr learned my addr 172.21.15.119:0/3981392203 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.149+0000 7f8132800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587361708b0 0x558736181b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8132800640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 -- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 msgr2=0x55873617e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 -- 172.21.15.119:0/3981392203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587361708b0 msgr2=0x558736181b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587361708b0 0x558736181b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 -- 172.21.15.119:0/3981392203 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55873617e9a0 con 0x558736171260 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8132800640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:50:09.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8131400640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f81200098b0 tx=0x7f812001c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d3d7ff1ac409412 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8113400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812001c5d0 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f812001c420 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f812001ca30 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8120039050 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f813b7ca740 10 client.27158 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.150+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5587361e61b0 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f812001c420 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8120003c70 con 0x558736171260 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 10 client.27158 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 10 client.27158 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 20 client.27158 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5587361e67f0 0x5587361e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 4 client.27158 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:09.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5587360dbb50 con 0x5587361e67f0 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.151+0000 7f8132800640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5587361e67f0 0x5587361e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f8132800640 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5587361e67f0 0x5587361e8c10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5d430e4a45dd740 server_cookie=d64bf8465205442f in_seq=0 out_seq=0 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f8113400640 10 client.27158 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f8113400640 1 -- 172.21.15.119:0/3981392203 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5587360dbb50 con 0x5587361e67f0 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f8113400640 10 client.27158 handle_command_reply: tid=0 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 client.27158 shutdown 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 msgr2=0x55873617df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f81200098b0 tx=0x7f812001c840 comp rx=0 tx=0).stop 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 shutdown_connections 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5587361e67f0 0x5587361e8c10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558736179de0 0x55873617e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558736171260 0x55873617df20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f8113400640 0 client.27158 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 --2- 172.21.15.119:0/3981392203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587361708b0 0x558736181b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.152+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 >> 172.21.15.119:0/3981392203 conn(0x55873616af80 msgr2=0x558736171640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 shutdown_connections 2024-03-28T18:50:09.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f813b7ca740 1 -- 172.21.15.119:0/3981392203 wait complete. 2024-03-28T18:50:09.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f813b7ca740 20 client.27158 trim_cache size 0 max 16384 2024-03-28T18:50:09.157 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 msgr2=0x7f81180a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 0x7f81180a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f81240cc870 tx=0x7f8124000bc0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.153+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 msgr2=0x7f8134199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f812400e9f0 tx=0x7f812400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 shutdown_connections 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f81180a29f0 0x7f81180a4eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8134109710 0x7f813419e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81341040c0 0x7f813419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 --2- 172.21.15.119:0/3140742250 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8134103700 0x7f8134199f60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 >> 172.21.15.119:0/3140742250 conn(0x7f81340ff1b0 msgr2=0x7f81340ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 shutdown_connections 2024-03-28T18:50:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.154+0000 7f8133200640 1 -- 172.21.15.119:0/3140742250 wait complete. 2024-03-28T18:50:09.171 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:09.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:50:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 -- 172.21.15.119:0/389961881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c10cea0 msgr2=0x7fe64c114a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 --2- 172.21.15.119:0/389961881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c10cea0 0x7fe64c114a70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe644009be0 tx=0x7fe64402bef0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 -- 172.21.15.119:0/389961881 shutdown_connections 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 --2- 172.21.15.119:0/389961881 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c10cea0 0x7fe64c114a70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 --2- 172.21.15.119:0/389961881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c108800 0x7fe64c10c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 --2- 172.21.15.119:0/389961881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c107e50 0x7fe64c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 -- 172.21.15.119:0/389961881 >> 172.21.15.119:0/389961881 conn(0x7fe64c0774c0 msgr2=0x7fe64c1064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 -- 172.21.15.119:0/389961881 shutdown_connections 2024-03-28T18:50:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.385+0000 7fe652c00640 1 -- 172.21.15.119:0/389961881 wait complete. 2024-03-28T18:50:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.386+0000 7fe652c00640 1 Processor -- start 2024-03-28T18:50:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.386+0000 7fe652c00640 1 -- start start 2024-03-28T18:50:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c10cea0 0x7fe64c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c1a2cb0 0x7fe64c1a3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c117fa0 con 0x7fe64c10cea0 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c117e20 con 0x7fe64c107e50 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c118120 con 0x7fe64c1a2cb0 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe650e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c10cea0 0x7fe64c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53600/0 (socket says 172.21.15.119:53600) 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 -- 172.21.15.119:0/2010771216 learned_addr learned my addr 172.21.15.119:0/2010771216 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe652200640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c1a2cb0 0x7fe64c1a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 -- 172.21.15.119:0/2010771216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c1a2cb0 msgr2=0x7fe64c1a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c1a2cb0 0x7fe64c1a3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 -- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c10cea0 msgr2=0x7fe64c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.387+0000 7fe651800640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c10cea0 0x7fe64c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe651800640 1 -- 172.21.15.119:0/2010771216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c1a37a0 con 0x7fe64c107e50 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe651800640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e9f0 tx=0x7fe64000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe64a000640 1 -- 172.21.15.119:0/2010771216 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64000cd80 con 0x7fe64c107e50 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe64a000640 1 -- 172.21.15.119:0/2010771216 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe640010910 con 0x7fe64c107e50 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c1a3a90 con 0x7fe64c107e50 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe64a000640 1 -- 172.21.15.119:0/2010771216 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe640018990 con 0x7fe64c107e50 2024-03-28T18:50:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.388+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c1ab220 con 0x7fe64c107e50 2024-03-28T18:50:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 Processor -- start 2024-03-28T18:50:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 -- start start 2024-03-28T18:50:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe64a000640 1 -- 172.21.15.119:0/2010771216 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe640010ac0 con 0x7fe64c107e50 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba61230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bb9b92c0 con 0x5599bba56260 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bb961c90 con 0x5599bba5ede0 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe656fcb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bb9b5360 con 0x5599bba558b0 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe652200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe652200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58372/0 (socket says 172.21.15.119:58372) 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe652200640 1 -- 172.21.15.119:0/3444323342 learned_addr learned my addr 172.21.15.119:0/3444323342 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe650e00640 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba61230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.392+0000 7fe651800640 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 -- 172.21.15.119:0/3444323342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 msgr2=0x5599bba61230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba61230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 -- 172.21.15.119:0/3444323342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 msgr2=0x5599bba5e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 -- 172.21.15.119:0/3444323342 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599bb986080 con 0x5599bba558b0 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe652200640 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe644009be0 tx=0x7fe644009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66a9ef1422780955 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe63ea00640 1 -- 172.21.15.119:0/3444323342 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644043030 con 0x5599bba558b0 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe63ea00640 1 -- 172.21.15.119:0/3444323342 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe644033c00 con 0x5599bba558b0 2024-03-28T18:50:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.393+0000 7fe63ea00640 1 -- 172.21.15.119:0/3444323342 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64402c930 con 0x5599bba558b0 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe64a000640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 0x7fe6340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 -- 172.21.15.119:0/3444323342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 msgr2=0x5599bba55c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe644009be0 tx=0x7fe644009840 comp rx=0 tx=0).stop 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe64a000640 1 -- 172.21.15.119:0/2010771216 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe6400cf200 con 0x7fe64c107e50 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe651800640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 0x7fe6340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 -- 172.21.15.119:0/3444323342 shutdown_connections 2024-03-28T18:50:09.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba61230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3444323342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba558b0 0x5599bba55c90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 -- 172.21.15.119:0/3444323342 >> 172.21.15.119:0/3444323342 conn(0x5599bba4ff80 msgr2=0x5599bba50360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 -- 172.21.15.119:0/3444323342 shutdown_connections 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.394+0000 7fe656fcb740 1 -- 172.21.15.119:0/3444323342 wait complete. 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.395+0000 7fe651800640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 0x7fe6340a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6400cc9a0 tx=0x7fe640000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.395+0000 7fe656fcb740 1 -- start start 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.395+0000 7fe656fcb740 1 Processor -- start 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 0x5599bba5ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba5c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bba61cf0 con 0x5599bba558b0 2024-03-28T18:50:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bb9750d0 con 0x5599bba5ede0 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599bba61e70 con 0x5599bba56260 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe656fcb740 10 client.0 _ll_register_callbacks cb 0x5599bb979e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe652200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe652200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55288/0 (socket says 172.21.15.119:55288) 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe652200640 1 -- 172.21.15.119:0/3335820212 learned_addr learned my addr 172.21.15.119:0/3335820212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 0x5599bba5ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe652200640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba5c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 -- 172.21.15.119:0/3335820212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 msgr2=0x5599bba5c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba5c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 -- 172.21.15.119:0/3335820212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 msgr2=0x5599bba5e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe650e00640 1 -- 172.21.15.119:0/3335820212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599bb6cd3f0 con 0x5599bba56260 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.396+0000 7fe652200640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe650e00640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 0x5599bba5ba90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe64c10e1b0 tx=0x7fe638012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a8406d739aa9da5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63800f8f0 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe63800fa90 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe644033c00 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe63800fcd0 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe656fcb740 10 client.27145 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5599bb7977d0 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe644033c00 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.397+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe63801c4b0 con 0x5599bba56260 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 10 client.27145 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 10 client.27145 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 20 client.27145 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bba6ec60 0x5599bba710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 4 client.27145 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599bb9c6800 con 0x5599bba6ec60 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe652200640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bba6ec60 0x5599bba710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe652200640 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bba6ec60 0x5599bba710a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ca97e7b04b7df9b server_cookie=c921e8587db5d1e2 in_seq=0 out_seq=0 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.398+0000 7fe633400640 10 client.27145 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe633400640 1 -- 172.21.15.119:0/3335820212 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5599bb9c6800 con 0x5599bba6ec60 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe633400640 10 client.27145 handle_command_reply: tid=0 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 client.27145 shutdown 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 msgr2=0x5599bba5ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 0x5599bba5ba90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe64c10e1b0 tx=0x7fe638012b60 comp rx=0 tx=0).stop 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 shutdown_connections 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bba6ec60 0x5599bba710a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba5ede0 0x5599bba5c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba56260 0x5599bba5ba90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 --2- 172.21.15.119:0/3335820212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba558b0 0x5599bba5e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 >> 172.21.15.119:0/3335820212 conn(0x5599bba4ff80 msgr2=0x5599bba51cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe633400640 0 client.27145 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:09.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.399+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 shutdown_connections 2024-03-28T18:50:09.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.400+0000 7fe656fcb740 1 -- 172.21.15.119:0/3335820212 wait complete. 2024-03-28T18:50:09.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.400+0000 7fe656fcb740 20 client.27145 trim_cache size 0 max 16384 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 Processor -- start 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 -- start start 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba58d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b37d0 con 0x5599bba56260 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b5150 con 0x5599bba558b0 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.413+0000 7fe656fcb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b3e30 con 0x5599bba5ede0 2024-03-28T18:50:09.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53634/0 (socket says 172.21.15.119:53634) 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 -- 172.21.15.119:0/1158206904 learned_addr learned my addr 172.21.15.119:0/1158206904 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba58d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe650e00640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 -- 172.21.15.119:0/1158206904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 msgr2=0x5599bba58d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba58d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 -- 172.21.15.119:0/1158206904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 msgr2=0x5599bba5e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 -- 172.21.15.119:0/1158206904 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599bba5c3a0 con 0x5599bba558b0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe650e00640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe652200640 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe644000f80 tx=0x7fe644003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be479c23259c1d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe63e000640 1 -- 172.21.15.119:0/1158206904 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644043040 con 0x5599bba558b0 2024-03-28T18:50:09.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe63e000640 1 -- 172.21.15.119:0/1158206904 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe644044040 con 0x5599bba558b0 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.414+0000 7fe63e000640 1 -- 172.21.15.119:0/1158206904 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644004140 con 0x5599bba558b0 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- 172.21.15.119:0/1158206904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 msgr2=0x5599bba5e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe644000f80 tx=0x7fe644003e50 comp rx=0 tx=0).stop 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- 172.21.15.119:0/1158206904 shutdown_connections 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba58d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba5e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 --2- 172.21.15.119:0/1158206904 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba5e1c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- 172.21.15.119:0/1158206904 >> 172.21.15.119:0/1158206904 conn(0x5599bba4ff80 msgr2=0x5599bba4f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- 172.21.15.119:0/1158206904 shutdown_connections 2024-03-28T18:50:09.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- 172.21.15.119:0/1158206904 wait complete. 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.415+0000 7fe656fcb740 1 -- start start 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 Processor -- start 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba65660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b4af0 con 0x5599bba56260 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b4490 con 0x5599bba558b0 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c1b1990 con 0x5599bba5ede0 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe656fcb740 10 client.0 _ll_register_callbacks cb 0x7fe638005b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe650e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe652200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe652200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53650/0 (socket says 172.21.15.119:53650) 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe650e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55318/0 (socket says 172.21.15.119:55318) 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe652200640 1 -- 172.21.15.119:0/91402779 learned_addr learned my addr 172.21.15.119:0/91402779 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.416+0000 7fe652200640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba65660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 -- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 msgr2=0x5599bba65660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba65660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 -- 172.21.15.119:0/91402779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 msgr2=0x5599bba66d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 -- 172.21.15.119:0/91402779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599bba65dd0 con 0x5599bba56260 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe652200640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe650e00640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe638005c80 tx=0x7fe63800fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d795be4df383a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe633400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:50:09.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63801c420 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fe63801c5c0 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.417+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe644044040 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe638004020 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 10 client.16944 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5599bbacb4f0 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe644044040 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe638004660 con 0x5599bba56260 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 10 client.16944 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 10 client.16944 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 20 client.16944 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bbacbc70 0x5599bbace090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 4 client.16944 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5599bb9c0b50 con 0x5599bbacbc70 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.418+0000 7fe652200640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bbacbc70 0x5599bbace090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe652200640 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bbacbc70 0x5599bbace090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c31a2c5b4a46bbeb server_cookie=db6078bc47aecd81 in_seq=0 out_seq=0 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe633400640 10 client.16944 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe633400640 1 -- 172.21.15.119:0/91402779 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5599bb9c0b50 con 0x5599bbacbc70 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe633400640 10 client.16944 handle_command_reply: tid=0 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 client.16944 shutdown 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 msgr2=0x5599bba65120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe638005c80 tx=0x7fe63800fe70 comp rx=0 tx=0).stop 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 shutdown_connections 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5599bbacbc70 0x5599bbace090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5599bba5ede0 0x5599bba65660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5599bba56260 0x5599bba65120 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 --2- 172.21.15.119:0/91402779 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5599bba558b0 0x5599bba66d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe633400640 0 client.16944 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 >> 172.21.15.119:0/91402779 conn(0x5599bba4ff80 msgr2=0x5599bba511d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 shutdown_connections 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.419+0000 7fe656fcb740 1 -- 172.21.15.119:0/91402779 wait complete. 2024-03-28T18:50:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe656fcb740 20 client.16944 trim_cache size 0 max 16384 2024-03-28T18:50:09.425 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:09.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 msgr2=0x7fe6340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 0x7fe6340a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6400cc9a0 tx=0x7fe640000bc0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 msgr2=0x7fe64c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe64000e9f0 tx=0x7fe64000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 shutdown_connections 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe6340a2970 0x7fe6340a4e30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe64c1a2cb0 0x7fe64c1a3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe64c10cea0 0x7fe64c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 --2- 172.21.15.119:0/2010771216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe64c107e50 0x7fe64c19e440 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 >> 172.21.15.119:0/2010771216 conn(0x7fe64c0774c0 msgr2=0x7fe64c077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 shutdown_connections 2024-03-28T18:50:09.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.420+0000 7fe652c00640 1 -- 172.21.15.119:0/2010771216 wait complete. 2024-03-28T18:50:09.438 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:09.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 -- 172.21.15.119:0/1288562503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 msgr2=0x7fcc68109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 --2- 172.21.15.119:0/1288562503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc68109c30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009930 tx=0x7fcc5c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 -- 172.21.15.119:0/1288562503 shutdown_connections 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 --2- 172.21.15.119:0/1288562503 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc68110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 --2- 172.21.15.119:0/1288562503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 0x7fcc68103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 --2- 172.21.15.119:0/1288562503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc68109c30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 -- 172.21.15.119:0/1288562503 >> 172.21.15.119:0/1288562503 conn(0x7fcc680ff2f0 msgr2=0x7fcc68101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 -- 172.21.15.119:0/1288562503 shutdown_connections 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.656+0000 7fcc70600640 1 -- 172.21.15.119:0/1288562503 wait complete. 2024-03-28T18:50:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.657+0000 7fcc70600640 1 Processor -- start 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.657+0000 7fcc70600640 1 -- start start 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 0x7fcc6819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc6819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68113c70 con 0x7fcc68103840 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68113af0 con 0x7fcc68104200 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc70600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68113df0 con 0x7fcc68109850 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 0x7fcc6819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53662/0 (socket says 172.21.15.119:53662) 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 -- 172.21.15.119:0/2071702082 learned_addr learned my addr 172.21.15.119:0/2071702082 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6e600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc6819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 -- 172.21.15.119:0/2071702082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 msgr2=0x7fcc6819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc6819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 -- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 msgr2=0x7fcc6819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 0x7fcc6819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.658+0000 7fcc6d200640 1 -- 172.21.15.119:0/2071702082 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6819f160 con 0x7fcc68104200 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc6d200640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400ea60 tx=0x7fcc5400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc66000640 1 -- 172.21.15.119:0/2071702082 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5400cde0 con 0x7fcc68104200 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc66000640 1 -- 172.21.15.119:0/2071702082 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc540035f0 con 0x7fcc68104200 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6819f450 con 0x7fcc68104200 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc66000640 1 -- 172.21.15.119:0/2071702082 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54010670 con 0x7fcc68104200 2024-03-28T18:50:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.659+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681a6ce0 con 0x7fcc68104200 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.662+0000 7fcc7341d740 1 Processor -- start 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.662+0000 7fcc7341d740 1 -- start start 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013e6d0 0x55d7f013eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc66000640 1 -- 172.21.15.119:0/2071702082 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fcc540037a0 con 0x7fcc68104200 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f014a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f00a22c0 con 0x55d7f013e6d0 2024-03-28T18:50:09.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f004ac90 con 0x55d7f0147c00 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc7341d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f009e360 con 0x55d7f013f080 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013e6d0 0x55d7f013eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58424/0 (socket says 172.21.15.119:58424) 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 -- 172.21.15.119:0/2668431743 learned_addr learned my addr 172.21.15.119:0/2668431743 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6d200640 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f014a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 -- 172.21.15.119:0/2668431743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 msgr2=0x55d7f014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f014a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 -- 172.21.15.119:0/2668431743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013e6d0 msgr2=0x55d7f013eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013e6d0 0x55d7f013eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.663+0000 7fcc6dc00640 1 -- 172.21.15.119:0/2668431743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7f006f080 con 0x55d7f013f080 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.664+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c005ab0 tx=0x7fcc5c005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2b2ab4e755b649 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.664+0000 7fcc5aa00640 1 -- 172.21.15.119:0/2668431743 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c045070 con 0x55d7f013f080 2024-03-28T18:50:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.664+0000 7fcc5aa00640 1 -- 172.21.15.119:0/2668431743 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c02c640 con 0x55d7f013f080 2024-03-28T18:50:09.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.664+0000 7fcc5aa00640 1 -- 172.21.15.119:0/2668431743 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c004b60 con 0x55d7f013f080 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc66000640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 0x7fcc500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 -- 172.21.15.119:0/2668431743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 msgr2=0x55d7f0147630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c005ab0 tx=0x7fcc5c005e40 comp rx=0 tx=0).stop 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc66000640 1 -- 172.21.15.119:0/2071702082 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fcc540cf170 con 0x7fcc68104200 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 -- 172.21.15.119:0/2668431743 shutdown_connections 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f014a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013f080 0x55d7f0147630 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 --2- 172.21.15.119:0/2668431743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013e6d0 0x55d7f013eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 -- 172.21.15.119:0/2668431743 >> 172.21.15.119:0/2668431743 conn(0x55d7f0138da0 msgr2=0x55d7f0139180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc6e600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 0x7fcc500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 -- 172.21.15.119:0/2668431743 shutdown_connections 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.665+0000 7fcc7341d740 1 -- 172.21.15.119:0/2668431743 wait complete. 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 -- start start 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc6e600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 0x7fcc500a4eb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc60005fd0 tx=0x7fcc60009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 Processor -- start 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013f080 0x55d7f01448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f0147c00 0x55d7f0144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f014ab10 con 0x55d7f0147c00 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f005e0d0 con 0x55d7f013f080 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.666+0000 7fcc7341d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f014ac90 con 0x55d7f013e6d0 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc7341d740 10 client.0 _ll_register_callbacks cb 0x55d7f0062e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58434/0 (socket says 172.21.15.119:58434) 2024-03-28T18:50:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3560856746 learned_addr learned my addr 172.21.15.119:0/3560856746 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6d200640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013f080 0x55d7f01448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6e600640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f0147c00 0x55d7f0144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013f080 msgr2=0x55d7f01448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013f080 0x55d7f01448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3560856746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f0147c00 msgr2=0x55d7f0144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f0147c00 0x55d7f0144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3560856746 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7efdb63f0 con 0x55d7f013e6d0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009900 tx=0x7fcc5c02caa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2918e8f9206b706e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc4f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.667+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c04e070 con 0x55d7f013e6d0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc5c02c640 con 0x55d7f013e6d0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fcc5c004ee0 con 0x55d7f013e6d0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc5c057050 con 0x55d7f013e6d0 2024-03-28T18:50:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 10 client.27163 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d7efe807d0 con 0x55d7f013e6d0 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c02c640 con 0x55d7f013e6d0 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcc5c051020 con 0x55d7f013e6d0 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 10 client.27163 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 10 client.27163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 20 client.27163 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f0159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 4 client.27163 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.668+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7f00af800 con 0x55d7f0157930 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.669+0000 7fcc6d200640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f0159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.669+0000 7fcc6d200640 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f0159d20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=680cf60335a458d7 server_cookie=27cc995a339e0e23 in_seq=0 out_seq=0 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.669+0000 7fcc4f400640 10 client.27163 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc4f400640 1 -- 172.21.15.119:0/3560856746 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d7f00af800 con 0x55d7f0157930 2024-03-28T18:50:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc4f400640 10 client.27163 handle_command_reply: tid=0 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 client.27163 shutdown 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 msgr2=0x55d7f0147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009900 tx=0x7fcc5c02caa0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 shutdown_connections 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f0159d20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f0147c00 0x55d7f0144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc4f400640 0 client.27163 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013f080 0x55d7f01448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3560856746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0147540 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 >> 172.21.15.119:0/3560856746 conn(0x55d7f0138da0 msgr2=0x55d7f013aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 shutdown_connections 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 1 -- 172.21.15.119:0/3560856746 wait complete. 2024-03-28T18:50:09.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.670+0000 7fcc7341d740 20 client.27163 trim_cache size 0 max 16384 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.684+0000 7fcc7341d740 1 Processor -- start 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.684+0000 7fcc7341d740 1 -- start start 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f0147540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681af490 con 0x55d7f013f080 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681b0e10 con 0x55d7f0147c00 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc7341d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681afaf0 con 0x55d7f013e6d0 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6e600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f0147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55352/0 (socket says 172.21.15.119:55352) 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 -- 172.21.15.119:0/1462482732 learned_addr learned my addr 172.21.15.119:0/1462482732 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58440/0 (socket says 172.21.15.119:58440) 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 -- 172.21.15.119:0/1462482732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 msgr2=0x55d7f0137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 -- 172.21.15.119:0/1462482732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 msgr2=0x55d7f0147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f0147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6d200640 1 -- 172.21.15.119:0/1462482732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7f013ee20 con 0x55d7f013f080 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.685+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc6d200640 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcc5409b6a0 tx=0x7fcc540d1630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888a4b8d3ca5555b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc5a000640 1 -- 172.21.15.119:0/1462482732 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5409b250 con 0x55d7f013f080 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc5a000640 1 -- 172.21.15.119:0/1462482732 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5409be10 con 0x55d7f013f080 2024-03-28T18:50:09.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc5a000640 1 -- 172.21.15.119:0/1462482732 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5409b070 con 0x55d7f013f080 2024-03-28T18:50:09.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc7341d740 1 -- 172.21.15.119:0/1462482732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 msgr2=0x55d7f0147000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.686+0000 7fcc7341d740 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcc5409b6a0 tx=0x7fcc540d1630 comp rx=0 tx=0).stop 2024-03-28T18:50:09.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 -- 172.21.15.119:0/1462482732 shutdown_connections 2024-03-28T18:50:09.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f0147c00 0x55d7f0147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f0147000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 --2- 172.21.15.119:0/1462482732 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f013e6d0 0x55d7f0137eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 -- 172.21.15.119:0/1462482732 >> 172.21.15.119:0/1462482732 conn(0x55d7f0138da0 msgr2=0x55d7f013f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 -- 172.21.15.119:0/1462482732 shutdown_connections 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 -- 172.21.15.119:0/1462482732 wait complete. 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 -- start start 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.687+0000 7fcc7341d740 1 Processor -- start 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f014dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 0x55d7f01ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681b07b0 con 0x55d7f013f080 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681b0150 con 0x55d7f013e6d0 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681ad650 con 0x55d7f0147c00 2024-03-28T18:50:09.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc7341d740 10 client.0 _ll_register_callbacks cb 0x7fcc60009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6e600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 0x55d7f01ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53700/0 (socket says 172.21.15.119:53700) 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6e600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 0x55d7f01ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58452/0 (socket says 172.21.15.119:58452) 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f014dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3919388249 learned_addr learned my addr 172.21.15.119:0/3919388249 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3919388249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 msgr2=0x55d7f01ba790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 0x55d7f01ba790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3919388249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 msgr2=0x55d7f014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f014dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 -- 172.21.15.119:0/3919388249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7f01bacd0 con 0x55d7f013e6d0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc6dc00640 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c02c230 tx=0x7fcc5c005a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a05d8ee13b2b21b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:09.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.688+0000 7fcc4f400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c04e070 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fcc5c04a650 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc5409be10 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc5c045070 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 10 client.27194 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d7f01bbe70 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5409be10 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcc5c051040 con 0x55d7f013e6d0 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 10 client.27194 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 10 client.27194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 20 client.27194 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:09.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f01be540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 4 client.27194 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.689+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d7f00a9b50 con 0x55d7f0157930 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc6d200640 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f01be540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc6d200640 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f01be540 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61aca64514ff0b61 server_cookie=f4583058c23f5180 in_seq=0 out_seq=0 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc4f400640 10 client.27194 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc4f400640 1 -- 172.21.15.119:0/3919388249 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d7f00a9b50 con 0x55d7f0157930 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc4f400640 10 client.27194 handle_command_reply: tid=0 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 client.27194 shutdown 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 msgr2=0x55d7f014d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c02c230 tx=0x7fcc5c005a60 comp rx=0 tx=0).stop 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 shutdown_connections 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d7f0157930 0x55d7f01be540 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d7f0147c00 0x55d7f01ba790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d7f013f080 0x55d7f014dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 --2- 172.21.15.119:0/3919388249 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d7f013e6d0 0x55d7f014d740 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc4f400640 0 client.27194 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.690+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 >> 172.21.15.119:0/3919388249 conn(0x55d7f0138da0 msgr2=0x55d7f0137d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 shutdown_connections 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc7341d740 1 -- 172.21.15.119:0/3919388249 wait complete. 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc7341d740 20 client.27194 trim_cache size 0 max 16384 2024-03-28T18:50:09.696 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 msgr2=0x7fcc500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 0x7fcc500a4eb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc60005fd0 tx=0x7fcc60009500 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 msgr2=0x7fcc6819a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400ea60 tx=0x7fcc5400ef20 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 shutdown_connections 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcc500a29f0 0x7fcc500a4eb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc68109850 0x7fcc6819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcc68104200 0x7fcc6819a6b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 --2- 172.21.15.119:0/2071702082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc68103840 0x7fcc6819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.691+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 >> 172.21.15.119:0/2071702082 conn(0x7fcc680ff2f0 msgr2=0x7fcc680ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.692+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 shutdown_connections 2024-03-28T18:50:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:09.692+0000 7fcc70600640 1 -- 172.21.15.119:0/2071702082 wait complete. 2024-03-28T18:50:10.711 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:50:10.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:10.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:50:10.798 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:10 smithi119 ceph-mon[31813]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:10 smithi196 ceph-mon[36300]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 -- 172.21.15.119:0/1652641527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c075bd0 msgr2=0x7f3a8c075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 --2- 172.21.15.119:0/1652641527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c075fb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009930 tx=0x7f3a8002bc20 comp rx=0 tx=0).stop 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 -- 172.21.15.119:0/1652641527 shutdown_connections 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 --2- 172.21.15.119:0/1652641527 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 --2- 172.21.15.119:0/1652641527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 --2- 172.21.15.119:0/1652641527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c075fb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 -- 172.21.15.119:0/1652641527 >> 172.21.15.119:0/1652641527 conn(0x7f3a8c0ff2f0 msgr2=0x7f3a8c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.940+0000 7f3a93a00640 1 -- 172.21.15.119:0/1652641527 shutdown_connections 2024-03-28T18:50:10.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.941+0000 7f3a93a00640 1 -- 172.21.15.119:0/1652641527 wait complete. 2024-03-28T18:50:10.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.942+0000 7f3a93a00640 1 Processor -- start 2024-03-28T18:50:10.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.942+0000 7f3a93a00640 1 -- start start 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c118090 con 0x7f3a8c075bd0 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c117f10 con 0x7f3a8c0764f0 2024-03-28T18:50:10.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a93a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c118210 con 0x7f3a8c1124e0 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a91000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53728/0 (socket says 172.21.15.119:53728) 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 -- 172.21.15.119:0/4231651027 learned_addr learned my addr 172.21.15.119:0/4231651027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a91000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55382/0 (socket says 172.21.15.119:55382) 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 -- 172.21.15.119:0/4231651027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 msgr2=0x7f3a8c1a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 -- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 msgr2=0x7f3a8c19e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a8be00640 1 -- 172.21.15.119:0/4231651027 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c1a3580 con 0x7f3a8c0764f0 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a91a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.943+0000 7f3a91000640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:10.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a8be00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00ea60 tx=0x7f3a7c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a89600640 1 -- 172.21.15.119:0/4231651027 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c00cde0 con 0x7f3a8c0764f0 2024-03-28T18:50:10.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a89600640 1 -- 172.21.15.119:0/4231651027 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c0035f0 con 0x7f3a8c0764f0 2024-03-28T18:50:10.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c1a3870 con 0x7f3a8c0764f0 2024-03-28T18:50:10.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a89600640 1 -- 172.21.15.119:0/4231651027 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c010670 con 0x7f3a8c0764f0 2024-03-28T18:50:10.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.944+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c1ab100 con 0x7f3a8c0764f0 2024-03-28T18:50:10.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:10 smithi131 ceph-mon[36065]: pgmap v186: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.947+0000 7f3a9667a740 1 Processor -- start 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.947+0000 7f3a89600640 1 -- 172.21.15.119:0/4231651027 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3a7c0037a0 con 0x7f3a8c0764f0 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.947+0000 7f3a9667a740 1 -- start start 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f34040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f36970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664e8feb0 con 0x557664f34580 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664e38a60 con 0x557664f2b9a0 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a9667a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664e8bf50 con 0x557664f2b080 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f34040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f36970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58462/0 (socket says 172.21.15.119:58462) 2024-03-28T18:50:10.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 -- 172.21.15.119:0/1233725736 learned_addr learned my addr 172.21.15.119:0/1233725736 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 -- 172.21.15.119:0/1233725736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 msgr2=0x557664f34040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f34040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 -- 172.21.15.119:0/1233725736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 msgr2=0x557664f36970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f36970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.948+0000 7f3a91a00640 1 -- 172.21.15.119:0/1233725736 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557664c6c7e0 con 0x557664f2b080 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a91a00640 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3a840106c0 tx=0x7f3a84010b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e26c63a4ec9630 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a89600640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 0x7f3a740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a7a000640 1 -- 172.21.15.119:0/1233725736 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8400e8d0 con 0x557664f2b080 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a7a000640 1 -- 172.21.15.119:0/1233725736 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a84012450 con 0x557664f2b080 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a7a000640 1 -- 172.21.15.119:0/1233725736 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8401c630 con 0x557664f2b080 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a89600640 1 -- 172.21.15.119:0/4231651027 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3a7c0cf110 con 0x7f3a8c0764f0 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a91000640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 0x7f3a740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a9667a740 1 -- 172.21.15.119:0/1233725736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 msgr2=0x557664f2b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.949+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3a840106c0 tx=0x7f3a84010b80 comp rx=0 tx=0).stop 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 -- 172.21.15.119:0/1233725736 shutdown_connections 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f36970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1233725736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f2b460 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 -- 172.21.15.119:0/1233725736 >> 172.21.15.119:0/1233725736 conn(0x557664f257b0 msgr2=0x557664f25b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 -- 172.21.15.119:0/1233725736 shutdown_connections 2024-03-28T18:50:10.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a91000640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 0x7f3a740a4eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009900 tx=0x7f3a80005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 -- 172.21.15.119:0/1233725736 wait complete. 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 -- start start 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.950+0000 7f3a9667a740 1 Processor -- start 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 0x557664f31260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f317c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664f37370 con 0x557664f34580 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664e4beb0 con 0x557664f2b080 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557664f374f0 con 0x557664f2b9a0 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a9667a740 10 client.0 _ll_register_callbacks cb 0x557664e40150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a91000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a91000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53754/0 (socket says 172.21.15.119:53754) 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a91000640 1 -- 172.21.15.119:0/1491494580 learned_addr learned my addr 172.21.15.119:0/1491494580 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 0x557664f31260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a91a00640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a8be00640 1 -- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 msgr2=0x557664f33ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.951+0000 7f3a8be00640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a8be00640 1 -- 172.21.15.119:0/1491494580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 msgr2=0x557664f317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a8be00640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a8be00640 1 -- 172.21.15.119:0/1491494580 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557664cccf70 con 0x557664f2b9a0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a91000640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a91a00640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a8be00640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 0x557664f31260 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0cc800 tx=0x7f3a7c0d2850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40fb34cd54a37382 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c09b4e0 con 0x557664f2b9a0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3a7c0cca80 con 0x557664f2b9a0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a84012450 con 0x557664f2b9a0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a7c09b070 con 0x557664f2b9a0 2024-03-28T18:50:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a9667a740 10 client.27181 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.952+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557664b64170 con 0x557664f2b9a0 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a84012450 con 0x557664f2b9a0 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3a7c097d80 con 0x557664f2b9a0 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 10 client.27181 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 10 client.27181 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 20 client.27181 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664f44630 0x557664f46a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 4 client.27181 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557664e9d3f0 con 0x557664f44630 2024-03-28T18:50:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.953+0000 7f3a91000640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664f44630 0x557664f46a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.954+0000 7f3a91000640 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664f44630 0x557664f46a20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3637b0942ba3272 server_cookie=3da05f3f6119c95e in_seq=0 out_seq=0 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.954+0000 7f3a6ea00640 10 client.27181 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.954+0000 7f3a6ea00640 1 -- 172.21.15.119:0/1491494580 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557664e9d3f0 con 0x557664f44630 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.954+0000 7f3a6ea00640 10 client.27181 handle_command_reply: tid=0 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.954+0000 7f3a9667a740 1 client.27181 shutdown 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 msgr2=0x557664f31260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 0x557664f31260 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0cc800 tx=0x7f3a7c0d2850 comp rx=0 tx=0).stop 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 shutdown_connections 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664f44630 0x557664f46a20 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f34580 0x557664f317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b9a0 0x557664f31260 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 --2- 172.21.15.119:0/1491494580 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b080 0x557664f33ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a6ea00640 0 client.27181 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 >> 172.21.15.119:0/1491494580 conn(0x557664f257b0 msgr2=0x557664f27500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 shutdown_connections 2024-03-28T18:50:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 1 -- 172.21.15.119:0/1491494580 wait complete. 2024-03-28T18:50:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.955+0000 7f3a9667a740 20 client.27181 trim_cache size 0 max 16384 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.972+0000 7f3a9667a740 1 Processor -- start 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.972+0000 7f3a9667a740 1 -- start start 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b3960 con 0x557664f2b080 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b52e0 con 0x557664f2b9a0 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a9667a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b3fc0 con 0x557664f34580 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55406/0 (socket says 172.21.15.119:55406) 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91000640 1 -- 172.21.15.119:0/251239566 learned_addr learned my addr 172.21.15.119:0/251239566 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53758/0 (socket says 172.21.15.119:53758) 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58472/0 (socket says 172.21.15.119:58472) 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 -- 172.21.15.119:0/251239566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 msgr2=0x557664f271c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 -- 172.21.15.119:0/251239566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 msgr2=0x557664f33990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a8be00640 1 -- 172.21.15.119:0/251239566 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557664f2c5a0 con 0x557664f2b9a0 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91000640 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.973+0000 7f3a91a00640 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.974+0000 7f3a8be00640 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0980e0 tx=0x7f3a7c003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4161ba5a689d35ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.974+0000 7f3a79600640 1 -- 172.21.15.119:0/251239566 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c09b250 con 0x557664f2b9a0 2024-03-28T18:50:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.974+0000 7f3a79600640 1 -- 172.21.15.119:0/251239566 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c09be10 con 0x557664f2b9a0 2024-03-28T18:50:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.974+0000 7f3a79600640 1 -- 172.21.15.119:0/251239566 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c097d80 con 0x557664f2b9a0 2024-03-28T18:50:10.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- 172.21.15.119:0/251239566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 msgr2=0x557664f33ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0980e0 tx=0x7f3a7c003f30 comp rx=0 tx=0).stop 2024-03-28T18:50:10.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- 172.21.15.119:0/251239566 shutdown_connections 2024-03-28T18:50:10.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f34580 0x557664f271c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f2b9a0 0x557664f33ed0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 --2- 172.21.15.119:0/251239566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b080 0x557664f33990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- 172.21.15.119:0/251239566 >> 172.21.15.119:0/251239566 conn(0x557664f257b0 msgr2=0x557664f26ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- 172.21.15.119:0/251239566 shutdown_connections 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- 172.21.15.119:0/251239566 wait complete. 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 -- start start 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.975+0000 7f3a9667a740 1 Processor -- start 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b9a0 0x557664f39c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f34580 0x557664f3a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b4c80 con 0x557664f2b9a0 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b4620 con 0x557664f34580 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1b1b20 con 0x557664f2b080 2024-03-28T18:50:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a9667a740 10 client.0 _ll_register_callbacks cb 0x7f3a7c0d2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58488/0 (socket says 172.21.15.119:58488) 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 -- 172.21.15.119:0/987767314 learned_addr learned my addr 172.21.15.119:0/987767314 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91a00640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f34580 0x557664f3a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a8be00640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b9a0 0x557664f39c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 -- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f34580 msgr2=0x557664f3a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f34580 0x557664f3a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 -- 172.21.15.119:0/987767314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b9a0 msgr2=0x557664f39c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b9a0 0x557664f39c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.976+0000 7f3a91000640 1 -- 172.21.15.119:0/987767314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557664f46d90 con 0x557664f2b080 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a91000640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a800044a0 tx=0x7f3a80009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7906908c57df7280 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a6ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80046070 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a7c09be10 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f3a800586f0 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a800588f0 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a9667a740 10 client.27187 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.977+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557664fa1270 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c09be10 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3a800035f0 con 0x557664f2b080 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 10 client.27187 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 10 client.27187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 20 client.27187 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664fa18f0 0x557664fa3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 4 client.27187 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557664e97740 con 0x557664fa18f0 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.978+0000 7f3a8be00640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664fa18f0 0x557664fa3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.979+0000 7f3a8be00640 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664fa18f0 0x557664fa3ce0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4984d0f737541e68 server_cookie=329c499c1ad8d24e in_seq=0 out_seq=0 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.979+0000 7f3a6ea00640 10 client.27187 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a6ea00640 1 -- 172.21.15.119:0/987767314 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557664e97740 con 0x557664fa18f0 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a6ea00640 10 client.27187 handle_command_reply: tid=0 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 client.27187 shutdown 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 msgr2=0x557664f39710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a800044a0 tx=0x7f3a80009260 comp rx=0 tx=0).stop 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 shutdown_connections 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557664fa18f0 0x557664fa3ce0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557664f34580 0x557664f3a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557664f2b9a0 0x557664f39c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 --2- 172.21.15.119:0/987767314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557664f2b080 0x557664f39710 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a6ea00640 0 client.27187 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 >> 172.21.15.119:0/987767314 conn(0x557664f257b0 msgr2=0x557664f27340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 shutdown_connections 2024-03-28T18:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 1 -- 172.21.15.119:0/987767314 wait complete. 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.980+0000 7f3a9667a740 20 client.27187 trim_cache size 0 max 16384 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "9fc4e523-9110-4b4b-9b91-145c348576d8", 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:50:10.985 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 msgr2=0x7f3a740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 0x7f3a740a4eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009900 tx=0x7f3a80005a20 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 msgr2=0x7f3a8c19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00ea60 tx=0x7f3a7c00ef20 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 shutdown_connections 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3a740a29f0 0x7f3a740a4eb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a8c1124e0 0x7f3a8c1a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3a8c0764f0 0x7f3a8c19ead0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 --2- 172.21.15.119:0/4231651027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a8c075bd0 0x7f3a8c19e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 >> 172.21.15.119:0/4231651027 conn(0x7f3a8c0ff2f0 msgr2=0x7f3a8c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:10.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 shutdown_connections 2024-03-28T18:50:10.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:10.981+0000 7f3a93a00640 1 -- 172.21.15.119:0/4231651027 wait complete. 2024-03-28T18:50:10.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:50:10.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:50:11.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 -- 172.21.15.119:0/1861267158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0109850 msgr2=0x7f65e0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 --2- 172.21.15.119:0/1861267158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0109850 0x7f65e0109c30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009900 tx=0x7f65d002bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 -- 172.21.15.119:0/1861267158 shutdown_connections 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 --2- 172.21.15.119:0/1861267158 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0104200 0x7f65e0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 --2- 172.21.15.119:0/1861267158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0103840 0x7f65e0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 --2- 172.21.15.119:0/1861267158 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0109850 0x7f65e0109c30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 -- 172.21.15.119:0/1861267158 >> 172.21.15.119:0/1861267158 conn(0x7f65e00ff2f0 msgr2=0x7f65e0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 -- 172.21.15.119:0/1861267158 shutdown_connections 2024-03-28T18:50:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.208+0000 7f65e4a00640 1 -- 172.21.15.119:0/1861267158 wait complete. 2024-03-28T18:50:11.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.209+0000 7f65e4a00640 1 Processor -- start 2024-03-28T18:50:11.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.209+0000 7f65e4a00640 1 -- start start 2024-03-28T18:50:11.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0104200 0x7f65e019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0109850 0x7f65e019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0113cb0 con 0x7f65e0109850 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0113b30 con 0x7f65e0104200 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65e4a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0113e30 con 0x7f65e0103840 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58510/0 (socket says 172.21.15.119:58510) 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65dde00640 1 -- 172.21.15.119:0/3937620030 learned_addr learned my addr 172.21.15.119:0/3937620030 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65d5400640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0104200 0x7f65e019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65de800640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0109850 0x7f65e019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65dde00640 1 -- 172.21.15.119:0/3937620030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0104200 msgr2=0x7f65e019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.210+0000 7f65dde00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0104200 0x7f65e019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65dde00640 1 -- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0109850 msgr2=0x7f65e019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65dde00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0109850 0x7f65e019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65dde00640 1 -- 172.21.15.119:0/3937620030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e019f180 con 0x7f65e0103840 2024-03-28T18:50:11.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65dde00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f65d002efd0 tx=0x7f65d00042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65d7400640 1 -- 172.21.15.119:0/3937620030 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0041070 con 0x7f65e0103840 2024-03-28T18:50:11.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65d7400640 1 -- 172.21.15.119:0/3937620030 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65d002ccc0 con 0x7f65e0103840 2024-03-28T18:50:11.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e019f410 con 0x7f65e0103840 2024-03-28T18:50:11.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65d7400640 1 -- 172.21.15.119:0/3937620030 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0003d40 con 0x7f65e0103840 2024-03-28T18:50:11.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.211+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e01a6d00 con 0x7f65e0103840 2024-03-28T18:50:11.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.214+0000 7f65d7400640 1 -- 172.21.15.119:0/3937620030 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f65d0044070 con 0x7f65e0103840 2024-03-28T18:50:11.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.214+0000 7f65e77ee740 1 Processor -- start 2024-03-28T18:50:11.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.214+0000 7f65e77ee740 1 -- start start 2024-03-28T18:50:11.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352500f20 0x55d3525013a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d35250c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d352465250 con 0x55d352501c00 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d35240da30 con 0x55d35250a290 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65e77ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3524612f0 con 0x55d352500f20 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352500f20 0x55d3525013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55442/0 (socket says 172.21.15.119:55442) 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65de800640 1 -- 172.21.15.119:0/3228859258 learned_addr learned my addr 172.21.15.119:0/3228859258 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65d5400640 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d35250c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65de800640 1 -- 172.21.15.119:0/3228859258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352500f20 msgr2=0x55d3525013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.215+0000 7f65de800640 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352500f20 0x55d3525013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65de800640 1 -- 172.21.15.119:0/3228859258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 msgr2=0x55d35250c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65de800640 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d35250c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65de800640 1 -- 172.21.15.119:0/3228859258 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3523f8690 con 0x55d352501c00 2024-03-28T18:50:11.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65d5400640 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d35250c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65d7400640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 0x7f65c40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65de800640 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009f50 tx=0x7f65d80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88daf7b29c0e4905 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65d7400640 1 -- 172.21.15.119:0/3937620030 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f65d00f07e0 con 0x7f65e0103840 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65dde00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 0x7f65c40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65cf000640 1 -- 172.21.15.119:0/3228859258 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8004d20 con 0x55d352501c00 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65cf000640 1 -- 172.21.15.119:0/3228859258 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65d8004050 con 0x55d352501c00 2024-03-28T18:50:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.216+0000 7f65cf000640 1 -- 172.21.15.119:0/3228859258 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8005280 con 0x55d352501c00 2024-03-28T18:50:11.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65dde00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 0x7f65c40a4eb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d352502150 tx=0x7f65d00f6530 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 -- 172.21.15.119:0/3228859258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 msgr2=0x55d352500950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f65d8009f50 tx=0x7f65d80118d0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 -- 172.21.15.119:0/3228859258 shutdown_connections 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d35250c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352500f20 0x55d3525013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 --2- 172.21.15.119:0/3228859258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352501c00 0x55d352500950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 -- 172.21.15.119:0/3228859258 >> 172.21.15.119:0/3228859258 conn(0x55d3524c8200 msgr2=0x55d3524fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 -- 172.21.15.119:0/3228859258 shutdown_connections 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.217+0000 7f65e77ee740 1 -- 172.21.15.119:0/3228859258 wait complete. 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 -- start start 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 Processor -- start 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d3525099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352507280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d35250d100 con 0x55d352500f20 2024-03-28T18:50:11.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d352420e80 con 0x55d35250a290 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d35250d280 con 0x55d352501c00 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.218+0000 7f65e77ee740 10 client.0 _ll_register_callbacks cb 0x55d352433130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d3525099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58526/0 (socket says 172.21.15.119:58526) 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 -- 172.21.15.119:0/1496130282 learned_addr learned my addr 172.21.15.119:0/1496130282 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65de800640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352507280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 -- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 msgr2=0x55d352507280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352507280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 -- 172.21.15.119:0/1496130282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 msgr2=0x55d3525099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d3525099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 -- 172.21.15.119:0/1496130282 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3522437d0 con 0x55d352501c00 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65de800640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352507280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65d5400640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f65c80126e0 tx=0x7f65c8012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c459de42c047e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65c3400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:11.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c80121d0 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f65c8004030 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.219+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65d8004050 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65c8004230 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 10 client.27199 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3521793f0 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65d8004050 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f65c80039e0 con 0x55d352501c00 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 10 client.27199 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 10 client.27199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 20 client.27199 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35251a230 0x55d35251c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 4 client.27199 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.220+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d352472790 con 0x55d35251a230 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.221+0000 7f65de800640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35251a230 0x55d35251c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.221+0000 7f65de800640 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35251a230 0x55d35251c620 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ad2d6e845008d8 server_cookie=cd68b90d76fac997 in_seq=0 out_seq=0 2024-03-28T18:50:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.221+0000 7f65c3400640 10 client.27199 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65c3400640 1 -- 172.21.15.119:0/1496130282 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d352472790 con 0x55d35251a230 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65c3400640 10 client.27199 handle_command_reply: tid=0 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 client.27199 shutdown 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 msgr2=0x55d352506d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f65c80126e0 tx=0x7f65c8012ba0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 shutdown_connections 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35251a230 0x55d35251c620 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352507280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352506d40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 --2- 172.21.15.119:0/1496130282 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d3525099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65c3400640 0 client.27199 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 >> 172.21.15.119:0/1496130282 conn(0x55d3524c8200 msgr2=0x55d3524fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 shutdown_connections 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 1 -- 172.21.15.119:0/1496130282 wait complete. 2024-03-28T18:50:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.222+0000 7f65e77ee740 20 client.27199 trim_cache size 0 max 16384 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 Processor -- start 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 -- start start 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352509470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 0x55d35250b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01af4e0 con 0x55d352500f20 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01b0e60 con 0x55d352501c00 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65e77ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01afb40 con 0x55d35250a290 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53802/0 (socket says 172.21.15.119:53802) 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65d5400640 1 -- 172.21.15.119:0/2668729445 learned_addr learned my addr 172.21.15.119:0/2668729445 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:11.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.240+0000 7f65de800640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352509470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65de800640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 0x55d35250b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 -- 172.21.15.119:0/2668729445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 msgr2=0x55d35250b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 0x55d35250b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 -- 172.21.15.119:0/2668729445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 msgr2=0x55d352509470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352509470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 -- 172.21.15.119:0/2668729445 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3521390f0 con 0x55d352501c00 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65de800640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 0x55d35250b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:50:11.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65d5400640 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009e00 tx=0x7f65c800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e22f44ef921692d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65ce600640 1 -- 172.21.15.119:0/2668729445 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8003fa0 con 0x55d352501c00 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65ce600640 1 -- 172.21.15.119:0/2668729445 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65c8010430 con 0x55d352501c00 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.241+0000 7f65ce600640 1 -- 172.21.15.119:0/2668729445 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8003850 con 0x55d352501c00 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- 172.21.15.119:0/2668729445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 msgr2=0x55d3525099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009e00 tx=0x7f65c800f790 comp rx=0 tx=0).stop 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- 172.21.15.119:0/2668729445 shutdown_connections 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d35250a290 0x55d35250b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d352501c00 0x55d3525099b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 --2- 172.21.15.119:0/2668729445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352509470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- 172.21.15.119:0/2668729445 >> 172.21.15.119:0/2668729445 conn(0x55d3524c8200 msgr2=0x55d3524fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- 172.21.15.119:0/2668729445 shutdown_connections 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- 172.21.15.119:0/2668729445 wait complete. 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.242+0000 7f65e77ee740 1 -- start start 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 Processor -- start 2024-03-28T18:50:11.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352504a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01b0800 con 0x55d352500f20 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01b01a0 con 0x55d35250a290 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01ad6a0 con 0x55d352501c00 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65e77ee740 10 client.0 _ll_register_callbacks cb 0x7f65c8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65d5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65de800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55484/0 (socket says 172.21.15.119:55484) 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65de800640 1 -- 172.21.15.119:0/4023914572 learned_addr learned my addr 172.21.15.119:0/4023914572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65d5400640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:58548/0 (socket says 172.21.15.119:58548) 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.243+0000 7f65de800640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352504a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 -- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 msgr2=0x55d352504a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352504a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 -- 172.21.15.119:0/4023914572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 msgr2=0x55d352503fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 -- 172.21.15.119:0/4023914572 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d35250ac50 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65de800640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65d5400640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f65c8003ed0 tx=0x7f65c80096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae6b6375def2e236 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65c3400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8033020 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65c8010430 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f65c8003800 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65c803b050 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65e77ee740 10 client.27202 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.244+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d35257e6b0 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65c8010430 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f65c802d450 con 0x55d352501c00 2024-03-28T18:50:11.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 10 client.27202 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 10 client.27202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 20 client.27202 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35257ece0 0x55d352581100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 4 client.27202 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d35246cae0 con 0x55d35257ece0 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.245+0000 7f65de800640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35257ece0 0x55d352581100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65de800640 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35257ece0 0x55d352581100 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139d79c3c0d4d46e server_cookie=518c1a94441b06ac in_seq=0 out_seq=0 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65c3400640 10 client.27202 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65c3400640 1 -- 172.21.15.119:0/4023914572 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d35246cae0 con 0x55d35257ece0 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65c3400640 10 client.27202 handle_command_reply: tid=0 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 client.27202 shutdown 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 msgr2=0x55d352504500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f65c8003ed0 tx=0x7f65c80096f0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 shutdown_connections 2024-03-28T18:50:11.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d35257ece0 0x55d352581100 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d35250a290 0x55d352504a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d352501c00 0x55d352504500 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65c3400640 0 client.27202 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 --2- 172.21.15.119:0/4023914572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d352500f20 0x55d352503fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 >> 172.21.15.119:0/4023914572 conn(0x55d3524c8200 msgr2=0x55d3525054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 shutdown_connections 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 1 -- 172.21.15.119:0/4023914572 wait complete. 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.246+0000 7f65e77ee740 20 client.27202 trim_cache size 0 max 16384 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:50:11.251 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: "9fc4e523-9110-4b4b-9b91-145c348576d8": { 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "9fc4e523-9110-4b4b-9b91-145c348576d8", 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:50:11.252 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 msgr2=0x7f65c40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 0x7f65c40a4eb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55d352502150 tx=0x7f65d00f6530 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 msgr2=0x7f65e019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f65d002efd0 tx=0x7f65d00042f0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 shutdown_connections 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f65c40a29f0 0x7f65c40a4eb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65e0109850 0x7f65e019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f65e0104200 0x7f65e019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 --2- 172.21.15.119:0/3937620030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65e0103840 0x7f65e019a190 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 >> 172.21.15.119:0/3937620030 conn(0x7f65e00ff2f0 msgr2=0x7f65e00ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 shutdown_connections 2024-03-28T18:50:11.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:11.247+0000 7f65e4a00640 1 -- 172.21.15.119:0/3937620030 wait complete. 2024-03-28T18:50:11.264 INFO:tasks.cephfs.filesystem:scrub status for tag:9fc4e523-9110-4b4b-9b91-145c348576d8 - {'path': '/', 'tag': '9fc4e523-9110-4b4b-9b91-145c348576d8', 'options': 'recursive,force'} 2024-03-28T18:50:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:11 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:50:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:11 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:11 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:50:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:11 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:11.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:11 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:50:11.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:11 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:11.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:11 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:50:11.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:11 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:11.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:11 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:50:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:11 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:11 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:50:11.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:11 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:12 smithi119 ceph-mon[31813]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:12.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:12 smithi196 ceph-mon[36300]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:12.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:12 smithi131 ceph-mon[36065]: pgmap v187: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:13 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:50:13.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:13 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:50:13.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:13 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:50:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:14 smithi119 ceph-mon[31813]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:14 smithi196 ceph-mon[36300]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:14 smithi131 ceph-mon[36065]: pgmap v188: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T18:50:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:16 smithi119 ceph-mon[31813]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:16 smithi196 ceph-mon[36300]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:16 smithi131 ceph-mon[36065]: pgmap v189: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T18:50:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:18 smithi119 ceph-mon[31813]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:50:18.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:18 smithi196 ceph-mon[36300]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:50:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:18 smithi131 ceph-mon[36065]: pgmap v190: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:50:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:20 smithi119 ceph-mon[31813]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:50:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:20 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:20 smithi196 ceph-mon[36300]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:50:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:20 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:20 smithi131 ceph-mon[36065]: pgmap v191: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:50:20.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:20 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:50:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:22 smithi119 ceph-mon[31813]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:22 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:22 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:22 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:50:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:22 smithi196 ceph-mon[36300]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:22 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:22 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:22 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:50:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:22 smithi131 ceph-mon[36065]: pgmap v192: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:22 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:50:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:22 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:50:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:22 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:50:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:24 smithi119 ceph-mon[31813]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:24 smithi196 ceph-mon[36300]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:24 smithi131 ceph-mon[36065]: pgmap v193: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:26 smithi119 ceph-mon[31813]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-28T18:50:26.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:26 smithi196 ceph-mon[36300]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-28T18:50:26.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:26 smithi131 ceph-mon[36065]: pgmap v194: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 op/s 2024-03-28T18:50:27.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:27 smithi196 ceph-mon[36300]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:50:27.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:27 smithi131 ceph-mon[36065]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:50:28.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:27 smithi119 ceph-mon[31813]: pgmap v195: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-28T18:50:30.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:30 smithi196 ceph-mon[36300]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:30 smithi131 ceph-mon[36065]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:30 smithi119 ceph-mon[31813]: pgmap v196: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:32.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:32 smithi196 ceph-mon[36300]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:32.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:32 smithi131 ceph-mon[36065]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:32 smithi119 ceph-mon[31813]: pgmap v197: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:34.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:34 smithi196 ceph-mon[36300]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:34.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:34 smithi131 ceph-mon[36065]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:34 smithi119 ceph-mon[31813]: pgmap v198: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:36.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:36 smithi196 ceph-mon[36300]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:36 smithi131 ceph-mon[36065]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:36 smithi119 ceph-mon[31813]: pgmap v199: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:38.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:38 smithi196 ceph-mon[36300]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:38.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:38 smithi131 ceph-mon[36065]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:38 smithi119 ceph-mon[31813]: pgmap v200: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:40.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:40 smithi196 ceph-mon[36300]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:40.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:40 smithi131 ceph-mon[36065]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:40 smithi119 ceph-mon[31813]: pgmap v201: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:41.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:50:41.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 -- 172.21.15.119:0/3259770900 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c103840 msgr2=0x7fbb3c103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 --2- 172.21.15.119:0/3259770900 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c103840 0x7fbb3c103cc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009930 tx=0x7fbb2402bc20 comp rx=0 tx=0).stop 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 -- 172.21.15.119:0/3259770900 shutdown_connections 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 --2- 172.21.15.119:0/3259770900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 0x7fbb3c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 --2- 172.21.15.119:0/3259770900 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c103840 0x7fbb3c103cc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 --2- 172.21.15.119:0/3259770900 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c109850 0x7fbb3c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 -- 172.21.15.119:0/3259770900 >> 172.21.15.119:0/3259770900 conn(0x7fbb3c0ff2f0 msgr2=0x7fbb3c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 -- 172.21.15.119:0/3259770900 shutdown_connections 2024-03-28T18:50:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.495+0000 7fbb41600640 1 -- 172.21.15.119:0/3259770900 wait complete. 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 Processor -- start 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 -- start start 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 0x7fbb3c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 0x7fbb3c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c113cb0 con 0x7fbb3c104200 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c113b30 con 0x7fbb3c103840 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb41600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c113e30 con 0x7fbb3c109850 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb33e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 0x7fbb3c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40812/0 (socket says 172.21.15.119:40812) 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3576167583 learned_addr learned my addr 172.21.15.119:0/3576167583 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.497+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 0x7fbb3c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 -- 172.21.15.119:0/3576167583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 msgr2=0x7fbb3c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 -- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 msgr2=0x7fbb3c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 0x7fbb3c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 -- 172.21.15.119:0/3576167583 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c19f1b0 con 0x7fbb3c109850 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 0x7fbb3c19ea80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009fd0 tx=0x7fbb3400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb33400640 1 -- 172.21.15.119:0/3576167583 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3400ce20 con 0x7fbb3c109850 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb33400640 1 -- 172.21.15.119:0/3576167583 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb34005020 con 0x7fbb3c109850 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb33400640 1 -- 172.21.15.119:0/3576167583 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34010720 con 0x7fbb3c109850 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.498+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c19f4a0 con 0x7fbb3c109850 2024-03-28T18:50:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.499+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c1a6d30 con 0x7fbb3c109850 2024-03-28T18:50:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.501+0000 7fbb33400640 1 -- 172.21.15.119:0/3576167583 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbb34005540 con 0x7fbb3c109850 2024-03-28T18:50:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 Processor -- start 2024-03-28T18:50:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 -- start start 2024-03-28T18:50:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c243950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c1a8250 con 0x55bf0c244c00 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.502+0000 7fbb44373740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c150a30 con 0x55bf0c243f20 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb44373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c1a42f0 con 0x55bf0c24d290 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40816/0 (socket says 172.21.15.119:40816) 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3465136932 learned_addr learned my addr 172.21.15.119:0/3465136932 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c243950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3465136932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 msgr2=0x55bf0c24f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3465136932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 msgr2=0x55bf0c243950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c243950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3465136932 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf0c13b690 con 0x55bf0c243f20 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb33400640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 0x7fbb200a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c243950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb33400640 1 -- 172.21.15.119:0/3576167583 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbb340cf570 con 0x7fbb3c109850 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.503+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0115f0 tx=0x7fbb2c011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0bee052241010e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 0x7fbb200a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb2b400640 1 -- 172.21.15.119:0/3465136932 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c01e070 con 0x55bf0c243f20 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb2b400640 1 -- 172.21.15.119:0/3465136932 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c00fdd0 con 0x55bf0c243f20 2024-03-28T18:50:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb2b400640 1 -- 172.21.15.119:0/3465136932 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c00e3b0 con 0x55bf0c243f20 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb44373740 1 -- 172.21.15.119:0/3465136932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 msgr2=0x55bf0c2443a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb44373740 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0115f0 tx=0x7fbb2c011ab0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.504+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 0x7fbb200a4eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009900 tx=0x7fbb24005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 -- 172.21.15.119:0/3465136932 shutdown_connections 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c2443a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 --2- 172.21.15.119:0/3465136932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c243950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 -- 172.21.15.119:0/3465136932 >> 172.21.15.119:0/3465136932 conn(0x55bf0c20b200 msgr2=0x55bf0c23e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 -- 172.21.15.119:0/3465136932 shutdown_connections 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 -- 172.21.15.119:0/3465136932 wait complete. 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 -- start start 2024-03-28T18:50:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.505+0000 7fbb44373740 1 Processor -- start 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c249ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c250100 con 0x55bf0c244c00 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c163e80 con 0x55bf0c243f20 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf0c250280 con 0x55bf0c24d290 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb44373740 10 client.0 _ll_register_callbacks cb 0x55bf0c176130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41700/0 (socket says 172.21.15.119:41700) 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40820/0 (socket says 172.21.15.119:40820) 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3b400640 1 -- 172.21.15.119:0/1330772234 learned_addr learned my addr 172.21.15.119:0/1330772234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c249ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 -- 172.21.15.119:0/1330772234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 msgr2=0x55bf0c24a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 -- 172.21.15.119:0/1330772234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 msgr2=0x55bf0c249ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c249ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.506+0000 7fbb3aa00640 1 -- 172.21.15.119:0/1330772234 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf0bf867d0 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c01c040 tx=0x7fbb2c019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b275736fb01beff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c01e050 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbb2c00fad0 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb2c00fdd0 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2c033050 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb44373740 10 client.27242 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bf0bebc3f0 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.507+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c00fdd0 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbb2c004a30 con 0x55bf0c243f20 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 10 client.27242 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:41.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 10 client.27242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 20 client.27242 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c25d0d0 0x55bf0c25f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 4 client.27242 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf0c1b5790 con 0x55bf0c25d0d0 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.508+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c25d0d0 0x55bf0c25f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.509+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c25d0d0 0x55bf0c25f4c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2f04f4eb20edced server_cookie=f0d33fe936c87d83 in_seq=0 out_seq=0 2024-03-28T18:50:41.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.509+0000 7fbb1fe00640 10 client.27242 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb1fe00640 1 -- 172.21.15.119:0/1330772234 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf0c1b5790 con 0x55bf0c25d0d0 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb1fe00640 10 client.27242 handle_command_reply: tid=0 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 client.27242 shutdown 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 msgr2=0x55bf0c24c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c01c040 tx=0x7fbb2c019660 comp rx=0 tx=0).stop 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 shutdown_connections 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c25d0d0 0x55bf0c25f4c0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c24a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c249ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb1fe00640 0 client.27242 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 --2- 172.21.15.119:0/1330772234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c24c910 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 >> 172.21.15.119:0/1330772234 conn(0x55bf0c20b200 msgr2=0x55bf0c2406c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 shutdown_connections 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 1 -- 172.21.15.119:0/1330772234 wait complete. 2024-03-28T18:50:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.510+0000 7fbb44373740 20 client.27242 trim_cache size 0 max 16384 2024-03-28T18:50:41.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 Processor -- start 2024-03-28T18:50:41.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 -- start start 2024-03-28T18:50:41.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c243f20 0x55bf0c24e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c24d290 0x55bf0c24c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1af560 con 0x55bf0c24d290 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1b0ee0 con 0x55bf0c244c00 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb44373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1afbc0 con 0x55bf0c243f20 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb33e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb33e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40836/0 (socket says 172.21.15.119:40836) 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb33e00640 1 -- 172.21.15.119:0/1850588766 learned_addr learned my addr 172.21.15.119:0/1850588766 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c243f20 0x55bf0c24e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.528+0000 7fbb3b400640 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c24d290 0x55bf0c24c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 -- 172.21.15.119:0/1850588766 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c243f20 msgr2=0x55bf0c24e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c243f20 0x55bf0c24e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 -- 172.21.15.119:0/1850588766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c24d290 msgr2=0x55bf0c24c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c24d290 0x55bf0c24c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 -- 172.21.15.119:0/1850588766 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf0be7c0f0 con 0x55bf0c244c00 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb33e00640 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbb2402c230 tx=0x7fbb24043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a49426d762c8ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb2aa00640 1 -- 172.21.15.119:0/1850588766 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24004840 con 0x55bf0c244c00 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb2aa00640 1 -- 172.21.15.119:0/1850588766 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb240049e0 con 0x55bf0c244c00 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.529+0000 7fbb2aa00640 1 -- 172.21.15.119:0/1850588766 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003850 con 0x55bf0c244c00 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 -- 172.21.15.119:0/1850588766 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 msgr2=0x55bf0c24ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbb2402c230 tx=0x7fbb24043000 comp rx=0 tx=0).stop 2024-03-28T18:50:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 -- 172.21.15.119:0/1850588766 shutdown_connections 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c24d290 0x55bf0c24c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c244c00 0x55bf0c24ebc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 --2- 172.21.15.119:0/1850588766 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c243f20 0x55bf0c24e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 -- 172.21.15.119:0/1850588766 >> 172.21.15.119:0/1850588766 conn(0x55bf0c20b200 msgr2=0x55bf0c23e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 -- 172.21.15.119:0/1850588766 shutdown_connections 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.530+0000 7fbb44373740 1 -- 172.21.15.119:0/1850588766 wait complete. 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 -- start start 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 Processor -- start 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c2514b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c2519f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1b0880 con 0x55bf0c244c00 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1b0220 con 0x55bf0c243f20 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1ad720 con 0x55bf0c24d290 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb44373740 10 client.0 _ll_register_callbacks cb 0x7fbb2c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb33e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c2514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb3aa00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40850/0 (socket says 172.21.15.119:40850) 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.531+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3113548762 learned_addr learned my addr 172.21.15.119:0/3113548762 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c2519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3113548762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 msgr2=0x55bf0c2519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c2519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3113548762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 msgr2=0x55bf0c2514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c2514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 -- 172.21.15.119:0/3113548762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf0c252030 con 0x55bf0c243f20 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3b400640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c2519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb3aa00640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009250 tx=0x7fbb2c0196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b319912cdab0c2e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c03c060 con 0x55bf0c243f20 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb240049e0 con 0x55bf0c243f20 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fbb2c00fad0 con 0x55bf0c243f20 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb2c01e040 con 0x55bf0c243f20 2024-03-28T18:50:41.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb44373740 10 client.27248 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.532+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bf0c2c1e50 con 0x55bf0c243f20 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb240049e0 con 0x55bf0c243f20 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbb2c019b60 con 0x55bf0c243f20 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 10 client.27248 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 20 client.27248 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c2c23f0 0x55bf0c2c4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 4 client.27248 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bf0c1afae0 con 0x55bf0c2c23f0 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.533+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c2c23f0 0x55bf0c2c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb1fe00640 10 client.27248 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb33e00640 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c2c23f0 0x55bf0c2c4810 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cafdb0907ecb3be server_cookie=2b7af256fb8d420c in_seq=0 out_seq=0 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb1fe00640 1 -- 172.21.15.119:0/3113548762 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bf0c1afae0 con 0x55bf0c2c23f0 2024-03-28T18:50:41.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb1fe00640 10 client.27248 handle_command_reply: tid=0 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 client.27248 shutdown 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 msgr2=0x55bf0c255130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009250 tx=0x7fbb2c0196d0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 shutdown_connections 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55bf0c2c23f0 0x55bf0c2c4810 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf0c24d290 0x55bf0c2519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bf0c244c00 0x55bf0c2514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb1fe00640 0 client.27248 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 --2- 172.21.15.119:0/3113548762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bf0c243f20 0x55bf0c255130 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 >> 172.21.15.119:0/3113548762 conn(0x55bf0c20b200 msgr2=0x55bf0c23d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 shutdown_connections 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 1 -- 172.21.15.119:0/3113548762 wait complete. 2024-03-28T18:50:41.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.534+0000 7fbb44373740 20 client.27248 trim_cache size 0 max 16384 2024-03-28T18:50:41.540 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:41.540 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:50:41.540 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:50:41.540 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 msgr2=0x7fbb200a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 0x7fbb200a4eb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009900 tx=0x7fbb24005fd0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 msgr2=0x7fbb3c19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 0x7fbb3c19ea80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009fd0 tx=0x7fbb3400ef10 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 shutdown_connections 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbb200a29f0 0x7fbb200a4eb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c109850 0x7fbb3c19ea80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3c104200 0x7fbb3c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 --2- 172.21.15.119:0/3576167583 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbb3c103840 0x7fbb3c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 >> 172.21.15.119:0/3576167583 conn(0x7fbb3c0ff2f0 msgr2=0x7fbb3c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 shutdown_connections 2024-03-28T18:50:41.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.535+0000 7fbb41600640 1 -- 172.21.15.119:0/3576167583 wait complete. 2024-03-28T18:50:41.553 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:50:41.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.761+0000 7f9a77200640 1 -- 172.21.15.119:0/1019670274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78107a60 msgr2=0x7f9a78107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.761+0000 7f9a77200640 1 --2- 172.21.15.119:0/1019670274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78107a60 0x7f9a78107ee0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009930 tx=0x7f9a6002bc20 comp rx=0 tx=0).stop 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 -- 172.21.15.119:0/1019670274 shutdown_connections 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 --2- 172.21.15.119:0/1019670274 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a78108420 0x7f9a78114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 --2- 172.21.15.119:0/1019670274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78107a60 0x7f9a78107ee0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 --2- 172.21.15.119:0/1019670274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7810da70 0x7f9a7810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 -- 172.21.15.119:0/1019670274 >> 172.21.15.119:0/1019670274 conn(0x7f9a780771c0 msgr2=0x7f9a780795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 -- 172.21.15.119:0/1019670274 shutdown_connections 2024-03-28T18:50:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.762+0000 7f9a77200640 1 -- 172.21.15.119:0/1019670274 wait complete. 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 Processor -- start 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 -- start start 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a78107a60 0x7f9a7819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a7810da70 0x7f9a781a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a78117ea0 con 0x7f9a78107a60 2024-03-28T18:50:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a78117d20 con 0x7f9a7810da70 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.763+0000 7f9a77200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a78118020 con 0x7f9a78108420 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41732/0 (socket says 172.21.15.119:41732) 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 -- 172.21.15.119:0/3402447279 learned_addr learned my addr 172.21.15.119:0/3402447279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a75e00640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a78107a60 0x7f9a7819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a76800640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a7810da70 0x7f9a781a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 -- 172.21.15.119:0/3402447279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a7810da70 msgr2=0x7f9a781a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a7810da70 0x7f9a781a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 -- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a78107a60 msgr2=0x7f9a7819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a78107a60 0x7f9a7819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 -- 172.21.15.119:0/3402447279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a781a3350 con 0x7f9a78108420 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6d400640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9a60005ab0 tx=0x7f9a60005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.764+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60043020 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.765+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a781a35e0 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.765+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a60005000 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.765+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60003ab0 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.765+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a781aaed0 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.766+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a78068800 con 0x7f9a78108420 2024-03-28T18:50:41.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.768+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9a60046020 con 0x7f9a78108420 2024-03-28T18:50:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.769+0000 7f9a6f400640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 0x7f9a5c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.769+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9a600f0ca0 con 0x7f9a78108420 2024-03-28T18:50:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.769+0000 7f9a75e00640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 0x7f9a5c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.770+0000 7f9a75e00640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 0x7f9a5c0a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009b50 tx=0x7f9a68005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:41.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:41.773+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9a6004d020 con 0x7f9a78108420 2024-03-28T18:50:42.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.011+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a7819f6b0 con 0x7f9a78108420 2024-03-28T18:50:42.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.012+0000 7f9a6f400640 1 -- 172.21.15.119:0/3402447279 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f9a600bda50 con 0x7f9a78108420 2024-03-28T18:50:42.013 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:50:42.013 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:50:42.013 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:50:42.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 msgr2=0x7f9a5c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 0x7f9a5c0a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009b50 tx=0x7f9a68005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 msgr2=0x7f9a7819e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9a60005ab0 tx=0x7f9a60005e80 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 shutdown_connections 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9a5c0a29f0 0x7f9a5c0a4eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9a7810da70 0x7f9a781a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a78108420 0x7f9a7819e8a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 --2- 172.21.15.119:0/3402447279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a78107a60 0x7f9a7819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 >> 172.21.15.119:0/3402447279 conn(0x7f9a780771c0 msgr2=0x7f9a780777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 shutdown_connections 2024-03-28T18:50:42.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.015+0000 7f9a77200640 1 -- 172.21.15.119:0/3402447279 wait complete. 2024-03-28T18:50:42.033 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:42.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 -- 172.21.15.119:0/2086828253 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec10ddb0 msgr2=0x7f84ec1101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 --2- 172.21.15.119:0/2086828253 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec10ddb0 0x7f84ec1101c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009ca0 tx=0x7f84e802c190 comp rx=0 tx=0).stop 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 -- 172.21.15.119:0/2086828253 shutdown_connections 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 --2- 172.21.15.119:0/2086828253 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec10ddb0 0x7f84ec1101c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 --2- 172.21.15.119:0/2086828253 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec074390 0x7f84ec10d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 --2- 172.21.15.119:0/2086828253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec073a70 0x7f84ec073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 -- 172.21.15.119:0/2086828253 >> 172.21.15.119:0/2086828253 conn(0x7f84ec0fd000 msgr2=0x7f84ec0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 -- 172.21.15.119:0/2086828253 shutdown_connections 2024-03-28T18:50:42.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.244+0000 7f84f3200640 1 -- 172.21.15.119:0/2086828253 wait complete. 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 Processor -- start 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 -- start start 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 0x7f84ec19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec113b10 con 0x7f84ec10ddb0 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec113990 con 0x7f84ec073a70 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f3200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec113c90 con 0x7f84ec074390 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 0x7f84ec19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.245+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40882/0 (socket says 172.21.15.119:40882) 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41758/0 (socket says 172.21.15.119:41758) 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1400640 1 -- 172.21.15.119:0/2067066126 learned_addr learned my addr 172.21.15.119:0/2067066126 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 -- 172.21.15.119:0/2067066126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 msgr2=0x7f84ec19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 -- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 msgr2=0x7f84ec19e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 0x7f84ec19e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 -- 172.21.15.119:0/2067066126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec19eff0 con 0x7f84ec073a70 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1400640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f2800640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 0x7f84ec19e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f1e00640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f84e000e9f0 tx=0x7f84e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84e6a00640 1 -- 172.21.15.119:0/2067066126 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e000ccc0 con 0x7f84ec073a70 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84e6a00640 1 -- 172.21.15.119:0/2067066126 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e0010910 con 0x7f84ec073a70 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec19f2e0 con 0x7f84ec073a70 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.246+0000 7f84e6a00640 1 -- 172.21.15.119:0/2067066126 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0018960 con 0x7f84ec073a70 2024-03-28T18:50:42.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.247+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec1a6b70 con 0x7f84ec073a70 2024-03-28T18:50:42.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84e6a00640 1 -- 172.21.15.119:0/2067066126 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f84e0010ac0 con 0x7f84ec073a70 2024-03-28T18:50:42.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 Processor -- start 2024-03-28T18:50:42.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 -- start start 2024-03-28T18:50:42.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f8969668b0 0x55f896966c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896972230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8968ca2c0 con 0x55f8969668b0 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f896872c90 con 0x55f89696fde0 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f742a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8968c6360 con 0x55f896967260 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.250+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896972230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f8969668b0 0x55f896966c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41762/0 (socket says 172.21.15.119:41762) 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 -- 172.21.15.119:0/206772281 learned_addr learned my addr 172.21.15.119:0/206772281 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 -- 172.21.15.119:0/206772281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 msgr2=0x55f896972230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896972230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 -- 172.21.15.119:0/206772281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f8969668b0 msgr2=0x55f896966c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f8969668b0 0x55f896966c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 -- 172.21.15.119:0/206772281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f896897080 con 0x55f896967260 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84f1e00640 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f84e00cc8b0 tx=0x7f84e00d7930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c5e460bf118c238 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84e6a00640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 0x7f84d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.251+0000 7f84db000640 1 -- 172.21.15.119:0/206772281 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00e2070 con 0x55f896967260 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84db000640 1 -- 172.21.15.119:0/206772281 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e009b930 con 0x55f896967260 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84db000640 1 -- 172.21.15.119:0/206772281 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00c34d0 con 0x55f896967260 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84e6a00640 1 -- 172.21.15.119:0/2067066126 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84e00cf2e0 con 0x7f84ec073a70 2024-03-28T18:50:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f2800640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 0x7f84d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 -- 172.21.15.119:0/206772281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 msgr2=0x55f89696f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f84e00cc8b0 tx=0x7f84e00d7930 comp rx=0 tx=0).stop 2024-03-28T18:50:42.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 -- 172.21.15.119:0/206772281 shutdown_connections 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896972230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f896967260 0x55f89696f810 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.252+0000 7f84f742a740 1 --2- 172.21.15.119:0/206772281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f8969668b0 0x55f896966c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f742a740 1 -- 172.21.15.119:0/206772281 >> 172.21.15.119:0/206772281 conn(0x55f896960f80 msgr2=0x55f896961360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f2800640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 0x7f84d40a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009bf0 tx=0x7f84e8005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f742a740 1 -- 172.21.15.119:0/206772281 shutdown_connections 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f742a740 1 -- 172.21.15.119:0/206772281 wait complete. 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f742a740 1 -- start start 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.253+0000 7f84f742a740 1 Processor -- start 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f89696cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f89696fde0 0x55f89696d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f896972d50 con 0x55f896967260 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8968860d0 con 0x55f8969668b0 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f896972ed0 con 0x55f89696fde0 2024-03-28T18:50:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f742a740 10 client.0 _ll_register_callbacks cb 0x55f89688ae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40894/0 (socket says 172.21.15.119:40894) 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f89696cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 -- 172.21.15.119:0/1062566976 learned_addr learned my addr 172.21.15.119:0/1062566976 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f2800640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f89696fde0 0x55f89696d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 -- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f89696fde0 msgr2=0x55f89696d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f89696fde0 0x55f89696d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 -- 172.21.15.119:0/1062566976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 msgr2=0x55f89696cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f89696cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.254+0000 7f84f1e00640 1 -- 172.21.15.119:0/1062566976 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8965de3f0 con 0x55f8969668b0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f1e00640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f84e00d7e30 tx=0x7f84e000ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79ce82fb62ba5463 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00c6530 con 0x55f8969668b0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84e00ccbf0 con 0x55f8969668b0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84e009b930 con 0x55f8969668b0 2024-03-28T18:50:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84e00c6f60 con 0x55f8969668b0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 10 client.27266 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f8966a87d0 con 0x55f8969668b0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e009b930 con 0x55f8969668b0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f84e00c7100 con 0x55f8969668b0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 10 client.27266 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 10 client.27266 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 20 client.27266 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f89697fb30 0x55f896981f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 4 client.27266 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.255+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8968d7800 con 0x55f89697fb30 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.256+0000 7f84f1400640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f89697fb30 0x55f896981f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.256+0000 7f84f1400640 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f89697fb30 0x55f896981f20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f8bffdea2682a02 server_cookie=607f7a1ea058938b in_seq=0 out_seq=0 2024-03-28T18:50:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.256+0000 7f84cf400640 10 client.27266 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:42.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84cf400640 1 -- 172.21.15.119:0/1062566976 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f8968d7800 con 0x55f89697fb30 2024-03-28T18:50:42.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84cf400640 10 client.27266 handle_command_reply: tid=0 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 client.27266 shutdown 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 msgr2=0x55f89696f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f84e00d7e30 tx=0x7f84e000ce60 comp rx=0 tx=0).stop 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 shutdown_connections 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f89697fb30 0x55f896981f20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f89696fde0 0x55f89696d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f89696cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84cf400640 0 client.27266 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 --2- 172.21.15.119:0/1062566976 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f8969668b0 0x55f89696f730 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 >> 172.21.15.119:0/1062566976 conn(0x55f896960f80 msgr2=0x55f896962cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 shutdown_connections 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.257+0000 7f84f742a740 1 -- 172.21.15.119:0/1062566976 wait complete. 2024-03-28T18:50:42.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.258+0000 7f84f742a740 20 client.27266 trim_cache size 0 max 16384 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 Processor -- start 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 -- start start 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f896967260 0x55f89696bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f89696fde0 0x55f89696f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1af4a0 con 0x55f89696fde0 2024-03-28T18:50:42.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1b0e20 con 0x55f896967260 2024-03-28T18:50:42.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.271+0000 7f84f742a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1afb00 con 0x55f8969668b0 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f2800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f89696fde0 0x55f89696f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41772/0 (socket says 172.21.15.119:41772) 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 -- 172.21.15.119:0/4219901767 learned_addr learned my addr 172.21.15.119:0/4219901767 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1400640 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f896967260 0x55f89696bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 -- 172.21.15.119:0/4219901767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f896967260 msgr2=0x55f89696bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f896967260 0x55f89696bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 -- 172.21.15.119:0/4219901767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f89696fde0 msgr2=0x55f89696f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f89696fde0 0x55f89696f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 -- 172.21.15.119:0/4219901767 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f896987730 con 0x55f8969668b0 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84f1e00640 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e000c6a0 tx=0x7f84e000ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=398254f65127710b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84da600640 1 -- 172.21.15.119:0/4219901767 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00c74e0 con 0x55f8969668b0 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84da600640 1 -- 172.21.15.119:0/4219901767 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e00d7e60 con 0x55f8969668b0 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.272+0000 7f84da600640 1 -- 172.21.15.119:0/4219901767 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00c5620 con 0x55f8969668b0 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- 172.21.15.119:0/4219901767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 msgr2=0x55f89697c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e000c6a0 tx=0x7f84e000ced0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- 172.21.15.119:0/4219901767 shutdown_connections 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f89696fde0 0x55f89696f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f896967260 0x55f89696bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 --2- 172.21.15.119:0/4219901767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f89697c130 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- 172.21.15.119:0/4219901767 >> 172.21.15.119:0/4219901767 conn(0x55f896960f80 msgr2=0x55f896962c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- 172.21.15.119:0/4219901767 shutdown_connections 2024-03-28T18:50:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- 172.21.15.119:0/4219901767 wait complete. 2024-03-28T18:50:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.273+0000 7f84f742a740 1 -- start start 2024-03-28T18:50:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 Processor -- start 2024-03-28T18:50:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f896984e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896982e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1b07c0 con 0x55f896967260 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1b0160 con 0x55f89696fde0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec1ad660 con 0x55f8969668b0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f742a740 10 client.0 _ll_register_callbacks cb 0x7f84e00e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f896984e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41782/0 (socket says 172.21.15.119:41782) 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f1e00640 1 -- 172.21.15.119:0/3358262504 learned_addr learned my addr 172.21.15.119:0/3358262504 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.274+0000 7f84f2800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896982e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 -- 172.21.15.119:0/3358262504 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 msgr2=0x55f896982e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896982e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 -- 172.21.15.119:0/3358262504 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 msgr2=0x55f896984e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f896984e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 -- 172.21.15.119:0/3358262504 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f896983680 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1400640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f896984e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f1e00640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f84e00982b0 tx=0x7f84e000ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31bcf9cf94761acc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84cf400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e00c57f0 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f84e00a3d90 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84e00d7e60 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84e00c5bc0 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f742a740 10 client.27244 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.275+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f8969dbc50 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84e00d7e60 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f84e009cbb0 con 0x55f8969668b0 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 10 client.27244 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 10 client.27244 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 20 client.27244 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f8969dc080 0x55f8969de4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 4 client.27244 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f8968d1b50 con 0x55f8969dc080 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f1400640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f8969dc080 0x55f8969de4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84f1400640 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f8969dc080 0x55f8969de4a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1db9abd69a9b114b server_cookie=e89dc61a79fe93e7 in_seq=0 out_seq=0 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.276+0000 7f84cf400640 10 client.27244 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84cf400640 1 -- 172.21.15.119:0/3358262504 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f8968d1b50 con 0x55f8969dc080 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84cf400640 10 client.27244 handle_command_reply: tid=0 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 client.27244 shutdown 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 msgr2=0x55f8969848e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f84e00982b0 tx=0x7f84e000ce60 comp rx=0 tx=0).stop 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 shutdown_connections 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f8969dc080 0x55f8969de4a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f89696fde0 0x55f896982e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f896967260 0x55f896984e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 --2- 172.21.15.119:0/3358262504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8969668b0 0x55f8969848e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84cf400640 0 client.27244 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 >> 172.21.15.119:0/3358262504 conn(0x55f896960f80 msgr2=0x55f896960160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 shutdown_connections 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 1 -- 172.21.15.119:0/3358262504 wait complete. 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.277+0000 7f84f742a740 20 client.27244 trim_cache size 0 max 16384 2024-03-28T18:50:42.282 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 msgr2=0x7f84d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 0x7f84d40a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009bf0 tx=0x7f84e8005d80 comp rx=0 tx=0).stop 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 msgr2=0x7f84ec19a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f84e000e9f0 tx=0x7f84e000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 shutdown_connections 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f84d40a2970 0x7f84d40a4e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ec10ddb0 0x7f84ec19e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84ec074390 0x7f84ec19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 --2- 172.21.15.119:0/2067066126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f84ec073a70 0x7f84ec19a050 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 >> 172.21.15.119:0/2067066126 conn(0x7f84ec0fd000 msgr2=0x7f84ec0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 shutdown_connections 2024-03-28T18:50:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.278+0000 7f84f3200640 1 -- 172.21.15.119:0/2067066126 wait complete. 2024-03-28T18:50:42.296 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:42.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:50:42.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.508+0000 7f2dece00640 1 -- 172.21.15.119:0/296876093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 msgr2=0x7f2de8103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.508+0000 7f2dece00640 1 --2- 172.21.15.119:0/296876093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de8103ba0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009930 tx=0x7f2dd402bc30 comp rx=0 tx=0).stop 2024-03-28T18:50:42.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 -- 172.21.15.119:0/296876093 shutdown_connections 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 --2- 172.21.15.119:0/296876093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de81040e0 0x7f2de81105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 --2- 172.21.15.119:0/296876093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de8103ba0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 --2- 172.21.15.119:0/296876093 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de8109730 0x7f2de8109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 -- 172.21.15.119:0/296876093 >> 172.21.15.119:0/296876093 conn(0x7f2de80ff1d0 msgr2=0x7f2de81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 -- 172.21.15.119:0/296876093 shutdown_connections 2024-03-28T18:50:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.509+0000 7f2dece00640 1 -- 172.21.15.119:0/296876093 wait complete. 2024-03-28T18:50:42.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.510+0000 7f2dece00640 1 Processor -- start 2024-03-28T18:50:42.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.510+0000 7f2dece00640 1 -- start start 2024-03-28T18:50:42.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de81040e0 0x7f2de819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de8109730 0x7f2de819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8113b50 con 0x7f2de8109730 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81139d0 con 0x7f2de81040e0 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2dece00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de8113cd0 con 0x7f2de8103720 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41814/0 (socket says 172.21.15.119:41814) 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 -- 172.21.15.119:0/220803743 learned_addr learned my addr 172.21.15.119:0/220803743 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de6a00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de81040e0 0x7f2de819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7e00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de8109730 0x7f2de819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 -- 172.21.15.119:0/220803743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de81040e0 msgr2=0x7f2de819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de81040e0 0x7f2de819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 -- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de8109730 msgr2=0x7f2de819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de8109730 0x7f2de819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.511+0000 7f2de7400640 1 -- 172.21.15.119:0/220803743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de819f060 con 0x7f2de8103720 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2de7400640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e9f0 tx=0x7f2dd800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2ddfe00640 1 -- 172.21.15.119:0/220803743 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800cd80 con 0x7f2de8103720 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de819f350 con 0x7f2de8103720 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2ddfe00640 1 -- 172.21.15.119:0/220803743 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8010910 con 0x7f2de8103720 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2ddfe00640 1 -- 172.21.15.119:0/220803743 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd80188f0 con 0x7f2de8103720 2024-03-28T18:50:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.512+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de81a6be0 con 0x7f2de8103720 2024-03-28T18:50:42.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.515+0000 7f2ddfe00640 1 -- 172.21.15.119:0/220803743 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2dd8010ac0 con 0x7f2de8103720 2024-03-28T18:50:42.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.515+0000 7f2df1010740 1 Processor -- start 2024-03-28T18:50:42.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 -- start start 2024-03-28T18:50:42.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0bfd2c0 con 0x5644d0c9a260 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0ba5c90 con 0x5644d0c998b0 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2df1010740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0bf9360 con 0x5644d0ca2de0 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:40946/0 (socket says 172.21.15.119:40946) 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de7e00640 1 -- 172.21.15.119:0/1001389246 learned_addr learned my addr 172.21.15.119:0/1001389246 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41820/0 (socket says 172.21.15.119:41820) 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.516+0000 7f2de7400640 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 -- 172.21.15.119:0/1001389246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 msgr2=0x5644d0c99c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 -- 172.21.15.119:0/1001389246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 msgr2=0x5644d0ca2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 -- 172.21.15.119:0/1001389246 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644d0bca080 con 0x5644d0ca2de0 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de7e00640 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2ddfe00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 0x7f2dcc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de6a00640 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036040 tx=0x7f2dd4009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369f4cbe4e5265e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2ddfe00640 1 -- 172.21.15.119:0/220803743 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dd80cf1a0 con 0x7f2de8103720 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2de7e00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 0x7f2dcc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2ddcc00640 1 -- 172.21.15.119:0/1001389246 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4045070 con 0x5644d0ca2de0 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2ddcc00640 1 -- 172.21.15.119:0/1001389246 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd402c650 con 0x5644d0ca2de0 2024-03-28T18:50:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.517+0000 7f2ddcc00640 1 -- 172.21.15.119:0/1001389246 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4004b20 con 0x5644d0ca2de0 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 -- 172.21.15.119:0/1001389246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 msgr2=0x5644d0ca5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2de7e00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 0x7f2dcc0a4eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2de0011890 tx=0x7f2de0009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036040 tx=0x7f2dd4009560 comp rx=0 tx=0).stop 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 -- 172.21.15.119:0/1001389246 shutdown_connections 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0ca5230 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 --2- 172.21.15.119:0/1001389246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0c99c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 -- 172.21.15.119:0/1001389246 >> 172.21.15.119:0/1001389246 conn(0x5644d0c93f80 msgr2=0x5644d0c94360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 -- 172.21.15.119:0/1001389246 shutdown_connections 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.518+0000 7f2df1010740 1 -- 172.21.15.119:0/1001389246 wait complete. 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 -- start start 2024-03-28T18:50:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 Processor -- start 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0c9fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0ca5cf0 con 0x5644d0ca2de0 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0bb90d0 con 0x5644d0c9a260 2024-03-28T18:50:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644d0ca5e70 con 0x5644d0c998b0 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2df1010740 10 client.0 _ll_register_callbacks cb 0x5644d0bbde80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.519+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0c9fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41834/0 (socket says 172.21.15.119:41834) 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de7400640 1 -- 172.21.15.119:0/2362534067 learned_addr learned my addr 172.21.15.119:0/2362534067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de7e00640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 -- 172.21.15.119:0/2362534067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 msgr2=0x5644d0ca26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 -- 172.21.15.119:0/2362534067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 msgr2=0x5644d0c9ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 -- 172.21.15.119:0/2362534067 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644d09113f0 con 0x5644d0c9a260 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de7400640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2de6a00640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0c9fa40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036040 tx=0x7f2dd402cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e41ce396b8d69cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2dd1600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd404e070 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2dd402c650 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.520+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dd40050e0 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dd4059050 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 10 client.27284 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5644d09db7d0 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dd402c650 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2dd404ab90 con 0x5644d0c9a260 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 10 client.27284 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 10 client.27284 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 20 client.27284 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0cb2aa0 0x5644d0cb4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 4 client.27284 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.521+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644d0c0a800 con 0x5644d0cb2aa0 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.522+0000 7f2de7400640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0cb2aa0 0x5644d0cb4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.522+0000 7f2de7400640 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0cb2aa0 0x5644d0cb4e90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e82fd69598a9df2 server_cookie=902d6237666329fa in_seq=0 out_seq=0 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.522+0000 7f2dd1600640 10 client.27284 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2dd1600640 1 -- 172.21.15.119:0/2362534067 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5644d0c0a800 con 0x5644d0cb2aa0 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2dd1600640 10 client.27284 handle_command_reply: tid=0 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 client.27284 shutdown 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 msgr2=0x5644d0c9fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0c9fa40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036040 tx=0x7f2dd402cab0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 shutdown_connections 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0cb2aa0 0x5644d0cb4e90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0c9fa40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 --2- 172.21.15.119:0/2362534067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0ca26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2dd1600640 0 client.27284 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.523+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 >> 172.21.15.119:0/2362534067 conn(0x5644d0c93f80 msgr2=0x5644d0c95cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.524+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 shutdown_connections 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.524+0000 7f2df1010740 1 -- 172.21.15.119:0/2362534067 wait complete. 2024-03-28T18:50:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.524+0000 7f2df1010740 20 client.27284 trim_cache size 0 max 16384 2024-03-28T18:50:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 Processor -- start 2024-03-28T18:50:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 -- start start 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0ca2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81af330 con 0x5644d0c9a260 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81b0cb0 con 0x5644d0c998b0 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.537+0000 7f2df1010740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81af990 con 0x5644d0ca2de0 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41836/0 (socket says 172.21.15.119:41836) 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 -- 172.21.15.119:0/3442502118 learned_addr learned my addr 172.21.15.119:0/3442502118 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7400640 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0ca2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de6a00640 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 -- 172.21.15.119:0/3442502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 msgr2=0x5644d0ca2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0ca2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 -- 172.21.15.119:0/3442502118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 msgr2=0x5644d0ca26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 -- 172.21.15.119:0/3442502118 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644d0c99ff0 con 0x5644d0ca2de0 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.538+0000 7f2de7e00640 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009fa0 tx=0x7f2de0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439ac6893e56ca8b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2dd3e00640 1 -- 172.21.15.119:0/3442502118 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0004b20 con 0x5644d0ca2de0 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2dd3e00640 1 -- 172.21.15.119:0/3442502118 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de0004cc0 con 0x5644d0ca2de0 2024-03-28T18:50:42.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2dd3e00640 1 -- 172.21.15.119:0/3442502118 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0028530 con 0x5644d0ca2de0 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 -- 172.21.15.119:0/3442502118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 msgr2=0x5644d0c9ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009fa0 tx=0x7f2de0016000 comp rx=0 tx=0).stop 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 -- 172.21.15.119:0/3442502118 shutdown_connections 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0ca2de0 0x5644d0c9ad50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0c9a260 0x5644d0ca26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 --2- 172.21.15.119:0/3442502118 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c998b0 0x5644d0ca2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 -- 172.21.15.119:0/3442502118 >> 172.21.15.119:0/3442502118 conn(0x5644d0c93f80 msgr2=0x5644d0c9a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 -- 172.21.15.119:0/3442502118 shutdown_connections 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.539+0000 7f2df1010740 1 -- 172.21.15.119:0/3442502118 wait complete. 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 -- start start 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 Processor -- start 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0cb7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0cb5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81b0650 con 0x5644d0ca2de0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81afff0 con 0x5644d0c9a260 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81ad4f0 con 0x5644d0c998b0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.540+0000 7f2df1010740 10 client.0 _ll_register_callbacks cb 0x7f2dd80d2b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41842/0 (socket says 172.21.15.119:41842) 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 -- 172.21.15.119:0/3113850826 learned_addr learned my addr 172.21.15.119:0/3113850826 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de6a00640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0cb7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7e00640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0cb5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 -- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 msgr2=0x5644d0cb7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0cb7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 -- 172.21.15.119:0/3113850826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 msgr2=0x5644d0cb5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0cb5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 -- 172.21.15.119:0/3113850826 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644d0cb6220 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de6a00640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0cb7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2de7400640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2dd809b4f0 tx=0x7f2dd80d15f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=526c2f252f676e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2dd1600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd809b330 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f2dd80d2cc0 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.541+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2de0004cc0 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2dd809b150 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 10 client.27262 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5644d0d0ea80 con 0x5644d0c998b0 2024-03-28T18:50:42.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de0004cc0 con 0x5644d0c998b0 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2dd809ac40 con 0x5644d0c998b0 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 10 client.27262 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 10 client.27262 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 20 client.27262 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0d0f180 0x5644d0d115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 4 client.27262 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5644d0c04b50 con 0x5644d0d0f180 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.542+0000 7f2de6a00640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0d0f180 0x5644d0d115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2de6a00640 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0d0f180 0x5644d0d115a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91b151d1da227db1 server_cookie=b995d53e51716b1f in_seq=0 out_seq=0 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2dd1600640 10 client.27262 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2dd1600640 1 -- 172.21.15.119:0/3113850826 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5644d0c04b50 con 0x5644d0d0f180 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2dd1600640 10 client.27262 handle_command_reply: tid=0 2024-03-28T18:50:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 client.27262 shutdown 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 msgr2=0x5644d0cb75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2dd809b4f0 tx=0x7f2dd80d15f0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 shutdown_connections 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5644d0d0f180 0x5644d0d115a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5644d0ca2de0 0x5644d0cb5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5644d0c9a260 0x5644d0cb7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 --2- 172.21.15.119:0/3113850826 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5644d0c998b0 0x5644d0cb75a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2dd1600640 0 client.27262 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 >> 172.21.15.119:0/3113850826 conn(0x5644d0c93f80 msgr2=0x5644d0c9aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 shutdown_connections 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.543+0000 7f2df1010740 1 -- 172.21.15.119:0/3113850826 wait complete. 2024-03-28T18:50:42.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2df1010740 20 client.27262 trim_cache size 0 max 16384 2024-03-28T18:50:42.550 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:42.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 msgr2=0x7f2dcc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 0x7f2dcc0a4eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2de0011890 tx=0x7f2de0009500 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 msgr2=0x7f2de819a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800e9f0 tx=0x7f2dd800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 shutdown_connections 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2dcc0a29f0 0x7f2dcc0a4eb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de8109730 0x7f2de819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2de81040e0 0x7f2de819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 --2- 172.21.15.119:0/220803743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2de8103720 0x7f2de819a070 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 >> 172.21.15.119:0/220803743 conn(0x7f2de80ff1d0 msgr2=0x7f2de80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 shutdown_connections 2024-03-28T18:50:42.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.544+0000 7f2dece00640 1 -- 172.21.15.119:0/220803743 wait complete. 2024-03-28T18:50:42.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:42 smithi119 ceph-mon[31813]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:42.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:42 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3402447279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:42.563 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:42.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:50:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:42 smithi131 ceph-mon[36065]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:42.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:42 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3402447279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.771+0000 7fa0ed600640 1 -- 172.21.15.119:0/2994876165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e8075bd0 msgr2=0x7fa0e8075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.771+0000 7fa0ed600640 1 --2- 172.21.15.119:0/2994876165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e8075bd0 0x7fa0e8075fb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009930 tx=0x7fa0d802bc20 comp rx=0 tx=0).stop 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 -- 172.21.15.119:0/2994876165 shutdown_connections 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 --2- 172.21.15.119:0/2994876165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e81124e0 0x7fa0e81148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 --2- 172.21.15.119:0/2994876165 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e80764f0 0x7fa0e8111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 --2- 172.21.15.119:0/2994876165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e8075bd0 0x7fa0e8075fb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 -- 172.21.15.119:0/2994876165 >> 172.21.15.119:0/2994876165 conn(0x7fa0e80ff2f0 msgr2=0x7fa0e8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 -- 172.21.15.119:0/2994876165 shutdown_connections 2024-03-28T18:50:42.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.772+0000 7fa0ed600640 1 -- 172.21.15.119:0/2994876165 wait complete. 2024-03-28T18:50:42.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 Processor -- start 2024-03-28T18:50:42.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 -- start start 2024-03-28T18:50:42.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e8075bd0 0x7fa0e819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 0x7fa0e81a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8118090 con 0x7fa0e8075bd0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8117f10 con 0x7fa0e81124e0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0ed600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e8118210 con 0x7fa0e80764f0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 0x7fa0e81a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41848/0 (socket says 172.21.15.119:41848) 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e8075bd0 0x7fa0e819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.773+0000 7fa0dfe00640 1 -- 172.21.15.119:0/473091008 learned_addr learned my addr 172.21.15.119:0/473091008 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 -- 172.21.15.119:0/473091008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 msgr2=0x7fa0e819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 -- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e8075bd0 msgr2=0x7fa0e819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e8075bd0 0x7fa0e819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 -- 172.21.15.119:0/473091008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e81a3590 con 0x7fa0e81124e0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0e7400640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 0x7fa0e81a2e60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0005fd0 tx=0x7fa0e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0df400640 1 -- 172.21.15.119:0/473091008 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000cdf0 con 0x7fa0e81124e0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0df400640 1 -- 172.21.15.119:0/473091008 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0e0005020 con 0x7fa0e81124e0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e81a3880 con 0x7fa0e81124e0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0df400640 1 -- 172.21.15.119:0/473091008 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00106d0 con 0x7fa0e81124e0 2024-03-28T18:50:42.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.774+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e81ab110 con 0x7fa0e81124e0 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0df400640 1 -- 172.21.15.119:0/473091008 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa0e00054d0 con 0x7fa0e81124e0 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 Processor -- start 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 -- start start 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777ee3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777eef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777e472c0 con 0x55d777ee36d0 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777defc90 con 0x55d777eecc00 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777e43360 con 0x55d777ee4080 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777ee3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41854/0 (socket says 172.21.15.119:41854) 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.778+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3071090685 learned_addr learned my addr 172.21.15.119:0/3071090685 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777eef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3071090685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 msgr2=0x55d777eef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777eef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3071090685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 msgr2=0x55d777ee3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777ee3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3071090685 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d777e14080 con 0x55d777ee4080 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777eef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:42.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ab0 tx=0x7fa0d8005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5bae819acd69cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0df400640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0d7400640 1 -- 172.21.15.119:0/3071090685 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8045070 con 0x55d777ee4080 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0d7400640 1 -- 172.21.15.119:0/3071090685 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d802c640 con 0x55d777ee4080 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.779+0000 7fa0d7400640 1 -- 172.21.15.119:0/3071090685 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8004b00 con 0x55d777ee4080 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0df400640 1 -- 172.21.15.119:0/473091008 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa0e00cf4f0 con 0x7fa0e81124e0 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3071090685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 msgr2=0x55d777eec630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ab0 tx=0x7fa0d8005e40 comp rx=0 tx=0).stop 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3071090685 shutdown_connections 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777eef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eec630 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3071090685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777ee3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3071090685 >> 172.21.15.119:0/3071090685 conn(0x55d777eddda0 msgr2=0x55d777ede180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00097f0 tx=0x7fa0d0015040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3071090685 shutdown_connections 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.780+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3071090685 wait complete. 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 -- start start 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 Processor -- start 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777ee9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777ee9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777eefb10 con 0x55d777ee4080 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777e030d0 con 0x55d777ee36d0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.781+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d777eefc90 con 0x55d777eecc00 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0f03a6740 10 client.0 _ll_register_callbacks cb 0x55d777e07e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41028/0 (socket says 172.21.15.119:41028) 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3924652007 learned_addr learned my addr 172.21.15.119:0/3924652007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777ee9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e7400640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777ee9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 msgr2=0x55d777ee9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777ee9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3924652007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 msgr2=0x55d777ee9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777ee9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3924652007 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d777b5b3f0 con 0x55d777ee36d0 2024-03-28T18:50:42.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009900 tx=0x7fa0d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a90e5d2d38bdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.782+0000 7fa0cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8045070 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa0d80050e0 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0d802c640 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0d8058050 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0f03a6740 10 client.27302 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d777c257d0 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d802c640 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa0d804ac70 con 0x55d777ee36d0 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0f03a6740 10 client.27302 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.783+0000 7fa0f03a6740 10 client.27302 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0f03a6740 20 client.27302 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777efc820 0x55d777efec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0f03a6740 4 client.27302 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:42.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d777e54800 con 0x55d777efc820 2024-03-28T18:50:42.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777efc820 0x55d777efec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777efc820 0x55d777efec10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5094837b4a378d74 server_cookie=611d2e348115e489 in_seq=0 out_seq=0 2024-03-28T18:50:42.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.784+0000 7fa0cfe00640 10 client.27302 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.785+0000 7fa0cfe00640 1 -- 172.21.15.119:0/3924652007 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d777e54800 con 0x55d777efc820 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.785+0000 7fa0cfe00640 10 client.27302 handle_command_reply: tid=0 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.785+0000 7fa0f03a6740 1 client.27302 shutdown 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.785+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 msgr2=0x55d777eec480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009900 tx=0x7fa0d80046c0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 shutdown_connections 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777efc820 0x55d777efec10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777ee9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777ee9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3924652007 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777eec480 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0cfe00640 0 client.27302 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:42.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 >> 172.21.15.119:0/3924652007 conn(0x55d777eddda0 msgr2=0x55d777edfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 shutdown_connections 2024-03-28T18:50:42.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3924652007 wait complete. 2024-03-28T18:50:42.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.786+0000 7fa0f03a6740 20 client.27302 trim_cache size 0 max 16384 2024-03-28T18:50:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.799+0000 7fa0f03a6740 1 Processor -- start 2024-03-28T18:50:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.799+0000 7fa0f03a6740 1 -- start start 2024-03-28T18:50:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.799+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777eebf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777eec480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b3950 con 0x55d777ee4080 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b52d0 con 0x55d777ee36d0 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b3fb0 con 0x55d777eecc00 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777eec480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777eebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41030/0 (socket says 172.21.15.119:41030) 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3397333856 learned_addr learned my addr 172.21.15.119:0/3397333856 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777eec480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41870/0 (socket says 172.21.15.119:41870) 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3397333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 msgr2=0x55d777eec480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777eec480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3397333856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 msgr2=0x55d777eebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777eebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 -- 172.21.15.119:0/3397333856 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d777ed9700 con 0x55d777ee36d0 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777eebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.800+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ab0 tx=0x7fa0d8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb26c46defbbdef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0d6a00640 1 -- 172.21.15.119:0/3397333856 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8045040 con 0x55d777ee36d0 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0d6a00640 1 -- 172.21.15.119:0/3397333856 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d804a450 con 0x55d777ee36d0 2024-03-28T18:50:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0d6a00640 1 -- 172.21.15.119:0/3397333856 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8005260 con 0x55d777ee36d0 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3397333856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 msgr2=0x55d777ee4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8005ab0 tx=0x7fa0d8005a20 comp rx=0 tx=0).stop 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3397333856 shutdown_connections 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777eecc00 0x55d777eec480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee4080 0x55d777eebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/3397333856 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777ee36d0 0x55d777ee4b90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3397333856 >> 172.21.15.119:0/3397333856 conn(0x55d777eddda0 msgr2=0x55d777ee4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3397333856 shutdown_connections 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.801+0000 7fa0f03a6740 1 -- 172.21.15.119:0/3397333856 wait complete. 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 -- start start 2024-03-28T18:50:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 Processor -- start 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777eff3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777f56d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b4c70 con 0x55d777ee36d0 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b4610 con 0x55d777eecc00 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.802+0000 7fa0f03a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e81b1b10 con 0x55d777ee4080 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0f03a6740 10 client.0 _ll_register_callbacks cb 0x7fa0d0009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0e6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777eff3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41872/0 (socket says 172.21.15.119:41872) 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 -- 172.21.15.119:0/2911480966 learned_addr learned my addr 172.21.15.119:0/2911480966 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0e7400640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777f56d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 -- 172.21.15.119:0/2911480966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 msgr2=0x55d777f56d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777f56d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 -- 172.21.15.119:0/2911480966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 msgr2=0x55d777eff3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777eff3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 -- 172.21.15.119:0/2911480966 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d777f001e0 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0e7400640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777f56d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777eff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0dfe00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005d00 tx=0x7fa0d0005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cadfd45d2e625a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.803+0000 7fa0cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00044e0 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fa0d0003ac0 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d777eea0e0 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0d003d050 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 10 client.27280 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d777f58420 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d777eea0e0 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa0d0004910 con 0x55d777ee4080 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 10 client.27280 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 10 client.27280 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:50:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 20 client.27280 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777f58b80 0x55d777f5afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 4 client.27280 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.804+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d777e4eb50 con 0x55d777f58b80 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777f58b80 0x55d777f5afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0e6a00640 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777f58b80 0x55d777f5afa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35390dbb9485a497 server_cookie=df7b5e1dbad60466 in_seq=0 out_seq=0 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0cfe00640 10 client.27280 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0cfe00640 1 -- 172.21.15.119:0/2911480966 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d777e4eb50 con 0x55d777f58b80 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0cfe00640 10 client.27280 handle_command_reply: tid=0 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0f03a6740 1 client.27280 shutdown 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 msgr2=0x55d777eff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005d00 tx=0x7fa0d0005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 shutdown_connections 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.805+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55d777f58b80 0x55d777f5afa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d777eecc00 0x55d777f56d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d777ee4080 0x55d777eff910 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 --2- 172.21.15.119:0/2911480966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d777ee36d0 0x55d777eff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 >> 172.21.15.119:0/2911480966 conn(0x55d777eddda0 msgr2=0x55d777edceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0cfe00640 0 client.27280 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 shutdown_connections 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 1 -- 172.21.15.119:0/2911480966 wait complete. 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0f03a6740 20 client.27280 trim_cache size 0 max 16384 2024-03-28T18:50:42.810 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 msgr2=0x7fa0c80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00097f0 tx=0x7fa0d0015040 comp rx=0 tx=0).stop 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 msgr2=0x7fa0e81a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 0x7fa0e81a2e60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0005fd0 tx=0x7fa0e000ef10 comp rx=0 tx=0).stop 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 shutdown_connections 2024-03-28T18:50:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa0e81124e0 0x7fa0e81a2e60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa0e80764f0 0x7fa0e819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 --2- 172.21.15.119:0/473091008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0e8075bd0 0x7fa0e819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.806+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 >> 172.21.15.119:0/473091008 conn(0x7fa0e80ff2f0 msgr2=0x7fa0e81016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.807+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 shutdown_connections 2024-03-28T18:50:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:42.807+0000 7fa0ed600640 1 -- 172.21.15.119:0/473091008 wait complete. 2024-03-28T18:50:42.824 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:42.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:50:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:42 smithi196 ceph-mon[36300]: pgmap v202: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:42.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:42 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3402447279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 -- 172.21.15.119:0/824536924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 msgr2=0x7f91001148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 --2- 172.21.15.119:0/824536924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001148d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009cd0 tx=0x7f90f802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 -- 172.21.15.119:0/824536924 shutdown_connections 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 --2- 172.21.15.119:0/824536924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001148d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 --2- 172.21.15.119:0/824536924 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f91000764f0 0x7f9100111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 --2- 172.21.15.119:0/824536924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9100075bd0 0x7f9100075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 -- 172.21.15.119:0/824536924 >> 172.21.15.119:0/824536924 conn(0x7f91000ff2f0 msgr2=0x7f9100101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 -- 172.21.15.119:0/824536924 shutdown_connections 2024-03-28T18:50:43.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.029+0000 7f9108200640 1 -- 172.21.15.119:0/824536924 wait complete. 2024-03-28T18:50:43.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 Processor -- start 2024-03-28T18:50:43.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 -- start start 2024-03-28T18:50:43.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 0x7f910019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 0x7f910019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9100118040 con 0x7f91000764f0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9100117ec0 con 0x7f9100075bd0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9108200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001181c0 con 0x7f91001124e0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9106200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9105800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 0x7f910019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9104e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 0x7f910019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9106200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001a2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41898/0 (socket says 172.21.15.119:41898) 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.031+0000 7f9106200640 1 -- 172.21.15.119:0/3360099824 learned_addr learned my addr 172.21.15.119:0/3360099824 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 -- 172.21.15.119:0/3360099824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 msgr2=0x7f91001a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 -- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 msgr2=0x7f910019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 0x7f910019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 -- 172.21.15.119:0/3360099824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91001a3530 con 0x7f9100075bd0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9104e00640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 0x7f910019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9105800640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 0x7f910019e540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9f0 tx=0x7f90f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f90fe000640 1 -- 172.21.15.119:0/3360099824 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400ccc0 con 0x7f9100075bd0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f90fe000640 1 -- 172.21.15.119:0/3360099824 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f4010910 con 0x7f9100075bd0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f90fe000640 1 -- 172.21.15.119:0/3360099824 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4018960 con 0x7f9100075bd0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.032+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91001a3820 con 0x7f9100075bd0 2024-03-28T18:50:43.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.033+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91001ab0b0 con 0x7f9100075bd0 2024-03-28T18:50:43.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.035+0000 7f910b015740 1 Processor -- start 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f90fe000640 1 -- 172.21.15.119:0/3360099824 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f90f4010ac0 con 0x7f9100075bd0 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 -- start start 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 0x5592f20d7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f202f2c0 con 0x5592f20cc080 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f1fd7c90 con 0x5592f20cb6d0 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f910b015740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f202b360 con 0x5592f20d4c00 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f9106200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f9104e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 0x5592f20d7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f9106200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41066/0 (socket says 172.21.15.119:41066) 2024-03-28T18:50:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f9106200640 1 -- 172.21.15.119:0/1984363860 learned_addr learned my addr 172.21.15.119:0/1984363860 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.036+0000 7f9105800640 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 -- 172.21.15.119:0/1984363860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 msgr2=0x5592f20cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 -- 172.21.15.119:0/1984363860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 msgr2=0x5592f20d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 -- 172.21.15.119:0/1984363860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592f1ffc080 con 0x5592f20d4c00 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9106200640 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f9104e00640 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 0x5592f20d7050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009f20 tx=0x7f90ec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00cbb32077c07f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f90fe000640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 0x7f90e80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f90f2a00640 1 -- 172.21.15.119:0/1984363860 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec004030 con 0x5592f20d4c00 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f90f2a00640 1 -- 172.21.15.119:0/1984363860 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90ec005500 con 0x5592f20d4c00 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.037+0000 7f90f2a00640 1 -- 172.21.15.119:0/1984363860 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec004e10 con 0x5592f20d4c00 2024-03-28T18:50:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f90fe000640 1 -- 172.21.15.119:0/3360099824 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f90f40cf1c0 con 0x7f9100075bd0 2024-03-28T18:50:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f9106200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 0x7f90e80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 -- 172.21.15.119:0/1984363860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 msgr2=0x5592f20d7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 0x5592f20d7050 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009f20 tx=0x7f90ec0118d0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 -- 172.21.15.119:0/1984363860 shutdown_connections 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20d4c00 0x5592f20d7050 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 --2- 172.21.15.119:0/1984363860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cb6d0 0x5592f20cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.038+0000 7f910b015740 1 -- 172.21.15.119:0/1984363860 >> 172.21.15.119:0/1984363860 conn(0x5592f20c5da0 msgr2=0x5592f20c6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.039+0000 7f9106200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 0x7f90e80a4eb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009a60 tx=0x7f90f8005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.039+0000 7f910b015740 1 -- 172.21.15.119:0/1984363860 shutdown_connections 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.039+0000 7f910b015740 1 -- 172.21.15.119:0/1984363860 wait complete. 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.039+0000 7f910b015740 1 -- start start 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.039+0000 7f910b015740 1 Processor -- start 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20d4c00 0x5592f20d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f20d7b70 con 0x5592f20cc080 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f1feb0d0 con 0x5592f20d4c00 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592f20d7cf0 con 0x5592f20cb6d0 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f910b015740 10 client.0 _ll_register_callbacks cb 0x5592f1fefe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9104e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41914/0 (socket says 172.21.15.119:41914) 2024-03-28T18:50:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 -- 172.21.15.119:0/973669040 learned_addr learned my addr 172.21.15.119:0/973669040 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9106200640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20d4c00 0x5592f20d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 -- 172.21.15.119:0/973669040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20d4c00 msgr2=0x5592f20d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20d4c00 0x5592f20d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 -- 172.21.15.119:0/973669040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 msgr2=0x5592f20d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 -- 172.21.15.119:0/973669040 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592f1d433f0 con 0x5592f20cb6d0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.040+0000 7f9105800640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9c0 tx=0x7f90f40d2f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1cfc251d6813759 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f409b640 con 0x5592f20cb6d0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f90f409acc0 con 0x5592f20cb6d0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90ec005500 con 0x5592f20cb6d0 2024-03-28T18:50:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90f409b1d0 con 0x5592f20cb6d0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 10 client.27292 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5592f1e0d7d0 con 0x5592f20cb6d0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90ec005500 con 0x5592f20cb6d0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f90f40afba0 con 0x5592f20cb6d0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 10 client.27292 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 10 client.27292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 20 client.27292 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f20e4a80 0x5592f20e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 4 client.27292 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.041+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592f203c800 con 0x5592f20e4a80 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.042+0000 7f9104e00640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f20e4a80 0x5592f20e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.042+0000 7f9104e00640 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f20e4a80 0x5592f20e6e70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73c9e31d47957e0b server_cookie=dfe04c187dcb470 in_seq=0 out_seq=0 2024-03-28T18:50:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.042+0000 7f90e7400640 10 client.27292 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.043+0000 7f90e7400640 1 -- 172.21.15.119:0/973669040 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5592f203c800 con 0x5592f20e4a80 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.043+0000 7f90e7400640 10 client.27292 handle_command_reply: tid=0 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.043+0000 7f910b015740 1 client.27292 shutdown 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.043+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 msgr2=0x5592f20d4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.043+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9c0 tx=0x7f90f40d2f80 comp rx=0 tx=0).stop 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 shutdown_connections 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f20e4a80 0x5592f20e6e70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20d4c00 0x5592f20d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20cc080 0x5592f20d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f90e7400640 0 client.27292 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 --2- 172.21.15.119:0/973669040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d4520 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 >> 172.21.15.119:0/973669040 conn(0x5592f20c5da0 msgr2=0x5592f20c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 shutdown_connections 2024-03-28T18:50:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 1 -- 172.21.15.119:0/973669040 wait complete. 2024-03-28T18:50:43.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.044+0000 7f910b015740 20 client.27292 trim_cache size 0 max 16384 2024-03-28T18:50:43.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.057+0000 7f910b015740 1 Processor -- start 2024-03-28T18:50:43.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.057+0000 7f910b015740 1 -- start start 2024-03-28T18:50:43.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20d4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b3a30 con 0x5592f20d4c00 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b53b0 con 0x5592f20cc080 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f910b015740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b4090 con 0x5592f20cb6d0 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9106200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41930/0 (socket says 172.21.15.119:41930) 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 -- 172.21.15.119:0/4037442778 learned_addr learned my addr 172.21.15.119:0/4037442778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9104e00640 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20d4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 -- 172.21.15.119:0/4037442778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 msgr2=0x5592f20d4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20d4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 -- 172.21.15.119:0/4037442778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 msgr2=0x5592f20ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 -- 172.21.15.119:0/4037442778 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592f20d7bc0 con 0x5592f20cb6d0 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.058+0000 7f9105800640 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000c00 tx=0x7f90f400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83eabfb4c1f71a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f90f2000640 1 -- 172.21.15.119:0/4037442778 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f409b3b0 con 0x5592f20cb6d0 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f90f2000640 1 -- 172.21.15.119:0/4037442778 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f409f910 con 0x5592f20cb6d0 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f90f2000640 1 -- 172.21.15.119:0/4037442778 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f409acc0 con 0x5592f20cb6d0 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 -- 172.21.15.119:0/4037442778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 msgr2=0x5592f20d3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000c00 tx=0x7f90f400cea0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 -- 172.21.15.119:0/4037442778 shutdown_connections 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20d4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 --2- 172.21.15.119:0/4037442778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20d3fe0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 -- 172.21.15.119:0/4037442778 >> 172.21.15.119:0/4037442778 conn(0x5592f20c5da0 msgr2=0x5592f20c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 -- 172.21.15.119:0/4037442778 shutdown_connections 2024-03-28T18:50:43.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.059+0000 7f910b015740 1 -- 172.21.15.119:0/4037442778 wait complete. 2024-03-28T18:50:43.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 -- start start 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 Processor -- start 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20db500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b4d50 con 0x5592f20d4c00 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b46f0 con 0x5592f20cc080 2024-03-28T18:50:43.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91001b1bf0 con 0x5592f20cb6d0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.060+0000 7f910b015740 10 client.0 _ll_register_callbacks cb 0x7f90f40d2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9104e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41946/0 (socket says 172.21.15.119:41946) 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9104e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41104/0 (socket says 172.21.15.119:41104) 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 -- 172.21.15.119:0/3823852725 learned_addr learned my addr 172.21.15.119:0/3823852725 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9106200640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20db500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 -- 172.21.15.119:0/3823852725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 msgr2=0x5592f20dafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 -- 172.21.15.119:0/3823852725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 msgr2=0x5592f20db500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20db500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 -- 172.21.15.119:0/3823852725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592f20dbd00 con 0x5592f20cb6d0 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9104e00640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:43.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f9105800640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f90f409b570 tx=0x7f90f40afdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ccd75a4d698c613 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f90e7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f409b3b0 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f90f409aec0 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.061+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90f409f910 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90f4018b00 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 10 client.27298 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5592f2141210 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90f409f910 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f90f409cc90 con 0x5592f20cb6d0 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 10 client.27298 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 10 client.27298 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 20 client.27298 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f2141500 0x5592f21438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 4 client.27298 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:43.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592f2036b50 con 0x5592f2141500 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.062+0000 7f9104e00640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f2141500 0x5592f21438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f9104e00640 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f2141500 0x5592f21438f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fe194ad464a7711 server_cookie=78c4f80fb3e1a8c0 in_seq=0 out_seq=0 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f90e7400640 10 client.27298 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f90e7400640 1 -- 172.21.15.119:0/3823852725 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592f2036b50 con 0x5592f2141500 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f90e7400640 10 client.27298 handle_command_reply: tid=0 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 client.27298 shutdown 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 msgr2=0x5592f20dcc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f90f409b570 tx=0x7f90f40afdd0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 shutdown_connections 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5592f2141500 0x5592f21438f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5592f20d4c00 0x5592f20db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5592f20cc080 0x5592f20dafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 --2- 172.21.15.119:0/3823852725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5592f20cb6d0 0x5592f20dcc20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f90e7400640 0 client.27298 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 >> 172.21.15.119:0/3823852725 conn(0x5592f20c5da0 msgr2=0x5592f20cc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 shutdown_connections 2024-03-28T18:50:43.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.063+0000 7f910b015740 1 -- 172.21.15.119:0/3823852725 wait complete. 2024-03-28T18:50:43.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f910b015740 20 client.27298 trim_cache size 0 max 16384 2024-03-28T18:50:43.069 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:43.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 msgr2=0x7f90e80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 0x7f90e80a4eb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009a60 tx=0x7f90f8005d80 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 msgr2=0x7f910019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 0x7f910019e540 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f90f400e9f0 tx=0x7f90f400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 shutdown_connections 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f90e80a29f0 0x7f90e80a4eb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001124e0 0x7f91001a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91000764f0 0x7f910019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 --2- 172.21.15.119:0/3360099824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9100075bd0 0x7f910019e540 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 >> 172.21.15.119:0/3360099824 conn(0x7f91000ff2f0 msgr2=0x7f91001016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 shutdown_connections 2024-03-28T18:50:43.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.064+0000 7f9108200640 1 -- 172.21.15.119:0/3360099824 wait complete. 2024-03-28T18:50:43.082 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:50:43.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 -- 172.21.15.119:0/400053501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 msgr2=0x7f8e18108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 --2- 172.21.15.119:0/400053501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e18108230 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009900 tx=0x7f8e0802bc50 comp rx=0 tx=0).stop 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 -- 172.21.15.119:0/400053501 shutdown_connections 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 --2- 172.21.15.119:0/400053501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e1810cea0 0x7f8e18114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 --2- 172.21.15.119:0/400053501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e18108800 0x7f8e1810c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 --2- 172.21.15.119:0/400053501 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e18108230 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 -- 172.21.15.119:0/400053501 >> 172.21.15.119:0/400053501 conn(0x7f8e180774c0 msgr2=0x7f8e181064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.303+0000 7f8e17200640 1 -- 172.21.15.119:0/400053501 shutdown_connections 2024-03-28T18:50:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.304+0000 7f8e17200640 1 -- 172.21.15.119:0/400053501 wait complete. 2024-03-28T18:50:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 Processor -- start 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 -- start start 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 0x7f8e1819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e181a2ce0 0x7f8e181a3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18117fa0 con 0x7f8e1810cea0 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18117e20 con 0x7f8e18107e50 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.305+0000 7f8e17200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e18118120 con 0x7f8e181a2ce0 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41124/0 (socket says 172.21.15.119:41124) 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 -- 172.21.15.119:0/540178216 learned_addr learned my addr 172.21.15.119:0/540178216 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e16800640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e181a2ce0 0x7f8e181a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 0x7f8e1819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 -- 172.21.15.119:0/540178216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e181a2ce0 msgr2=0x7f8e181a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e181a2ce0 0x7f8e181a3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 -- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 msgr2=0x7f8e1819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 0x7f8e1819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 -- 172.21.15.119:0/540178216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e181a37d0 con 0x7f8e18107e50 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 0x7f8e1819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e15e00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8e08005ab0 tx=0x7f8e080042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.306+0000 7f8e0ea00640 1 -- 172.21.15.119:0/540178216 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08041070 con 0x7f8e18107e50 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.307+0000 7f8e0ea00640 1 -- 172.21.15.119:0/540178216 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e080044c0 con 0x7f8e18107e50 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.307+0000 7f8e0ea00640 1 -- 172.21.15.119:0/540178216 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e08003d80 con 0x7f8e18107e50 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.307+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e181a3a60 con 0x7f8e18107e50 2024-03-28T18:50:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.307+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e181ab1d0 con 0x7f8e18107e50 2024-03-28T18:50:43.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.310+0000 7f8e0ea00640 1 -- 172.21.15.119:0/540178216 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8e08044070 con 0x7f8e18107e50 2024-03-28T18:50:43.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.310+0000 7f8e1ffaa740 1 Processor -- start 2024-03-28T18:50:43.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.310+0000 7f8e1ffaa740 1 -- start start 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.310+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f44e2c0 con 0x561f5f4eb260 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f3f6c90 con 0x561f5f4f3de0 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f44a360 con 0x561f5f4ea8b0 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e16800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e16800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41974/0 (socket says 172.21.15.119:41974) 2024-03-28T18:50:43.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e16800640 1 -- 172.21.15.119:0/1937651265 learned_addr learned my addr 172.21.15.119:0/1937651265 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 -- 172.21.15.119:0/1937651265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 msgr2=0x561f5f4eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 -- 172.21.15.119:0/1937651265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 msgr2=0x561f5f4f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0fe00640 1 -- 172.21.15.119:0/1937651265 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f5f41b080 con 0x561f5f4f3de0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e16800640 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.311+0000 7f8e0ea00640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 0x7f8dfc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e0ea00640 1 -- 172.21.15.119:0/540178216 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8e080f0760 con 0x7f8e18107e50 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f6230 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e1806abd0 tx=0x7f8e000117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23acc30f37c1b73a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e16800640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 0x7f8dfc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e07400640 1 -- 172.21.15.119:0/1937651265 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00012450 con 0x561f5f4f3de0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e07400640 1 -- 172.21.15.119:0/1937651265 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e00012ad0 con 0x561f5f4f3de0 2024-03-28T18:50:43.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.312+0000 7f8e07400640 1 -- 172.21.15.119:0/1937651265 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000fa70 con 0x561f5f4f3de0 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/1937651265 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 msgr2=0x561f5f4f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f6230 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e1806abd0 tx=0x7f8e000117b0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e16800640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 0x7f8dfc0a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8e100117f0 tx=0x7f8e10009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/1937651265 shutdown_connections 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f6230 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/1937651265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/1937651265 >> 172.21.15.119:0/1937651265 conn(0x561f5f4e4f80 msgr2=0x561f5f4e5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/1937651265 shutdown_connections 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/1937651265 wait complete. 2024-03-28T18:50:43.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 -- start start 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.313+0000 7f8e1ffaa740 1 Processor -- start 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f4f6d50 con 0x561f5f4eb260 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f40a0d0 con 0x561f5f4f3de0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f5f4f6ed0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e1ffaa740 10 client.0 _ll_register_callbacks cb 0x561f5f40ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e16800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41984/0 (socket says 172.21.15.119:41984) 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e15e00640 1 -- 172.21.15.119:0/136373151 learned_addr learned my addr 172.21.15.119:0/136373151 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49174/0 (socket says 172.21.15.119:49174) 2024-03-28T18:50:43.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.314+0000 7f8e15e00640 1 -- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 msgr2=0x561f5f4f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e15e00640 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e15e00640 1 -- 172.21.15.119:0/136373151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 msgr2=0x561f5f4f0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e15e00640 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e15e00640 1 -- 172.21.15.119:0/136373151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f5f1623f0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e15e00640 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e080bcb10 tx=0x7f8e0802ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e0c5aa4c227ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8dfbe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0804b050 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8e080c5120 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e00012ad0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e080bcbe0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e1ffaa740 10 client.27310 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561f5f22c7d0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.315+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e00012ad0 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8e080f0b50 con 0x561f5f4ea8b0 2024-03-28T18:50:43.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 10 client.27310 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 20 client.27310 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f503a40 0x561f5f505eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 4 client.27310 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f5f45b800 con 0x561f5f503a40 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f503a40 0x561f5f505eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f503a40 0x561f5f505eb0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1743b83de0db5570 server_cookie=93cd26d2b42d9525 in_seq=0 out_seq=0 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.316+0000 7f8dfbe00640 10 client.27310 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8dfbe00640 1 -- 172.21.15.119:0/136373151 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561f5f45b800 con 0x561f5f503a40 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8dfbe00640 10 client.27310 handle_command_reply: tid=0 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 client.27310 shutdown 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 msgr2=0x561f5f4f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e080bcb10 tx=0x7f8e0802ccb0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 shutdown_connections 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f503a40 0x561f5f505eb0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4f3de0 0x561f5f4f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4eb260 0x561f5f4f0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/136373151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4f35e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 >> 172.21.15.119:0/136373151 conn(0x561f5f4e4f80 msgr2=0x561f5f4e6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.317+0000 7f8dfbe00640 0 client.27310 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.318+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 shutdown_connections 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.318+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/136373151 wait complete. 2024-03-28T18:50:43.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.318+0000 7f8e1ffaa740 20 client.27310 trim_cache size 0 max 16384 2024-03-28T18:50:43.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.331+0000 7f8e1ffaa740 1 Processor -- start 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.331+0000 7f8e1ffaa740 1 -- start start 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.331+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4e43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b36b0 con 0x561f5f4f3de0 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b5030 con 0x561f5f4eb260 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b3d10 con 0x561f5f4ea8b0 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4e43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41148/0 (socket says 172.21.15.119:41148) 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 -- 172.21.15.119:0/4079486692 learned_addr learned my addr 172.21.15.119:0/4079486692 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e16800640 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 -- 172.21.15.119:0/4079486692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 msgr2=0x561f5f4e43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4e43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 -- 172.21.15.119:0/4079486692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 msgr2=0x561f5f4f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 -- 172.21.15.119:0/4079486692 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f5f4e5b50 con 0x561f5f4eb260 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e16800640 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.332+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e00012e40 tx=0x7f8e00005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=845c390d73dcb9b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e06a00640 1 -- 172.21.15.119:0/4079486692 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0002e020 con 0x561f5f4eb260 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e06a00640 1 -- 172.21.15.119:0/4079486692 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e0000f440 con 0x561f5f4eb260 2024-03-28T18:50:43.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e06a00640 1 -- 172.21.15.119:0/4079486692 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005040 con 0x561f5f4eb260 2024-03-28T18:50:43.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/4079486692 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 msgr2=0x561f5f4f30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e00012e40 tx=0x7f8e00005f90 comp rx=0 tx=0).stop 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/4079486692 shutdown_connections 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f30a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/4079486692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4e43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/4079486692 >> 172.21.15.119:0/4079486692 conn(0x561f5f4e4f80 msgr2=0x561f5f4ebed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/4079486692 shutdown_connections 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.333+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/4079486692 wait complete. 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 -- start start 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 Processor -- start 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b49d0 con 0x561f5f4f3de0 2024-03-28T18:50:43.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b4370 con 0x561f5f4eb260 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.334+0000 7f8e1ffaa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e181b1870 con 0x561f5f4ea8b0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e1ffaa740 10 client.0 _ll_register_callbacks cb 0x7f8e08005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42004/0 (socket says 172.21.15.119:42004) 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 -- 172.21.15.119:0/3075782980 learned_addr learned my addr 172.21.15.119:0/3075782980 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e16800640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 -- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 msgr2=0x561f5f4f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 -- 172.21.15.119:0/3075782980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 msgr2=0x561f5f4f8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 -- 172.21.15.119:0/3075782980 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f5f4f8d80 con 0x561f5f4ea8b0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8e15e00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e080f6190 tx=0x7f8e0802bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b02a11f87380bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.335+0000 7f8dfbe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:43.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0804b040 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f8e080c0070 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e0000f440 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e080a2050 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 10 client.27316 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561f5f560230 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e0000f440 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8e080bc260 con 0x561f5f4ea8b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 10 client.27316 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 10 client.27316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 20 client.27316 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f5606b0 0x561f5f562ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 4 client.27316 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561f5f455b50 con 0x561f5f5606b0 2024-03-28T18:50:43.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.336+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f5606b0 0x561f5f562ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e0fe00640 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f5606b0 0x561f5f562ad0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8006524e3df40369 server_cookie=42017dd5d7d3ea84 in_seq=0 out_seq=0 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8dfbe00640 10 client.27316 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8dfbe00640 1 -- 172.21.15.119:0/3075782980 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561f5f455b50 con 0x561f5f5606b0 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8dfbe00640 10 client.27316 handle_command_reply: tid=0 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 client.27316 shutdown 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 msgr2=0x561f5f4fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e080f6190 tx=0x7f8e0802bc50 comp rx=0 tx=0).stop 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 shutdown_connections 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561f5f5606b0 0x561f5f562ad0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561f5f4f3de0 0x561f5f4f8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561f5f4eb260 0x561f5f4f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8dfbe00640 0 client.27316 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 --2- 172.21.15.119:0/3075782980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561f5f4ea8b0 0x561f5f4fbd80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 >> 172.21.15.119:0/3075782980 conn(0x561f5f4e4f80 msgr2=0x561f5f50a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.337+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 shutdown_connections 2024-03-28T18:50:43.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e1ffaa740 1 -- 172.21.15.119:0/3075782980 wait complete. 2024-03-28T18:50:43.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e1ffaa740 20 client.27316 trim_cache size 0 max 16384 2024-03-28T18:50:43.343 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 msgr2=0x7f8dfc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 0x7f8dfc0a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8e100117f0 tx=0x7f8e10009500 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 msgr2=0x7f8e1819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8e08005ab0 tx=0x7f8e080042d0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 shutdown_connections 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8dfc0a29f0 0x7f8dfc0a4eb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e181a2ce0 0x7f8e181a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e1810cea0 0x7f8e1819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 --2- 172.21.15.119:0/540178216 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8e18107e50 0x7f8e1819e420 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.338+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 >> 172.21.15.119:0/540178216 conn(0x7f8e180774c0 msgr2=0x7f8e180790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.339+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 shutdown_connections 2024-03-28T18:50:43.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:43.339+0000 7f8e17200640 1 -- 172.21.15.119:0/540178216 wait complete. 2024-03-28T18:50:44.357 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:50:44.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:50:44.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:50:44.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 -- 172.21.15.119:0/1542616913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0102000 msgr2=0x7fd6a010e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 --2- 172.21.15.119:0/1542616913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0102000 0x7fd6a010e510 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd698009cd0 tx=0x7fd69802c0e0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 -- 172.21.15.119:0/1542616913 shutdown_connections 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 --2- 172.21.15.119:0/1542616913 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0102000 0x7fd6a010e510 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 --2- 172.21.15.119:0/1542616913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a0101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 --2- 172.21.15.119:0/1542616913 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0107650 0x7fd6a0107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 -- 172.21.15.119:0/1542616913 >> 172.21.15.119:0/1542616913 conn(0x7fd6a00fd0d0 msgr2=0x7fd6a00ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 -- 172.21.15.119:0/1542616913 shutdown_connections 2024-03-28T18:50:44.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.589+0000 7fd6a8400640 1 -- 172.21.15.119:0/1542616913 wait complete. 2024-03-28T18:50:44.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.590+0000 7fd6a8400640 1 Processor -- start 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.590+0000 7fd6a8400640 1 -- start start 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 0x7fd6a019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0111a60 con 0x7fd6a0101640 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01118e0 con 0x7fd6a0102000 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a8400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a0111be0 con 0x7fd6a0107650 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 0x7fd6a019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49210/0 (socket says 172.21.15.119:49210) 2024-03-28T18:50:44.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42022/0 (socket says 172.21.15.119:42022) 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a5a00640 1 -- 172.21.15.119:0/2854850716 learned_addr learned my addr 172.21.15.119:0/2854850716 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 -- 172.21.15.119:0/2854850716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 msgr2=0x7fd6a019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 0x7fd6a019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 -- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 msgr2=0x7fd6a019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a6400640 1 -- 172.21.15.119:0/2854850716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a019f110 con 0x7fd6a0107650 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.591+0000 7fd6a5000640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 0x7fd6a019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd6a6400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd698037040 tx=0x7fd698032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd69e000640 1 -- 172.21.15.119:0/2854850716 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698039040 con 0x7fd6a0107650 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd69e000640 1 -- 172.21.15.119:0/2854850716 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd698044070 con 0x7fd6a0107650 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a019f3a0 con 0x7fd6a0107650 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd69e000640 1 -- 172.21.15.119:0/2854850716 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980042a0 con 0x7fd6a0107650 2024-03-28T18:50:44.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.592+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a01a6c90 con 0x7fd6a0107650 2024-03-28T18:50:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.595+0000 7fd6ab1ab740 1 Processor -- start 2024-03-28T18:50:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.595+0000 7fd69e000640 1 -- 172.21.15.119:0/2854850716 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fd698032590 con 0x7fd6a0107650 2024-03-28T18:50:44.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 -- start start 2024-03-28T18:50:44.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 0x5578ab150460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab15b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab0b4eb0 con 0x5578ab150080 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab05da60 con 0x5578ab159580 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab0b0f50 con 0x5578ab1509a0 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 0x5578ab150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 0x5578ab150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49226/0 (socket says 172.21.15.119:49226) 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42038/0 (socket says 172.21.15.119:42038) 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5000640 1 -- 172.21.15.119:0/1754164609 learned_addr learned my addr 172.21.15.119:0/1754164609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.596+0000 7fd6a5000640 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1754164609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 msgr2=0x5578ab15b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab15b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1754164609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 msgr2=0x5578ab150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 0x5578ab150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1754164609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578aae917e0 con 0x5578ab1509a0 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5000640 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab15b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd690009d20 tx=0x7fd690011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75f7997fcd20a040 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd69e000640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 0x7fd6840a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd696a00640 1 -- 172.21.15.119:0/1754164609 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690005220 con 0x5578ab1509a0 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd696a00640 1 -- 172.21.15.119:0/1754164609 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd690004c80 con 0x5578ab1509a0 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd696a00640 1 -- 172.21.15.119:0/1754164609 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6900197f0 con 0x5578ab1509a0 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd69e000640 1 -- 172.21.15.119:0/2854850716 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd6980f11a0 con 0x7fd6a0107650 2024-03-28T18:50:44.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.597+0000 7fd6a5000640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 0x7fd6840a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1754164609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 msgr2=0x5578ab159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd690009d20 tx=0x7fd690011810 comp rx=0 tx=0).stop 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1754164609 shutdown_connections 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab15b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab1509a0 0x5578ab159040 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1754164609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab150080 0x5578ab150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1754164609 >> 172.21.15.119:0/1754164609 conn(0x5578ab14a7b0 msgr2=0x5578ab14ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6a5000640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 0x7fd6840a4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5578ab14b940 tx=0x7fd68c015040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1754164609 shutdown_connections 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.598+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1754164609 wait complete. 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 -- start start 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 Processor -- start 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab156860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab15c370 con 0x5578ab1509a0 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab070eb0 con 0x5578ab159580 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.599+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578ab15c4f0 con 0x5578ab150080 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6ab1ab740 10 client.0 _ll_register_callbacks cb 0x5578ab065150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a6400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42044/0 (socket says 172.21.15.119:42044) 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1741532288 learned_addr learned my addr 172.21.15.119:0/1741532288 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5000640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49228/0 (socket says 172.21.15.119:49228) 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 msgr2=0x5578ab156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1741532288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 msgr2=0x5578ab1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 -- 172.21.15.119:0/1741532288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578aaef1f70 con 0x5578ab150080 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a6400640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd69001b040 tx=0x7fd6900048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb0c89be7c202037 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.600+0000 7fd68b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690029070 con 0x5578ab150080 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd690004220 con 0x5578ab150080 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd690004c80 con 0x5578ab150080 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd690034050 con 0x5578ab150080 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 10 client.27328 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:44.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578aad89170 con 0x5578ab150080 2024-03-28T18:50:44.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd690004c80 con 0x5578ab150080 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd690019510 con 0x5578ab150080 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 10 client.27328 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 10 client.27328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 20 client.27328 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab169500 0x5578ab16b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 4 client.27328 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.601+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578ab0c23f0 con 0x5578ab169500 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.602+0000 7fd6a5000640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab169500 0x5578ab16b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.602+0000 7fd68b400640 10 client.27328 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.602+0000 7fd6a5000640 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab169500 0x5578ab16b8f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba13c373e6bda797 server_cookie=eabccb49f69d644f in_seq=0 out_seq=0 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd68b400640 1 -- 172.21.15.119:0/1741532288 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5578ab0c23f0 con 0x5578ab169500 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd68b400640 10 client.27328 handle_command_reply: tid=0 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 client.27328 shutdown 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 msgr2=0x5578ab158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd69001b040 tx=0x7fd6900048f0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 shutdown_connections 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab169500 0x5578ab16b8f0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd68b400640 0 client.27328 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/1741532288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab158f30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 >> 172.21.15.119:0/1741532288 conn(0x5578ab14a7b0 msgr2=0x5578ab14c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 shutdown_connections 2024-03-28T18:50:44.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/1741532288 wait complete. 2024-03-28T18:50:44.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.603+0000 7fd6ab1ab740 20 client.27328 trim_cache size 0 max 16384 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 Processor -- start 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 -- start start 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab155170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab158f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01af570 con 0x5578ab1509a0 2024-03-28T18:50:44.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01b0ef0 con 0x5578ab159580 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.621+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01afbd0 con 0x5578ab150080 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49244/0 (socket says 172.21.15.119:49244) 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 -- 172.21.15.119:0/543771993 learned_addr learned my addr 172.21.15.119:0/543771993 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a6400640 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 -- 172.21.15.119:0/543771993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 msgr2=0x5578ab155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 -- 172.21.15.119:0/543771993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 msgr2=0x5578ab158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 -- 172.21.15.119:0/543771993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578ab145c20 con 0x5578ab1509a0 2024-03-28T18:50:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd6a5000640 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5578ab157890 tx=0x7fd68c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11732f7b7c983fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.622+0000 7fd696000640 1 -- 172.21.15.119:0/543771993 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02d800 con 0x5578ab1509a0 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd696000640 1 -- 172.21.15.119:0/543771993 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd68c032070 con 0x5578ab1509a0 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd696000640 1 -- 172.21.15.119:0/543771993 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c003ca0 con 0x5578ab1509a0 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/543771993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 msgr2=0x5578ab1589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5578ab157890 tx=0x7fd68c004730 comp rx=0 tx=0).stop 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/543771993 shutdown_connections 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab1589f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/543771993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/543771993 >> 172.21.15.119:0/543771993 conn(0x5578ab14a7b0 msgr2=0x5578ab154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/543771993 shutdown_connections 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.623+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/543771993 wait complete. 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 -- start start 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 Processor -- start 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab153560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab1cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01b0890 con 0x5578ab1509a0 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01b0230 con 0x5578ab159580 2024-03-28T18:50:44.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.624+0000 7fd6ab1ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a01ad730 con 0x5578ab150080 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6ab1ab740 10 client.0 _ll_register_callbacks cb 0x7fd68c009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42060/0 (socket says 172.21.15.119:42060) 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 -- 172.21.15.119:0/988249879 learned_addr learned my addr 172.21.15.119:0/988249879 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a6400640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab1cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 -- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 msgr2=0x5578ab1cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab1cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 -- 172.21.15.119:0/988249879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 msgr2=0x5578ab153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 -- 172.21.15.119:0/988249879 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578ab1cc9a0 con 0x5578ab150080 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd6a5a00640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd690001eb0 tx=0x7fd690004e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=715c0cc0df242f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd68b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690029050 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.625+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7fd690005380 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd68c032070 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd690019c70 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 10 client.27334 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5578ab1cdce0 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd68c032070 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd6900035f0 con 0x5578ab150080 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 10 client.27334 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 10 client.27334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 20 client.27334 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:44.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab1ce2e0 0x5578ab1d0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 4 client.27334 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:50:44.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.626+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5578ab0bc740 con 0x5578ab1ce2e0 2024-03-28T18:50:44.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.627+0000 7fd6a5000640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab1ce2e0 0x5578ab1d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.627+0000 7fd6a5000640 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab1ce2e0 0x5578ab1d0700 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b802874e7c0e4dd server_cookie=69d4afc36d6d8dd0 in_seq=0 out_seq=0 2024-03-28T18:50:44.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.627+0000 7fd68b400640 10 client.27334 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd68b400640 1 -- 172.21.15.119:0/988249879 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5578ab0bc740 con 0x5578ab1ce2e0 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd68b400640 10 client.27334 handle_command_reply: tid=0 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 client.27334 shutdown 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 msgr2=0x5578ab153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd690001eb0 tx=0x7fd690004e00 comp rx=0 tx=0).stop 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 shutdown_connections 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5578ab1ce2e0 0x5578ab1d0700 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5578ab159580 0x5578ab1cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5578ab1509a0 0x5578ab153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 --2- 172.21.15.119:0/988249879 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578ab150080 0x5578ab153020 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd68b400640 0 client.27334 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 >> 172.21.15.119:0/988249879 conn(0x5578ab14a7b0 msgr2=0x5578ab1552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.628+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 shutdown_connections 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6ab1ab740 1 -- 172.21.15.119:0/988249879 wait complete. 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6ab1ab740 20 client.27334 trim_cache size 0 max 16384 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:50:44.633 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "3f28ccce-3d6a-457e-91a9-52dbe1d1abe0", 2024-03-28T18:50:44.634 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:50:44.634 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:44.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 msgr2=0x7fd6840a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 0x7fd6840a4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5578ab14b940 tx=0x7fd68c015040 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 msgr2=0x7fd6a019e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd698037040 tx=0x7fd698032460 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 shutdown_connections 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6840a29f0 0x7fd6840a4eb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6a0107650 0x7fd6a019e9e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6a0102000 0x7fd6a019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 --2- 172.21.15.119:0/2854850716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a0101640 0x7fd6a019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.629+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 >> 172.21.15.119:0/2854850716 conn(0x7fd6a00fd0d0 msgr2=0x7fd6a010cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.630+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 shutdown_connections 2024-03-28T18:50:44.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.630+0000 7fd6a8400640 1 -- 172.21.15.119:0/2854850716 wait complete. 2024-03-28T18:50:44.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:50:44.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:50:44.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:44 smithi131 ceph-mon[36065]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:44.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:44 smithi119 ceph-mon[31813]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 -- 172.21.15.119:0/4286220998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f181040c0 msgr2=0x7f9f181105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 --2- 172.21.15.119:0/4286220998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f181040c0 0x7f9f181105d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009cd0 tx=0x7f9f1002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 -- 172.21.15.119:0/4286220998 shutdown_connections 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 --2- 172.21.15.119:0/4286220998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f181040c0 0x7f9f181105d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 --2- 172.21.15.119:0/4286220998 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f18103700 0x7f9f18103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 --2- 172.21.15.119:0/4286220998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 0x7f9f18109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 -- 172.21.15.119:0/4286220998 >> 172.21.15.119:0/4286220998 conn(0x7f9f180ff1f0 msgr2=0x7f9f18101610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 -- 172.21.15.119:0/4286220998 shutdown_connections 2024-03-28T18:50:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.861+0000 7f9f1d400640 1 -- 172.21.15.119:0/4286220998 wait complete. 2024-03-28T18:50:44.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.862+0000 7f9f1d400640 1 Processor -- start 2024-03-28T18:50:44.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 -- start start 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 0x7f9f1819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18113ae0 con 0x7f9f18109710 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18113960 con 0x7f9f181040c0 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f1d400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18113c60 con 0x7f9f18103700 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.863+0000 7f9f17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41222/0 (socket says 172.21.15.119:41222) 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42072/0 (socket says 172.21.15.119:42072) 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 -- 172.21.15.119:0/1957263736 learned_addr learned my addr 172.21.15.119:0/1957263736 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 0x7f9f1819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 -- 172.21.15.119:0/1957263736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 msgr2=0x7f9f18199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 -- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 msgr2=0x7f9f1819e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 0x7f9f1819e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 -- 172.21.15.119:0/1957263736 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1819ef90 con 0x7f9f181040c0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17e00640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f17400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ea60 tx=0x7f9f0400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f14c00640 1 -- 172.21.15.119:0/1957263736 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0400cde0 con 0x7f9f181040c0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1819f280 con 0x7f9f181040c0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f14c00640 1 -- 172.21.15.119:0/1957263736 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f040035f0 con 0x7f9f181040c0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.864+0000 7f9f14c00640 1 -- 172.21.15.119:0/1957263736 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04010670 con 0x7f9f181040c0 2024-03-28T18:50:44.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.865+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f181a6b10 con 0x7f9f181040c0 2024-03-28T18:50:44.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 Processor -- start 2024-03-28T18:50:44.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 -- start start 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e551b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e55d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e4b6250 con 0x55ce3e55b460 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e45ea30 con 0x55ce3e552dd0 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f21631740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e4b22f0 con 0x55ce3e5520f0 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f14c00640 1 -- 172.21.15.119:0/1957263736 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9f040037a0 con 0x7f9f181040c0 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f1ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e551b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f1ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e551b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41230/0 (socket says 172.21.15.119:41230) 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42086/0 (socket says 172.21.15.119:42086) 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.868+0000 7f9f1ca00640 1 -- 172.21.15.119:0/3983620668 learned_addr learned my addr 172.21.15.119:0/3983620668 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17400640 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e55d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 -- 172.21.15.119:0/3983620668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 msgr2=0x55ce3e551b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e551b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 -- 172.21.15.119:0/3983620668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 msgr2=0x55ce3e55d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e55d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 -- 172.21.15.119:0/3983620668 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce3e449690 con 0x55ce3e5520f0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f17e00640 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009f50 tx=0x7f9f080118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9c2632a2af23585 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f0d200640 1 -- 172.21.15.119:0/3983620668 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08004030 con 0x55ce3e5520f0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f0d200640 1 -- 172.21.15.119:0/3983620668 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f08005500 con 0x55ce3e5520f0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.869+0000 7f9f0d200640 1 -- 172.21.15.119:0/3983620668 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08004db0 con 0x55ce3e5520f0 2024-03-28T18:50:44.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f14c00640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 0x7f9efc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 -- 172.21.15.119:0/3983620668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 msgr2=0x55ce3e552570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009f50 tx=0x7f9f080118d0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f14c00640 1 -- 172.21.15.119:0/1957263736 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9f040cf210 con 0x7f9f181040c0 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 0x7f9efc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 -- 172.21.15.119:0/3983620668 shutdown_connections 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e55d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e5520f0 0x55ce3e552570 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 --2- 172.21.15.119:0/3983620668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e551b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 -- 172.21.15.119:0/3983620668 >> 172.21.15.119:0/3983620668 conn(0x55ce3e519200 msgr2=0x55ce3e54cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 -- 172.21.15.119:0/3983620668 shutdown_connections 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.870+0000 7f9f21631740 1 -- 172.21.15.119:0/3983620668 wait complete. 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.871+0000 7f9f21631740 1 -- start start 2024-03-28T18:50:44.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.871+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 0x7f9efc0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f10005e30 tx=0x7f9f10005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.871+0000 7f9f21631740 1 Processor -- start 2024-03-28T18:50:44.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e557e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e558370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e55e2d0 con 0x55ce3e55b460 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e471e80 con 0x55ce3e5520f0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce3e55e450 con 0x55ce3e552dd0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f21631740 10 client.0 _ll_register_callbacks cb 0x55ce3e484130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41246/0 (socket says 172.21.15.119:41246) 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17e00640 1 -- 172.21.15.119:0/1698402450 learned_addr learned my addr 172.21.15.119:0/1698402450 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e557e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e558370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 -- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 msgr2=0x55ce3e55aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 -- 172.21.15.119:0/1698402450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 msgr2=0x55ce3e558370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e558370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 -- 172.21.15.119:0/1698402450 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce3e2947d0 con 0x55ce3e552dd0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17e00640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f17400640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e557e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f040cc8a0 tx=0x7f9f0409b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a53fc9076aaf1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.872+0000 7f9f01600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0409beb0 con 0x55ce3e552dd0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9f04097e20 con 0x55ce3e552dd0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f08005500 con 0x55ce3e552dd0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f0409b7b0 con 0x55ce3e552dd0 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 10 client.27346 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ce3e1ca3f0 con 0x55ce3e552dd0 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f08005500 con 0x55ce3e552dd0 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9f0409ac00 con 0x55ce3e552dd0 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 10 client.27346 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 10 client.27346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 20 client.27346 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e56d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 4 client.27346 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.873+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce3e4c3790 con 0x55ce3e56b340 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.874+0000 7f9f17e00640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e56d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.874+0000 7f9f17e00640 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e56d730 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a168da2d5001900 server_cookie=d1e367a109c6bf52 in_seq=0 out_seq=0 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.874+0000 7f9f01600640 10 client.27346 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f01600640 1 -- 172.21.15.119:0/1698402450 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ce3e4c3790 con 0x55ce3e56b340 2024-03-28T18:50:44.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f01600640 10 client.27346 handle_command_reply: tid=0 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 client.27346 shutdown 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 msgr2=0x55ce3e557e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e557e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9f040cc8a0 tx=0x7f9f0409b580 comp rx=0 tx=0).stop 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 shutdown_connections 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e56d730 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f01600640 0 client.27346 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e55b460 0x55ce3e558370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e557e30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 --2- 172.21.15.119:0/1698402450 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e5520f0 0x55ce3e55aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 >> 172.21.15.119:0/1698402450 conn(0x55ce3e519200 msgr2=0x55ce3e54e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 shutdown_connections 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.875+0000 7f9f21631740 1 -- 172.21.15.119:0/1698402450 wait complete. 2024-03-28T18:50:44.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.876+0000 7f9f21631740 20 client.27346 trim_cache size 0 max 16384 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.893+0000 7f9f21631740 1 Processor -- start 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.893+0000 7f9f21631740 1 -- start start 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e55c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 0x55ce3e55aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181af380 con 0x55ce3e5520f0 2024-03-28T18:50:44.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181b0d00 con 0x55ce3e552dd0 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f21631740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181af9e0 con 0x55ce3e55b460 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e55c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f1ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 0x55ce3e55aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f17400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:41260/0 (socket says 172.21.15.119:41260) 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f1ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 0x55ce3e55aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42118/0 (socket says 172.21.15.119:42118) 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.894+0000 7f9f17400640 1 -- 172.21.15.119:0/4113401591 learned_addr learned my addr 172.21.15.119:0/4113401591 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 -- 172.21.15.119:0/4113401591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 msgr2=0x55ce3e55aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 0x55ce3e55aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 -- 172.21.15.119:0/4113401591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 msgr2=0x55ce3e55c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e55c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 -- 172.21.15.119:0/4113401591 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce3e18a0f0 con 0x55ce3e552dd0 2024-03-28T18:50:44.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f17400640 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9f04097c10 tx=0x7f9f040862c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50aff7e75ee5cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f03e00640 1 -- 172.21.15.119:0/4113401591 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0409b2f0 con 0x55ce3e552dd0 2024-03-28T18:50:44.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f03e00640 1 -- 172.21.15.119:0/4113401591 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f0409beb0 con 0x55ce3e552dd0 2024-03-28T18:50:44.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.895+0000 7f9f03e00640 1 -- 172.21.15.119:0/4113401591 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0409ac00 con 0x55ce3e552dd0 2024-03-28T18:50:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 -- 172.21.15.119:0/4113401591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 msgr2=0x55ce3e55a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9f04097c10 tx=0x7f9f040862c0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 -- 172.21.15.119:0/4113401591 shutdown_connections 2024-03-28T18:50:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e55b460 0x55ce3e55aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e552dd0 0x55ce3e55a560 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 --2- 172.21.15.119:0/4113401591 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e55c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 -- 172.21.15.119:0/4113401591 >> 172.21.15.119:0/4113401591 conn(0x55ce3e519200 msgr2=0x55ce3e54c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 -- 172.21.15.119:0/4113401591 shutdown_connections 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.896+0000 7f9f21631740 1 -- 172.21.15.119:0/4113401591 wait complete. 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 -- start start 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 Processor -- start 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e555a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 0x55ce3e5549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181b06a0 con 0x55ce3e5520f0 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181b0040 con 0x55ce3e55b460 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181ad540 con 0x55ce3e552dd0 2024-03-28T18:50:44.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f21631740 10 client.0 _ll_register_callbacks cb 0x7f9f04003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f17e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e555a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.897+0000 7f9f17400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42130/0 (socket says 172.21.15.119:42130) 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17400640 1 -- 172.21.15.119:0/589529215 learned_addr learned my addr 172.21.15.119:0/589529215 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 0x55ce3e5549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 -- 172.21.15.119:0/589529215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 msgr2=0x55ce3e554470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 -- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 msgr2=0x55ce3e5549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 0x55ce3e5549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 -- 172.21.15.119:0/589529215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce3e555220 con 0x55ce3e5520f0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17400640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f1ca00640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 0x55ce3e5549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f17e00640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e555a30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9f08000c00 tx=0x7f9f080049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6723e7c6affd44eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:50:44.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f01600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.898+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0802f030 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8622+0+0 (secure 0 0 0) 0x7f9f08016040 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f0409beb0 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f08004040 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 10 client.17148 fetch_fsmap learned FSMap version 28 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ce3e5c6c50 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f0409beb0 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9f080035f0 con 0x55ce3e5520f0 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 10 client.17148 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 10 client.17148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 20 client.17148 populate_metadata read hostname 'smithi119' 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e5c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:50:44.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 4 client.17148 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.899+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ce3e4bdae0 con 0x55ce3e56b340 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f17400640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e5c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f17400640 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e5c91a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e588f16f6d259650 server_cookie=832aa14ebae01e47 in_seq=0 out_seq=0 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f01600640 10 client.17148 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f01600640 1 -- 172.21.15.119:0/589529215 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ce3e4bdae0 con 0x55ce3e56b340 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f01600640 10 client.17148 handle_command_reply: tid=0 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 client.17148 shutdown 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 msgr2=0x55ce3e555a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e555a30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9f08000c00 tx=0x7f9f080049f0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 shutdown_connections 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ce3e56b340 0x55ce3e5c91a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ce3e55b460 0x55ce3e5549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ce3e552dd0 0x55ce3e554470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 --2- 172.21.15.119:0/589529215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ce3e5520f0 0x55ce3e555a30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 >> 172.21.15.119:0/589529215 conn(0x55ce3e519200 msgr2=0x55ce3e5566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.900+0000 7f9f01600640 0 client.17148 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 shutdown_connections 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f21631740 1 -- 172.21.15.119:0/589529215 wait complete. 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f21631740 20 client.17148 trim_cache size 0 max 16384 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stdout: "3f28ccce-3d6a-457e-91a9-52dbe1d1abe0": { 2024-03-28T18:50:44.906 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "3f28ccce-3d6a-457e-91a9-52dbe1d1abe0", 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 msgr2=0x7f9efc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 0x7f9efc0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9f10005e30 tx=0x7f9f10005d80 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 msgr2=0x7f9f1819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.901+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ea60 tx=0x7f9f0400ef20 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 shutdown_connections 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f9efc0a2970 0x7f9efc0a4e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f18109710 0x7f9f1819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9f181040c0 0x7f9f1819a4e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 --2- 172.21.15.119:0/1957263736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f18103700 0x7f9f18199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 >> 172.21.15.119:0/1957263736 conn(0x7f9f180ff1f0 msgr2=0x7f9f180ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 shutdown_connections 2024-03-28T18:50:44.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:50:44.902+0000 7f9f1d400640 1 -- 172.21.15.119:0/1957263736 wait complete. 2024-03-28T18:50:44.919 INFO:tasks.cephfs.filesystem:scrub status for tag:3f28ccce-3d6a-457e-91a9-52dbe1d1abe0 - {'path': '/', 'tag': '3f28ccce-3d6a-457e-91a9-52dbe1d1abe0', 'options': 'recursive,force'} 2024-03-28T18:50:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:44 smithi196 ceph-mon[36300]: pgmap v203: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:45.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:45 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:50:45.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:45 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:45.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:45 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:50:45.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:45 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:45.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:45 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:50:45.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:45 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:45.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:45 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:50:45.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:45 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:45.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:45 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:50:45.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:45 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:45.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:45 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:50:45.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:45 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:50:46.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:46 smithi119 ceph-mon[31813]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:46 smithi196 ceph-mon[36300]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:46 smithi131 ceph-mon[36065]: pgmap v204: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:50:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:48 smithi119 ceph-mon[31813]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:48 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:50:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:48 smithi196 ceph-mon[36300]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:48.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:48 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:50:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:48 smithi131 ceph-mon[36065]: pgmap v205: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:50:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:48 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:50:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:50 smithi119 ceph-mon[31813]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:50.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:50 smithi196 ceph-mon[36300]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:50 smithi131 ceph-mon[36065]: pgmap v206: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:52.745 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:52 smithi131 ceph-mon[36065]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:52 smithi119 ceph-mon[31813]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:52 smithi196 ceph-mon[36300]: pgmap v207: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:54 smithi119 ceph-mon[31813]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:54.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:54 smithi196 ceph-mon[36300]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:54.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:54 smithi131 ceph-mon[36065]: pgmap v208: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:56 smithi119 ceph-mon[31813]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:56 smithi196 ceph-mon[36300]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:56 smithi131 ceph-mon[36065]: pgmap v209: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:50:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:50:58 smithi119 ceph-mon[31813]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:50:58 smithi196 ceph-mon[36300]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:50:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:50:58 smithi131 ceph-mon[36065]: pgmap v210: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:00 smithi119 ceph-mon[31813]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:00 smithi196 ceph-mon[36300]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:00 smithi131 ceph-mon[36065]: pgmap v211: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:02 smithi119 ceph-mon[31813]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:02 smithi196 ceph-mon[36300]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:02 smithi131 ceph-mon[36065]: pgmap v212: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:04 smithi119 ceph-mon[31813]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:04.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:04 smithi196 ceph-mon[36300]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:04 smithi131 ceph-mon[36065]: pgmap v213: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:06 smithi119 ceph-mon[31813]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:51:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:06 smithi196 ceph-mon[36300]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:51:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:06 smithi131 ceph-mon[36065]: pgmap v214: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:51:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:08 smithi119 ceph-mon[31813]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:51:08.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:08 smithi196 ceph-mon[36300]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:51:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:08 smithi131 ceph-mon[36065]: pgmap v215: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:51:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:09 smithi119 ceph-mon[31813]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:51:09.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:09 smithi196 ceph-mon[36300]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:51:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:09 smithi131 ceph-mon[36065]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T18:51:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:10 smithi119 ceph-mon[31813]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:51:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:10 smithi119 ceph-mon[31813]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:51:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:10 smithi196 ceph-mon[36300]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:51:10.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:10 smithi196 ceph-mon[36300]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:51:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:10 smithi131 ceph-mon[36065]: pgmap v217: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:51:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:10 smithi131 ceph-mon[36065]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T18:51:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:12 smithi119 ceph-mon[31813]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:51:12.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:12 smithi196 ceph-mon[36300]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:51:12.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:12 smithi131 ceph-mon[36065]: pgmap v219: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:51:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:14 smithi119 ceph-mon[31813]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:51:14.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:51:14.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:51:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:14 smithi196 ceph-mon[36300]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:51:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:14 smithi131 ceph-mon[36065]: pgmap v220: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:51:15.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 -- 172.21.15.119:0/1612990102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c103840 msgr2=0x7eff6c103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 --2- 172.21.15.119:0/1612990102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c103840 0x7eff6c103cc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7eff54009860 tx=0x7eff5402bb70 comp rx=0 tx=0).stop 2024-03-28T18:51:15.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 -- 172.21.15.119:0/1612990102 shutdown_connections 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 --2- 172.21.15.119:0/1612990102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c104200 0x7eff6c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 --2- 172.21.15.119:0/1612990102 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c103840 0x7eff6c103cc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 --2- 172.21.15.119:0/1612990102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c109850 0x7eff6c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 -- 172.21.15.119:0/1612990102 >> 172.21.15.119:0/1612990102 conn(0x7eff6c0ff350 msgr2=0x7eff6c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 -- 172.21.15.119:0/1612990102 shutdown_connections 2024-03-28T18:51:15.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.145+0000 7eff73400640 1 -- 172.21.15.119:0/1612990102 wait complete. 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.146+0000 7eff73400640 1 Processor -- start 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.146+0000 7eff73400640 1 -- start start 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 0x7eff6c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 0x7eff6c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 0x7eff6c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c113c00 con 0x7eff6c103840 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c113a80 con 0x7eff6c104200 2024-03-28T18:51:15.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff73400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c113d80 con 0x7eff6c109850 2024-03-28T18:51:15.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff71400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 0x7eff6c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 0x7eff6c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff70a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 0x7eff6c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff71400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 0x7eff6c19ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49610/0 (socket says 172.21.15.119:49610) 2024-03-28T18:51:15.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.147+0000 7eff71400640 1 -- 172.21.15.119:0/398310071 learned_addr learned my addr 172.21.15.119:0/398310071 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 -- 172.21.15.119:0/398310071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 msgr2=0x7eff6c19ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 0x7eff6c19ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 -- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 msgr2=0x7eff6c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 0x7eff6c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 -- 172.21.15.119:0/398310071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c19f1b0 con 0x7eff6c104200 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff70a00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 0x7eff6c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff63e00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 0x7eff6c19a700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7eff5402c0b0 tx=0x7eff540045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff6a000640 1 -- 172.21.15.119:0/398310071 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54041020 con 0x7eff6c104200 2024-03-28T18:51:15.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff6a000640 1 -- 172.21.15.119:0/398310071 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff5402c9f0 con 0x7eff6c104200 2024-03-28T18:51:15.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c19f440 con 0x7eff6c104200 2024-03-28T18:51:15.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff6a000640 1 -- 172.21.15.119:0/398310071 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54039040 con 0x7eff6c104200 2024-03-28T18:51:15.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.148+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c1a6d30 con 0x7eff6c104200 2024-03-28T18:51:15.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.151+0000 7eff7602f740 1 Processor -- start 2024-03-28T18:51:15.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.151+0000 7eff6a000640 1 -- 172.21.15.119:0/398310071 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7eff54044020 con 0x7eff6c104200 2024-03-28T18:51:15.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 -- start start 2024-03-28T18:51:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390ba290 0x55d1390bc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d139015250 con 0x55d1390b1c00 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d138fbda30 con 0x55d1390b0f20 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff7602f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1390112f0 con 0x55d1390ba290 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff71400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff70a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff71400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38828/0 (socket says 172.21.15.119:38828) 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff71400640 1 -- 172.21.15.119:0/3518639423 learned_addr learned my addr 172.21.15.119:0/3518639423 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.152+0000 7eff63e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390ba290 0x55d1390bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 -- 172.21.15.119:0/3518639423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390ba290 msgr2=0x55d1390bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390ba290 0x55d1390bc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 -- 172.21.15.119:0/3518639423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 msgr2=0x55d1390b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 -- 172.21.15.119:0/3518639423 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d138fa8690 con 0x55d1390b1c00 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff70a00640 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff71400640 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7eff64009d70 tx=0x7eff64011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8fc5ceddc9e3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff6a000640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 0x7eff500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff6a000640 1 -- 172.21.15.119:0/398310071 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7eff540f0450 con 0x7eff6c104200 2024-03-28T18:51:15.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff61c00640 1 -- 172.21.15.119:0/3518639423 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6401e070 con 0x55d1390b1c00 2024-03-28T18:51:15.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff61c00640 1 -- 172.21.15.119:0/3518639423 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff64019b10 con 0x55d1390b1c00 2024-03-28T18:51:15.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff61c00640 1 -- 172.21.15.119:0/3518639423 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6400eb80 con 0x55d1390b1c00 2024-03-28T18:51:15.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.153+0000 7eff70a00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 0x7eff500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 -- 172.21.15.119:0/3518639423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 msgr2=0x55d1390b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7eff64009d70 tx=0x7eff64011810 comp rx=0 tx=0).stop 2024-03-28T18:51:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 -- 172.21.15.119:0/3518639423 shutdown_connections 2024-03-28T18:51:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390ba290 0x55d1390bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 --2- 172.21.15.119:0/3518639423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b1c00 0x55d1390b0950 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 -- 172.21.15.119:0/3518639423 >> 172.21.15.119:0/3518639423 conn(0x55d139078200 msgr2=0x55d1390ab970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff70a00640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 0x7eff500a4eb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55d1390b2150 tx=0x7eff5c009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 -- 172.21.15.119:0/3518639423 shutdown_connections 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.154+0000 7eff7602f740 1 -- 172.21.15.119:0/3518639423 wait complete. 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 -- start start 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 Processor -- start 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b0f20 0x55d1390b9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1390bd100 con 0x55d1390ba290 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d138fd0e80 con 0x55d1390b1c00 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1390bd280 con 0x55d1390b0f20 2024-03-28T18:51:15.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.155+0000 7eff7602f740 10 client.0 _ll_register_callbacks cb 0x55d138fe3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff70a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b0f20 0x55d1390b9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff71400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52572/0 (socket says 172.21.15.119:52572) 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 -- 172.21.15.119:0/3515956527 learned_addr learned my addr 172.21.15.119:0/3515956527 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 -- 172.21.15.119:0/3515956527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b0f20 msgr2=0x55d1390b9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b0f20 0x55d1390b9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 -- 172.21.15.119:0/3515956527 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 msgr2=0x55d1390b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 -- 172.21.15.119:0/3515956527 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d138df37d0 con 0x55d1390b1c00 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff63e00640 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff5402c180 tx=0x7eff54005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ab7cd05c71198c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff5a000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540b8d70 con 0x55d1390b1c00 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7eff5402cba0 con 0x55d1390b1c00 2024-03-28T18:51:15.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.156+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff64019b10 con 0x55d1390b1c00 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff540bc430 con 0x55d1390b1c00 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 10 client.27392 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d138d293f0 con 0x55d1390b1c00 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff64019b10 con 0x55d1390b1c00 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7eff5403c030 con 0x55d1390b1c00 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 10 client.27392 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 10 client.27392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 20 client.27392 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d1390ca190 0x55d1390cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 4 client.27392 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d139022790 con 0x55d1390ca190 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.157+0000 7eff70a00640 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d1390ca190 0x55d1390cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.158+0000 7eff70a00640 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d1390ca190 0x55d1390cc5b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f295cce24728ebf1 server_cookie=c840078c2662305d in_seq=0 out_seq=0 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.158+0000 7eff5a000640 10 client.27392 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff5a000640 1 -- 172.21.15.119:0/3515956527 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d139022790 con 0x55d1390ca190 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff5a000640 10 client.27392 handle_command_reply: tid=0 2024-03-28T18:51:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 client.27392 shutdown 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 msgr2=0x55d1390b6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7eff5402c180 tx=0x7eff54005bd0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 shutdown_connections 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d1390ca190 0x55d1390cc5b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b1c00 0x55d1390b6ce0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 --2- 172.21.15.119:0/3515956527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b0f20 0x55d1390b9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 >> 172.21.15.119:0/3515956527 conn(0x55d139078200 msgr2=0x55d1390ad6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff5a000640 0 client.27392 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 shutdown_connections 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 1 -- 172.21.15.119:0/3515956527 wait complete. 2024-03-28T18:51:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.159+0000 7eff7602f740 20 client.27392 trim_cache size 0 max 16384 2024-03-28T18:51:15.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 Processor -- start 2024-03-28T18:51:15.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 -- start start 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390bb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1af6c0 con 0x55d1390ba290 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1b1040 con 0x55d1390b0f20 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.177+0000 7eff7602f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1afd20 con 0x55d1390b1c00 2024-03-28T18:51:15.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38848/0 (socket says 172.21.15.119:38848) 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 -- 172.21.15.119:0/827089922 learned_addr learned my addr 172.21.15.119:0/827089922 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff63e00640 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff70a00640 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390bb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 -- 172.21.15.119:0/827089922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 msgr2=0x55d1390bbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390bbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 -- 172.21.15.119:0/827089922 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 msgr2=0x55d1390bb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390bb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 -- 172.21.15.119:0/827089922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1390cc920 con 0x55d1390ba290 2024-03-28T18:51:15.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.178+0000 7eff71400640 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7eff6400e680 tx=0x7eff6400e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e19e28832691e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff61200640 1 -- 172.21.15.119:0/827089922 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64028020 con 0x55d1390ba290 2024-03-28T18:51:15.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff61200640 1 -- 172.21.15.119:0/827089922 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff64005590 con 0x55d1390ba290 2024-03-28T18:51:15.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff61200640 1 -- 172.21.15.119:0/827089922 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6401e040 con 0x55d1390ba290 2024-03-28T18:51:15.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 -- 172.21.15.119:0/827089922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 msgr2=0x55d1390b3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7eff6400e680 tx=0x7eff6400e6e0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 -- 172.21.15.119:0/827089922 shutdown_connections 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390ba290 0x55d1390b3fc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390bbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 --2- 172.21.15.119:0/827089922 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390b0f20 0x55d1390bb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.179+0000 7eff7602f740 1 -- 172.21.15.119:0/827089922 >> 172.21.15.119:0/827089922 conn(0x55d139078200 msgr2=0x55d1390b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.180+0000 7eff7602f740 1 -- 172.21.15.119:0/827089922 shutdown_connections 2024-03-28T18:51:15.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.180+0000 7eff7602f740 1 -- 172.21.15.119:0/827089922 wait complete. 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.180+0000 7eff7602f740 1 -- start start 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.180+0000 7eff7602f740 1 Processor -- start 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390c2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 0x55d1390b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1b09e0 con 0x55d1390b0f20 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1b0380 con 0x55d1390ba290 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1ad8a0 con 0x55d1390b1c00 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff7602f740 10 client.0 _ll_register_callbacks cb 0x7eff54005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff70a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff63e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 0x55d1390b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff70a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38858/0 (socket says 172.21.15.119:38858) 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff70a00640 1 -- 172.21.15.119:0/947086853 learned_addr learned my addr 172.21.15.119:0/947086853 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 -- 172.21.15.119:0/947086853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 msgr2=0x55d1390c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 -- 172.21.15.119:0/947086853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 msgr2=0x55d1390c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff71400640 1 -- 172.21.15.119:0/947086853 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d138ce90f0 con 0x55d1390ba290 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.181+0000 7eff70a00640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff71400640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 0x55d1390b3f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7eff6400e6b0 tx=0x7eff64005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493cf5d15d6a94c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:15.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64031070 con 0x55d1390ba290 2024-03-28T18:51:15.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff64005590 con 0x55d1390ba290 2024-03-28T18:51:15.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7eff640050b0 con 0x55d1390ba290 2024-03-28T18:51:15.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff6403d050 con 0x55d1390ba290 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff7602f740 10 client.27398 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d13912ecb0 con 0x55d1390ba290 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff64005590 con 0x55d1390ba290 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.182+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7eff64024480 con 0x55d1390ba290 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 10 client.27398 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 10 client.27398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 20 client.27398 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d13912f080 0x55d139131470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 4 client.27398 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d13901cae0 con 0x55d13912f080 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff70a00640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d13912f080 0x55d139131470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff70a00640 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d13912f080 0x55d139131470 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b93951ffdf9dfc8 server_cookie=2352358a4729bf6b in_seq=0 out_seq=0 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.183+0000 7eff5a000640 10 client.27398 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff5a000640 1 -- 172.21.15.119:0/947086853 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d13901cae0 con 0x55d13912f080 2024-03-28T18:51:15.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff5a000640 10 client.27398 handle_command_reply: tid=0 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 client.27398 shutdown 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 msgr2=0x55d1390b3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 0x55d1390b3f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7eff6400e6b0 tx=0x7eff64005740 comp rx=0 tx=0).stop 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 shutdown_connections 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55d13912f080 0x55d139131470 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff5a000640 0 client.27398 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d1390ba290 0x55d1390b3f00 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d1390b1c00 0x55d1390c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 --2- 172.21.15.119:0/947086853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d1390b0f20 0x55d1390c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 >> 172.21.15.119:0/947086853 conn(0x55d139078200 msgr2=0x55d1390aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 shutdown_connections 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 1 -- 172.21.15.119:0/947086853 wait complete. 2024-03-28T18:51:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.184+0000 7eff7602f740 20 client.27398 trim_cache size 0 max 16384 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 msgr2=0x7eff500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 0x7eff500a4eb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55d1390b2150 tx=0x7eff5c009500 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 msgr2=0x7eff6c19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 0x7eff6c19a700 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7eff5402c0b0 tx=0x7eff540045b0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 shutdown_connections 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7eff500a29f0 0x7eff500a4eb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c109850 0x7eff6c19ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7eff6c104200 0x7eff6c19a700 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 --2- 172.21.15.119:0/398310071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff6c103840 0x7eff6c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 >> 172.21.15.119:0/398310071 conn(0x7eff6c0ff350 msgr2=0x7eff6c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 shutdown_connections 2024-03-28T18:51:15.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.185+0000 7eff73400640 1 -- 172.21.15.119:0/398310071 wait complete. 2024-03-28T18:51:15.204 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:51:15.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.408+0000 7fa8ce000640 1 -- 172.21.15.119:0/3108031648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 msgr2=0x7fa8c8103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.408+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3108031648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c8103cd0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009930 tx=0x7fa8b402bc20 comp rx=0 tx=0).stop 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 -- 172.21.15.119:0/3108031648 shutdown_connections 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3108031648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 0x7fa8c8110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3108031648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c8103cd0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3108031648 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c8109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 -- 172.21.15.119:0/3108031648 >> 172.21.15.119:0/3108031648 conn(0x7fa8c80ff2e0 msgr2=0x7fa8c8101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 -- 172.21.15.119:0/3108031648 shutdown_connections 2024-03-28T18:51:15.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.409+0000 7fa8ce000640 1 -- 172.21.15.119:0/3108031648 wait complete. 2024-03-28T18:51:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 Processor -- start 2024-03-28T18:51:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 -- start start 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 0x7fa8c819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8113c70 con 0x7fa8c8104210 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8113af0 con 0x7fa8c8109860 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.410+0000 7fa8ce000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8113df0 con 0x7fa8c8103850 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49674/0 (socket says 172.21.15.119:49674) 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 -- 172.21.15.119:0/3676784567 learned_addr learned my addr 172.21.15.119:0/3676784567 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c6a00640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 0x7fa8c819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 -- 172.21.15.119:0/3676784567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 msgr2=0x7fa8c819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 -- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 msgr2=0x7fa8c819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 0x7fa8c819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 -- 172.21.15.119:0/3676784567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c819f200 con 0x7fa8c8103850 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7e00640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.411+0000 7fa8c7400640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800e9f0 tx=0x7fa8b800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.412+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800ccc0 con 0x7fa8c8103850 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.412+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b800ce60 con 0x7fa8c8103850 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.412+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8010690 con 0x7fa8c8103850 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.412+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c819f4f0 con 0x7fa8c8103850 2024-03-28T18:51:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.412+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c81a6d80 con 0x7fa8c8103850 2024-03-28T18:51:15.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.413+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c8068490 con 0x7fa8c8103850 2024-03-28T18:51:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.415+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fa8b80046c0 con 0x7fa8c8103850 2024-03-28T18:51:15.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.416+0000 7fa8bfe00640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 0x7fa8ac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.416+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fa8b80cf220 con 0x7fa8c8103850 2024-03-28T18:51:15.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.417+0000 7fa8c6a00640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 0x7fa8ac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.417+0000 7fa8c6a00640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 0x7fa8ac0a4eb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009900 tx=0x7fa8b4000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.420+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8b809be90 con 0x7fa8c8103850 2024-03-28T18:51:15.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.661+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c819b560 con 0x7fa8c8103850 2024-03-28T18:51:15.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.662+0000 7fa8bfe00640 1 -- 172.21.15.119:0/3676784567 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fa8b809b790 con 0x7fa8c8103850 2024-03-28T18:51:15.664 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:51:15.664 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:51:15.664 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:51:15.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 msgr2=0x7fa8ac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 0x7fa8ac0a4eb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009900 tx=0x7fa8b4000bc0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 msgr2=0x7fa8c819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800e9f0 tx=0x7fa8b800edf0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 shutdown_connections 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fa8ac0a29f0 0x7fa8ac0a4eb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa8c8109860 0x7fa8c819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8c8104210 0x7fa8c819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 --2- 172.21.15.119:0/3676784567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8c8103850 0x7fa8c819a210 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 >> 172.21.15.119:0/3676784567 conn(0x7fa8c80ff2e0 msgr2=0x7fa8c80ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.665+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 shutdown_connections 2024-03-28T18:51:15.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.666+0000 7fa8ce000640 1 -- 172.21.15.119:0/3676784567 wait complete. 2024-03-28T18:51:15.685 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:15.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:51:15.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 -- 172.21.15.119:0/4280094699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 msgr2=0x7f3d3c075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 --2- 172.21.15.119:0/4280094699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c075fb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009930 tx=0x7f3d3002bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:15.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 -- 172.21.15.119:0/4280094699 shutdown_connections 2024-03-28T18:51:15.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 --2- 172.21.15.119:0/4280094699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c1124e0 0x7f3d3c1148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 --2- 172.21.15.119:0/4280094699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c0764f0 0x7f3d3c111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.904+0000 7f3d43e00640 1 --2- 172.21.15.119:0/4280094699 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c075fb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.905+0000 7f3d43e00640 1 -- 172.21.15.119:0/4280094699 >> 172.21.15.119:0/4280094699 conn(0x7f3d3c0ff2f0 msgr2=0x7f3d3c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.905+0000 7f3d43e00640 1 -- 172.21.15.119:0/4280094699 shutdown_connections 2024-03-28T18:51:15.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.905+0000 7f3d43e00640 1 -- 172.21.15.119:0/4280094699 wait complete. 2024-03-28T18:51:15.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.906+0000 7f3d43e00640 1 Processor -- start 2024-03-28T18:51:15.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.906+0000 7f3d43e00640 1 -- start start 2024-03-28T18:51:15.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.906+0000 7f3d43e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d43e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c0764f0 0x7f3d3c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d43e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c1124e0 0x7f3d3c1a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d43e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c118090 con 0x7f3d3c1124e0 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d43e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c117f10 con 0x7f3d3c075bd0 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d43e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c118210 con 0x7f3d3c0764f0 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52624/0 (socket says 172.21.15.119:52624) 2024-03-28T18:51:15.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 -- 172.21.15.119:0/3713147257 learned_addr learned my addr 172.21.15.119:0/3713147257 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d40a00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c0764f0 0x7f3d3c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c1124e0 0x7f3d3c1a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 -- 172.21.15.119:0/3713147257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c0764f0 msgr2=0x7f3d3c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c0764f0 0x7f3d3c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 -- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c1124e0 msgr2=0x7f3d3c1a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c1124e0 0x7f3d3c1a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 -- 172.21.15.119:0/3713147257 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c1a3640 con 0x7f3d3c075bd0 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.907+0000 7f3d41400640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009900 tx=0x7f3d30004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.908+0000 7f3d3a000640 1 -- 172.21.15.119:0/3713147257 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30043020 con 0x7f3d3c075bd0 2024-03-28T18:51:15.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.908+0000 7f3d3a000640 1 -- 172.21.15.119:0/3713147257 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d300054e0 con 0x7f3d3c075bd0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.908+0000 7f3d3a000640 1 -- 172.21.15.119:0/3713147257 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30003bb0 con 0x7f3d3c075bd0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.908+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c1a38d0 con 0x7f3d3c075bd0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.908+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c1ab1c0 con 0x7f3d3c075bd0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.911+0000 7f3d3a000640 1 -- 172.21.15.119:0/3713147257 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f3d30046020 con 0x7f3d3c075bd0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.911+0000 7f3d46c38740 1 Processor -- start 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.911+0000 7f3d46c38740 1 -- start start 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 0x55d99fa40050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99f9982c0 con 0x55d99fa35080 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99f940c90 con 0x55d99fa346d0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d46c38740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99f994360 con 0x55d99fa3dc00 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52630/0 (socket says 172.21.15.119:52630) 2024-03-28T18:51:15.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 -- 172.21.15.119:0/4258061834 learned_addr learned my addr 172.21.15.119:0/4258061834 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d40a00640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 0x55d99fa40050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41400640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 -- 172.21.15.119:0/4258061834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 msgr2=0x55d99fa40050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 0x55d99fa40050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 -- 172.21.15.119:0/4258061834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 msgr2=0x55d99fa3d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d41e00640 1 -- 172.21.15.119:0/4258061834 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d99f965080 con 0x55d99fa346d0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.912+0000 7f3d40a00640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 0x55d99fa40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d3a000640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 0x7f3d240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d41e00640 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009f00 tx=0x7f3d340107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=917e36ffa4a30523 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d3a000640 1 -- 172.21.15.119:0/3713147257 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f3d300f0ee0 con 0x7f3d3c075bd0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d2e600640 1 -- 172.21.15.119:0/4258061834 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34003b70 con 0x55d99fa346d0 2024-03-28T18:51:15.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d2e600640 1 -- 172.21.15.119:0/4258061834 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d34003d10 con 0x55d99fa346d0 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d2e600640 1 -- 172.21.15.119:0/4258061834 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011a40 con 0x55d99fa346d0 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d41400640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 0x7f3d240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d46c38740 1 -- 172.21.15.119:0/4258061834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 msgr2=0x55d99fa34ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.913+0000 7f3d46c38740 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009f00 tx=0x7f3d340107a0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d41400640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 0x7f3d240a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55d99fa341a0 tx=0x7f3d30004ee0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 -- 172.21.15.119:0/4258061834 shutdown_connections 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa3dc00 0x55d99fa40050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 --2- 172.21.15.119:0/4258061834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa346d0 0x55d99fa34ab0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 -- 172.21.15.119:0/4258061834 >> 172.21.15.119:0/4258061834 conn(0x55d99fa2eda0 msgr2=0x55d99fa2f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 -- 172.21.15.119:0/4258061834 shutdown_connections 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 -- 172.21.15.119:0/4258061834 wait complete. 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 -- start start 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.914+0000 7f3d46c38740 1 Processor -- start 2024-03-28T18:51:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 0x55d99fa3ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99fa40b70 con 0x55d99fa35080 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99f9540d0 con 0x55d99fa3dc00 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d99fa40cf0 con 0x55d99fa346d0 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d46c38740 10 client.0 _ll_register_callbacks cb 0x55d99f958e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49700/0 (socket says 172.21.15.119:49700) 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d41e00640 1 -- 172.21.15.119:0/1490966868 learned_addr learned my addr 172.21.15.119:0/1490966868 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38900/0 (socket says 172.21.15.119:38900) 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d41e00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 0x55d99fa3ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 -- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 msgr2=0x55d99fa3d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 -- 172.21.15.119:0/1490966868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 msgr2=0x55d99fa3ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 0x55d99fa3ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d40a00640 1 -- 172.21.15.119:0/1490966868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d99f6ac3f0 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.915+0000 7f3d41e00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 0x55d99fa3ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d40a00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d280126d0 tx=0x7f3d28012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dfecfbdeff79d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d22a00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28004030 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f3d280041d0 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d34003d10 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d280043d0 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d46c38740 10 client.17184 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.916+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d99f7767d0 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d34003d10 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3d28022030 con 0x55d99fa35080 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 10 client.17184 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 10 client.17184 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:15.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 20 client.17184 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99fa4dae0 0x55d99fa4fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 4 client.17184 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d99f9a5800 con 0x55d99fa4dae0 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.917+0000 7f3d41e00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99fa4dae0 0x55d99fa4fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.918+0000 7f3d41e00640 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99fa4dae0 0x55d99fa4fed0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e59dd413a32905cd server_cookie=50cda763181b6519 in_seq=0 out_seq=0 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.918+0000 7f3d22a00640 10 client.17184 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d22a00640 1 -- 172.21.15.119:0/1490966868 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d99f9a5800 con 0x55d99fa4dae0 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d22a00640 10 client.17184 handle_command_reply: tid=0 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 client.17184 shutdown 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 msgr2=0x55d99fa3a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d280126d0 tx=0x7f3d28012b90 comp rx=0 tx=0).stop 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 shutdown_connections 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99fa4dae0 0x55d99fa4fed0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa3dc00 0x55d99fa3ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa35080 0x55d99fa3a8a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d22a00640 0 client.17184 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 --2- 172.21.15.119:0/1490966868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 >> 172.21.15.119:0/1490966868 conn(0x55d99fa2eda0 msgr2=0x55d99fa30af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 shutdown_connections 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 1 -- 172.21.15.119:0/1490966868 wait complete. 2024-03-28T18:51:15.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.919+0000 7f3d46c38740 20 client.17184 trim_cache size 0 max 16384 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 Processor -- start 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 -- start start 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa37b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b3aa0 con 0x55d99fa3dc00 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b5420 con 0x55d99fa35080 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d46c38740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b4100 con 0x55d99fa346d0 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52648/0 (socket says 172.21.15.119:52648) 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49710/0 (socket says 172.21.15.119:49710) 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d40a00640 1 -- 172.21.15.119:0/212139318 learned_addr learned my addr 172.21.15.119:0/212139318 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.933+0000 7f3d41e00640 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa37b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 -- 172.21.15.119:0/212139318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 msgr2=0x55d99fa3cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 -- 172.21.15.119:0/212139318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 msgr2=0x55d99fa37b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa37b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 -- 172.21.15.119:0/212139318 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d99fa50240 con 0x55d99fa35080 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d40a00640 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009560 tx=0x7f3d28005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8fb8ad030f5ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d2dc00640 1 -- 172.21.15.119:0/212139318 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28003e30 con 0x55d99fa35080 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d2dc00640 1 -- 172.21.15.119:0/212139318 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d28003fd0 con 0x55d99fa35080 2024-03-28T18:51:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.934+0000 7f3d2dc00640 1 -- 172.21.15.119:0/212139318 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800fea0 con 0x55d99fa35080 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- 172.21.15.119:0/212139318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 msgr2=0x55d99fa3d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009560 tx=0x7f3d28005870 comp rx=0 tx=0).stop 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- 172.21.15.119:0/212139318 shutdown_connections 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa37b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa3d510 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 --2- 172.21.15.119:0/212139318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa3cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- 172.21.15.119:0/212139318 >> 172.21.15.119:0/212139318 conn(0x55d99fa2eda0 msgr2=0x55d99fa2deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- 172.21.15.119:0/212139318 shutdown_connections 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- 172.21.15.119:0/212139318 wait complete. 2024-03-28T18:51:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 -- start start 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.935+0000 7f3d46c38740 1 Processor -- start 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa45c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa444e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b4dc0 con 0x55d99fa3dc00 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b4760 con 0x55d99fa35080 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1b1c80 con 0x55d99fa346d0 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d46c38740 10 client.0 _ll_register_callbacks cb 0x7f3d28009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d40a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52650/0 (socket says 172.21.15.119:52650) 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d40a00640 1 -- 172.21.15.119:0/2564079764 learned_addr learned my addr 172.21.15.119:0/2564079764 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d41e00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa45c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.936+0000 7f3d41e00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 -- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 msgr2=0x55d99fa45c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa45c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 -- 172.21.15.119:0/2564079764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 msgr2=0x55d99fa444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 -- 172.21.15.119:0/2564079764 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d99fa44c50 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d41e00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d40a00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3d28003c40 tx=0x7f3d28005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67cf117877cd690f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d22a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d280037f0 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f3d28003990 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.937+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d28003fd0 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d28005250 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 10 client.27422 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d99faaa320 con 0x55d99fa35080 2024-03-28T18:51:15.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d28003fd0 con 0x55d99fa35080 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3d2802b070 con 0x55d99fa35080 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 10 client.27422 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 10 client.27422 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 20 client.27422 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99faaa970 0x55d99faacd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 4 client.27422 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d99f99fb50 con 0x55d99faaa970 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.938+0000 7f3d41e00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99faaa970 0x55d99faacd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d41e00640 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99faaa970 0x55d99faacd60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e07d86c3a042b76b server_cookie=f10bf14449ed94b1 in_seq=0 out_seq=0 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d22a00640 10 client.27422 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d22a00640 1 -- 172.21.15.119:0/2564079764 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d99f99fb50 con 0x55d99faaa970 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d22a00640 10 client.27422 handle_command_reply: tid=0 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 client.27422 shutdown 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 msgr2=0x55d99fa43fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f3d28003c40 tx=0x7f3d28005f10 comp rx=0 tx=0).stop 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 shutdown_connections 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55d99faaa970 0x55d99faacd60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d99fa3dc00 0x55d99fa444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55d99fa35080 0x55d99fa43fa0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 --2- 172.21.15.119:0/2564079764 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d99fa346d0 0x55d99fa45c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 >> 172.21.15.119:0/2564079764 conn(0x55d99fa2eda0 msgr2=0x55d99fa2d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d22a00640 0 client.27422 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 shutdown_connections 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.939+0000 7f3d46c38740 1 -- 172.21.15.119:0/2564079764 wait complete. 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d46c38740 20 client.27422 trim_cache size 0 max 16384 2024-03-28T18:51:15.945 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:15.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 msgr2=0x7f3d240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 0x7f3d240a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55d99fa341a0 tx=0x7f3d30004ee0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 msgr2=0x7f3d3c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009900 tx=0x7f3d30004360 comp rx=0 tx=0).stop 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 shutdown_connections 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f3d240a2970 0x7f3d240a4e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d3c1124e0 0x7f3d3c1a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d3c0764f0 0x7f3d3c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 --2- 172.21.15.119:0/3713147257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3d3c075bd0 0x7f3d3c19e650 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:15.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 >> 172.21.15.119:0/3713147257 conn(0x7f3d3c0ff2f0 msgr2=0x7f3d3c1016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:15.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 shutdown_connections 2024-03-28T18:51:15.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:15.940+0000 7f3d43e00640 1 -- 172.21.15.119:0/3713147257 wait complete. 2024-03-28T18:51:15.959 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:15.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 -- 172.21.15.119:0/2446657262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c109850 msgr2=0x7f204c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 --2- 172.21.15.119:0/2446657262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c109850 0x7f204c109c30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f203c009930 tx=0x7f203c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 -- 172.21.15.119:0/2446657262 shutdown_connections 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 --2- 172.21.15.119:0/2446657262 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 --2- 172.21.15.119:0/2446657262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c103840 0x7f204c103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 --2- 172.21.15.119:0/2446657262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c109850 0x7f204c109c30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.166+0000 7f2052400640 1 -- 172.21.15.119:0/2446657262 >> 172.21.15.119:0/2446657262 conn(0x7f204c0ff2f0 msgr2=0x7f204c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.167+0000 7f2052400640 1 -- 172.21.15.119:0/2446657262 shutdown_connections 2024-03-28T18:51:16.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.167+0000 7f2052400640 1 -- 172.21.15.119:0/2446657262 wait complete. 2024-03-28T18:51:16.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.167+0000 7f2052400640 1 Processor -- start 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 -- start start 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 0x7f204c19ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c113c70 con 0x7f204c109850 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c113af0 con 0x7f204c104200 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2052400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c113df0 con 0x7f204c103840 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f204b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f204b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49734/0 (socket says 172.21.15.119:49734) 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f204b400640 1 -- 172.21.15.119:0/1046923759 learned_addr learned my addr 172.21.15.119:0/1046923759 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f2042a00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.168+0000 7f204be00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 0x7f204c19ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 -- 172.21.15.119:0/1046923759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 msgr2=0x7f204c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 -- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 msgr2=0x7f204c19ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 0x7f204c19ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 -- 172.21.15.119:0/1046923759 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c19f160 con 0x7f204c103840 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204be00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 0x7f204c19ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f2042a00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f204b400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f203c005ab0 tx=0x7f203c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f2048c00640 1 -- 172.21.15.119:0/1046923759 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c043020 con 0x7f204c103840 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f2048c00640 1 -- 172.21.15.119:0/1046923759 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f203c005000 con 0x7f204c103840 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f2048c00640 1 -- 172.21.15.119:0/1046923759 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c003a80 con 0x7f204c103840 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.169+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c19f3f0 con 0x7f204c103840 2024-03-28T18:51:16.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.170+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c1a6ce0 con 0x7f204c103840 2024-03-28T18:51:16.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f2048c00640 1 -- 172.21.15.119:0/1046923759 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f203c046020 con 0x7f204c103840 2024-03-28T18:51:16.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 Processor -- start 2024-03-28T18:51:16.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 -- start start 2024-03-28T18:51:16.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99b9dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99b012c0 con 0x559d99b9d6d0 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99aa9c90 con 0x559d99ba6c00 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f205502e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99afd360 con 0x559d99b9e080 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f204b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f204be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99b9dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f204b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49740/0 (socket says 172.21.15.119:49740) 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f2042a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.173+0000 7f204b400640 1 -- 172.21.15.119:0/916174834 learned_addr learned my addr 172.21.15.119:0/916174834 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 -- 172.21.15.119:0/916174834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 msgr2=0x559d99ba9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 -- 172.21.15.119:0/916174834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 msgr2=0x559d99b9dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99b9dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 -- 172.21.15.119:0/916174834 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d99ace080 con 0x559d99b9e080 2024-03-28T18:51:16.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f204b400640 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f203c0ee520 tx=0x7f203c0f4e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755bc62a62f0c869 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f2048c00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 0x7f20300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f2040c00640 1 -- 172.21.15.119:0/916174834 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0f9bc0 con 0x559d99b9e080 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f2040c00640 1 -- 172.21.15.119:0/916174834 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f203c0fe070 con 0x559d99b9e080 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.174+0000 7f2040c00640 1 -- 172.21.15.119:0/916174834 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c082b00 con 0x559d99b9e080 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f2048c00640 1 -- 172.21.15.119:0/1046923759 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f203c0f0de0 con 0x7f204c103840 2024-03-28T18:51:16.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f204be00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 0x7f20300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 -- 172.21.15.119:0/916174834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 msgr2=0x559d99ba6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f203c0ee520 tx=0x7f203c0f4e90 comp rx=0 tx=0).stop 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 -- 172.21.15.119:0/916174834 shutdown_connections 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6630 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 --2- 172.21.15.119:0/916174834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99b9dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 -- 172.21.15.119:0/916174834 >> 172.21.15.119:0/916174834 conn(0x559d99b97da0 msgr2=0x559d99b98180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f204be00640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 0x7f20300a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2044005fd0 tx=0x7f2044009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 -- 172.21.15.119:0/916174834 shutdown_connections 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.175+0000 7f205502e740 1 -- 172.21.15.119:0/916174834 wait complete. 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.176+0000 7f205502e740 1 -- start start 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.176+0000 7f205502e740 1 Processor -- start 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.176+0000 7f205502e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99ba6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.176+0000 7f205502e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f205502e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99ba3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f205502e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99ba9930 con 0x559d99ba6c00 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f205502e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99abd0d0 con 0x559d99b9d6d0 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f205502e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d99ba9ab0 con 0x559d99b9e080 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f205502e740 10 client.0 _ll_register_callbacks cb 0x559d99ac1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49752/0 (socket says 172.21.15.119:49752) 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f204b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99ba6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 -- 172.21.15.119:0/113981084 learned_addr learned my addr 172.21.15.119:0/113981084 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f204be00640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99ba3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 -- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 msgr2=0x559d99ba6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99ba6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 -- 172.21.15.119:0/113981084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 msgr2=0x559d99ba3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99ba3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 -- 172.21.15.119:0/113981084 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d998153f0 con 0x559d99b9e080 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f204b400640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99ba6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2042a00640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2034012d20 tx=0x7f203400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c18eaa1c744cd71f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.177+0000 7f2039600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034012110 con 0x559d99b9e080 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2034004050 con 0x559d99b9e080 2024-03-28T18:51:16.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f203c0fe070 con 0x559d99b9e080 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20340122b0 con 0x559d99b9e080 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f205502e740 10 client.27406 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d998df7d0 con 0x559d99b9e080 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.178+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f203c0fe070 con 0x559d99b9e080 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2034022030 con 0x559d99b9e080 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 10 client.27406 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 10 client.27406 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 20 client.27406 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99bb6790 0x559d99bb8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 4 client.27406 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d99b0e800 con 0x559d99bb6790 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f204b400640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99bb6790 0x559d99bb8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f2039600640 10 client.27406 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:16.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.179+0000 7f204b400640 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99bb6790 0x559d99bb8b80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=41358164baaf1e09 server_cookie=ea0ea21d2533759a in_seq=0 out_seq=0 2024-03-28T18:51:16.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f2039600640 1 -- 172.21.15.119:0/113981084 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d99b0e800 con 0x559d99bb6790 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f2039600640 10 client.27406 handle_command_reply: tid=0 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 client.27406 shutdown 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 msgr2=0x559d99ba3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2034012d20 tx=0x7f203400f790 comp rx=0 tx=0).stop 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 shutdown_connections 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99bb6790 0x559d99bb8b80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99ba3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba3890 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 --2- 172.21.15.119:0/113981084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99ba6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f2039600640 0 client.27406 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 >> 172.21.15.119:0/113981084 conn(0x559d99b97da0 msgr2=0x559d99b99af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 shutdown_connections 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.180+0000 7f205502e740 1 -- 172.21.15.119:0/113981084 wait complete. 2024-03-28T18:51:16.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.181+0000 7f205502e740 20 client.27406 trim_cache size 0 max 16384 2024-03-28T18:51:16.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 Processor -- start 2024-03-28T18:51:16.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 -- start start 2024-03-28T18:51:16.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99ba5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1af660 con 0x559d99b9d6d0 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1b0fe0 con 0x559d99ba6c00 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.194+0000 7f205502e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1afcc0 con 0x559d99b9e080 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99ba5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f2042a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52672/0 (socket says 172.21.15.119:52672) 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204be00640 1 -- 172.21.15.119:0/260886279 learned_addr learned my addr 172.21.15.119:0/260886279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 -- 172.21.15.119:0/260886279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 msgr2=0x559d99ba6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 -- 172.21.15.119:0/260886279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 msgr2=0x559d99ba0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 -- 172.21.15.119:0/260886279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d99ba2840 con 0x559d99b9d6d0 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f2042a00640 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.195+0000 7f204b400640 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99ba5fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x559d99b9d1a0 tx=0x7f203c0dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0981fef64538c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f203be00640 1 -- 172.21.15.119:0/260886279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c086ac0 con 0x559d99b9d6d0 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f203be00640 1 -- 172.21.15.119:0/260886279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f203c0fe040 con 0x559d99b9d6d0 2024-03-28T18:51:16.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f203be00640 1 -- 172.21.15.119:0/260886279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c086c90 con 0x559d99b9d6d0 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 -- 172.21.15.119:0/260886279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 msgr2=0x559d99ba5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99ba5fc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x559d99b9d1a0 tx=0x7f203c0dd040 comp rx=0 tx=0).stop 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 -- 172.21.15.119:0/260886279 shutdown_connections 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99ba6c00 0x559d99ba0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99ba6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 --2- 172.21.15.119:0/260886279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99b9d6d0 0x559d99ba5fc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 -- 172.21.15.119:0/260886279 >> 172.21.15.119:0/260886279 conn(0x559d99b97da0 msgr2=0x559d99b96eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 -- 172.21.15.119:0/260886279 shutdown_connections 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.196+0000 7f205502e740 1 -- 172.21.15.119:0/260886279 wait complete. 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 -- start start 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 Processor -- start 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99baccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1b0980 con 0x559d99ba6c00 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1b0320 con 0x559d99b9d6d0 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1ad840 con 0x559d99b9e080 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.197+0000 7f205502e740 10 client.0 _ll_register_callbacks cb 0x7f2034009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2042a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52682/0 (socket says 172.21.15.119:52682) 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2042a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49780/0 (socket says 172.21.15.119:49780) 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 -- 172.21.15.119:0/1747832741 learned_addr learned my addr 172.21.15.119:0/1747832741 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204be00640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99baccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 -- 172.21.15.119:0/1747832741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 msgr2=0x559d99bac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 -- 172.21.15.119:0/1747832741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 msgr2=0x559d99baccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99baccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 -- 172.21.15.119:0/1747832741 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d99c11240 con 0x559d99b9d6d0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2042a00640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f204b400640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f203c0bd450 tx=0x7f203c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26ab84edf29ae1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2039600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0873e0 con 0x559d99b9d6d0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f203c0ee720 con 0x559d99b9d6d0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.198+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f203c0fe040 con 0x559d99b9d6d0 2024-03-28T18:51:16.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f203c087710 con 0x559d99b9d6d0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 10 client.27440 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d99c12520 con 0x559d99b9d6d0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f203c0fe040 con 0x559d99b9d6d0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f203c0f3040 con 0x559d99b9d6d0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 10 client.27440 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 10 client.27440 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 20 client.27440 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99c128b0 0x559d99c14ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 4 client.27440 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d99b08b50 con 0x559d99c128b0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.199+0000 7f2042a00640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99c128b0 0x559d99c14ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f2042a00640 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99c128b0 0x559d99c14ca0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=70612c3db5bbfdcb server_cookie=5ef749787e154e57 in_seq=0 out_seq=0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f2039600640 10 client.27440 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f2039600640 1 -- 172.21.15.119:0/1747832741 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d99b08b50 con 0x559d99c128b0 2024-03-28T18:51:16.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f2039600640 10 client.27440 handle_command_reply: tid=0 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 client.27440 shutdown 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 msgr2=0x559d99bac230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f203c0bd450 tx=0x7f203c009260 comp rx=0 tx=0).stop 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 shutdown_connections 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x559d99c128b0 0x559d99c14ca0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559d99ba6c00 0x559d99baccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559d99b9e080 0x559d99bac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f2039600640 0 client.27440 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 --2- 172.21.15.119:0/1747832741 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559d99b9d6d0 0x559d99bac230 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 >> 172.21.15.119:0/1747832741 conn(0x559d99b97da0 msgr2=0x559d99b9e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 shutdown_connections 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.200+0000 7f205502e740 1 -- 172.21.15.119:0/1747832741 wait complete. 2024-03-28T18:51:16.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f205502e740 20 client.27440 trim_cache size 0 max 16384 2024-03-28T18:51:16.207 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:16.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 msgr2=0x7f20300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 0x7f20300a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2044005fd0 tx=0x7f2044009500 comp rx=0 tx=0).stop 2024-03-28T18:51:16.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 msgr2=0x7f204c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f203c005ab0 tx=0x7f203c005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 shutdown_connections 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f20300a2970 0x7f20300a4e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f204c109850 0x7f204c19ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f204c104200 0x7f204c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 --2- 172.21.15.119:0/1046923759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f204c103840 0x7f204c19a170 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 >> 172.21.15.119:0/1046923759 conn(0x7f204c0ff2f0 msgr2=0x7f204c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 shutdown_connections 2024-03-28T18:51:16.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.201+0000 7f2052400640 1 -- 172.21.15.119:0/1046923759 wait complete. 2024-03-28T18:51:16.220 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:16.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:51:16.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 -- 172.21.15.119:0/1283036204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 msgr2=0x7f05fc107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 --2- 172.21.15.119:0/1283036204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc107a20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009930 tx=0x7f05f002bc20 comp rx=0 tx=0).stop 2024-03-28T18:51:16.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 -- 172.21.15.119:0/1283036204 shutdown_connections 2024-03-28T18:51:16.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 --2- 172.21.15.119:0/1283036204 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101ff0 0x7f05fc10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 --2- 172.21.15.119:0/1283036204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101630 0x7f05fc101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 --2- 172.21.15.119:0/1283036204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc107a20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 -- 172.21.15.119:0/1283036204 >> 172.21.15.119:0/1283036204 conn(0x7f05fc078aa0 msgr2=0x7f05fc078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 -- 172.21.15.119:0/1283036204 shutdown_connections 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.435+0000 7f0604800640 1 -- 172.21.15.119:0/1283036204 wait complete. 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.436+0000 7f0604800640 1 Processor -- start 2024-03-28T18:51:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.436+0000 7f0604800640 1 -- start start 2024-03-28T18:51:16.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101ff0 0x7f05fc073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc111a60 con 0x7f05fc101ff0 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1118e0 con 0x7f05fc101630 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0604800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc111be0 con 0x7f05fc107640 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0601e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0601e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52696/0 (socket says 172.21.15.119:52696) 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0601e00640 1 -- 172.21.15.119:0/2243038977 learned_addr learned my addr 172.21.15.119:0/2243038977 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0602800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.437+0000 7f0601400640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101ff0 0x7f05fc073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 -- 172.21.15.119:0/2243038977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 msgr2=0x7f05fc074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 -- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101ff0 msgr2=0x7f05fc073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101ff0 0x7f05fc073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 -- 172.21.15.119:0/2243038977 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc1a6400 con 0x7f05fc101630 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0601e00640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05f0005ab0 tx=0x7f05f0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f05f6a00640 1 -- 172.21.15.119:0/2243038977 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0043020 con 0x7f05fc101630 2024-03-28T18:51:16.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f05f6a00640 1 -- 172.21.15.119:0/2243038977 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0005000 con 0x7f05fc101630 2024-03-28T18:51:16.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1a6690 con 0x7f05fc101630 2024-03-28T18:51:16.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f05f6a00640 1 -- 172.21.15.119:0/2243038977 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0003ab0 con 0x7f05fc101630 2024-03-28T18:51:16.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.438+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc1a6c30 con 0x7f05fc101630 2024-03-28T18:51:16.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.441+0000 7f05f6a00640 1 -- 172.21.15.119:0/2243038977 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f05f0046020 con 0x7f05fc101630 2024-03-28T18:51:16.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.441+0000 7f06075e7740 1 Processor -- start 2024-03-28T18:51:16.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 -- start start 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f11fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f128c00 0x555d1f12b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f0832c0 con 0x555d1f128c00 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f02bc90 con 0x555d1f11f6d0 2024-03-28T18:51:16.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f06075e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f07f360 con 0x555d1f120080 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f0601e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f0602800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f11fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f0601e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49814/0 (socket says 172.21.15.119:49814) 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f0601e00640 1 -- 172.21.15.119:0/3944567195 learned_addr learned my addr 172.21.15.119:0/3944567195 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.442+0000 7f0601400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f128c00 0x555d1f12b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 -- 172.21.15.119:0/3944567195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 msgr2=0x555d1f11fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f11fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 -- 172.21.15.119:0/3944567195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f128c00 msgr2=0x555d1f12b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f128c00 0x555d1f12b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 -- 172.21.15.119:0/3944567195 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1f050080 con 0x555d1f120080 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0601e00640 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ee410 tx=0x7f05f00f4e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5ce7033711a3530 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f05f6a00640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 0x7f05e40a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f05eb000640 1 -- 172.21.15.119:0/3944567195 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0100020 con 0x555d1f120080 2024-03-28T18:51:16.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f0602800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 0x7f05e40a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f05eb000640 1 -- 172.21.15.119:0/3944567195 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0091e60 con 0x555d1f120080 2024-03-28T18:51:16.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f05eb000640 1 -- 172.21.15.119:0/3944567195 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00b2ad0 con 0x555d1f120080 2024-03-28T18:51:16.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.443+0000 7f05f6a00640 1 -- 172.21.15.119:0/2243038977 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f05f00f0d60 con 0x7f05fc101630 2024-03-28T18:51:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 -- 172.21.15.119:0/3944567195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 msgr2=0x555d1f128630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ee410 tx=0x7f05f00f4e00 comp rx=0 tx=0).stop 2024-03-28T18:51:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f0602800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 0x7f05e40a4ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005fd0 tx=0x7f05f8009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 -- 172.21.15.119:0/3944567195 shutdown_connections 2024-03-28T18:51:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f128c00 0x555d1f12b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f120080 0x555d1f128630 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 --2- 172.21.15.119:0/3944567195 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f11fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 -- 172.21.15.119:0/3944567195 >> 172.21.15.119:0/3944567195 conn(0x555d1f119da0 msgr2=0x555d1f11a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 -- 172.21.15.119:0/3944567195 shutdown_connections 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.444+0000 7f06075e7740 1 -- 172.21.15.119:0/3944567195 wait complete. 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.445+0000 7f06075e7740 1 -- start start 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.445+0000 7f06075e7740 1 Processor -- start 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f125cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f12b930 con 0x555d1f11f6d0 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f03f0d0 con 0x555d1f120080 2024-03-28T18:51:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1f12bab0 con 0x555d1f128c00 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f06075e7740 10 client.0 _ll_register_callbacks cb 0x555d1f043e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0602800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f125cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52716/0 (socket says 172.21.15.119:52716) 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39008/0 (socket says 172.21.15.119:39008) 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 -- 172.21.15.119:0/2118009267 learned_addr learned my addr 172.21.15.119:0/2118009267 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 -- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 msgr2=0x555d1f125cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f125cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 -- 172.21.15.119:0/2118009267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 msgr2=0x555d1f128420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601400640 1 -- 172.21.15.119:0/2118009267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d1ed973f0 con 0x555d1f120080 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.446+0000 7f0601e00640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f0601400640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f05ec012700 tx=0x7f05ec012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a95a94da4f7559 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f05df400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec004320 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f05ec0119a0 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f0091e60 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05ec0044c0 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f06075e7740 10 client.27452 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555d1ee617d0 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.447+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0091e60 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f05ec022030 con 0x555d1f120080 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 10 client.27452 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 10 client.27452 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 20 client.27452 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f138770 0x555d1f13ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 4 client.27452 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d1f090800 con 0x555d1f138770 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.448+0000 7f0601e00640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f138770 0x555d1f13ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.449+0000 7f05df400640 10 client.27452 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:16.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.449+0000 7f0601e00640 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f138770 0x555d1f13ab80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52fb64cd214165f9 server_cookie=66b4866c90fa1275 in_seq=0 out_seq=0 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.449+0000 7f05df400640 1 -- 172.21.15.119:0/2118009267 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555d1f090800 con 0x555d1f138770 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.449+0000 7f05df400640 10 client.27452 handle_command_reply: tid=0 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 client.27452 shutdown 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 msgr2=0x555d1f1257b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f05ec012700 tx=0x7f05ec012bc0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 shutdown_connections 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f138770 0x555d1f13ab80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f125cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f1257b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 --2- 172.21.15.119:0/2118009267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f128420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f05df400640 0 client.27452 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:16.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 >> 172.21.15.119:0/2118009267 conn(0x555d1f119da0 msgr2=0x555d1f11baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 shutdown_connections 2024-03-28T18:51:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 1 -- 172.21.15.119:0/2118009267 wait complete. 2024-03-28T18:51:16.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.450+0000 7f06075e7740 20 client.27452 trim_cache size 0 max 16384 2024-03-28T18:51:16.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.463+0000 7f06075e7740 1 Processor -- start 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.463+0000 7f06075e7740 1 -- start start 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f128420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f1341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1af620 con 0x555d1f11f6d0 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1b0fa0 con 0x555d1f120080 2024-03-28T18:51:16.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f06075e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1afc80 con 0x555d1f128c00 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0602800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f1341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39014/0 (socket says 172.21.15.119:39014) 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 -- 172.21.15.119:0/53959075 learned_addr learned my addr 172.21.15.119:0/53959075 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f128420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 -- 172.21.15.119:0/53959075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 msgr2=0x555d1f1341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f1341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 -- 172.21.15.119:0/53959075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 msgr2=0x555d1f128420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f128420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.464+0000 7f0601e00640 1 -- 172.21.15.119:0/53959075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1f11e9f0 con 0x555d1f11f6d0 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f0601400640 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f128420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f0602800640 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f1341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f0601e00640 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009900 tx=0x7f05f00b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4477622ae2aa11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f05ea600640 1 -- 172.21.15.119:0/53959075 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00b65f0 con 0x555d1f11f6d0 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f05ea600640 1 -- 172.21.15.119:0/53959075 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0050cd0 con 0x555d1f11f6d0 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.465+0000 7f05ea600640 1 -- 172.21.15.119:0/53959075 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00b4a20 con 0x555d1f11f6d0 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- 172.21.15.119:0/53959075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 msgr2=0x555d1f127ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009900 tx=0x7f05f00b9040 comp rx=0 tx=0).stop 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- 172.21.15.119:0/53959075 shutdown_connections 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f1341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f120080 0x555d1f128420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 --2- 172.21.15.119:0/53959075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f11f6d0 0x555d1f127ee0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- 172.21.15.119:0/53959075 >> 172.21.15.119:0/53959075 conn(0x555d1f119da0 msgr2=0x555d1f120d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- 172.21.15.119:0/53959075 shutdown_connections 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- 172.21.15.119:0/53959075 wait complete. 2024-03-28T18:51:16.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 -- start start 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.466+0000 7f06075e7740 1 Processor -- start 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f120080 0x555d1f12ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f18d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1b0940 con 0x555d1f120080 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1b02e0 con 0x555d1f11f6d0 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1ad800 con 0x555d1f128c00 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f06075e7740 10 client.0 _ll_register_callbacks cb 0x7f05ec0096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52738/0 (socket says 172.21.15.119:52738) 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 -- 172.21.15.119:0/3999311892 learned_addr learned my addr 172.21.15.119:0/3999311892 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0602800640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f18d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601400640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f120080 0x555d1f12ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 -- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 msgr2=0x555d1f18d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f18d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 -- 172.21.15.119:0/3999311892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f120080 msgr2=0x555d1f12ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f120080 0x555d1f12ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.467+0000 7f0601e00640 1 -- 172.21.15.119:0/3999311892 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d1f18d640 con 0x555d1f11f6d0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f0601e00640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f05f002c200 tx=0x7f05f00f4810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6772755f795cd56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0100070 con 0x555d1f11f6d0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f05f00bb6e0 con 0x555d1f11f6d0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05f0050cd0 con 0x555d1f11f6d0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f00b4da0 con 0x555d1f11f6d0 2024-03-28T18:51:16.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 10 client.27458 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555d1f1949e0 con 0x555d1f11f6d0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05f0050cd0 con 0x555d1f11f6d0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f05f00b9a50 con 0x555d1f11f6d0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 10 client.27458 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 10 client.27458 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 20 client.27458 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f194df0 0x555d1f1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 4 client.27458 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.468+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555d1f126080 con 0x555d1f194df0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f0601400640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f194df0 0x555d1f1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f0601400640 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f194df0 0x555d1f1971e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bbbc791d3677e8ca server_cookie=e1bc691c2a436fc7 in_seq=0 out_seq=0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f05df400640 10 client.27458 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f05df400640 1 -- 172.21.15.119:0/3999311892 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555d1f126080 con 0x555d1f194df0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f05df400640 10 client.27458 handle_command_reply: tid=0 2024-03-28T18:51:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.469+0000 7f06075e7740 1 client.27458 shutdown 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 msgr2=0x555d1f12c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f05f002c200 tx=0x7f05f00f4810 comp rx=0 tx=0).stop 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 shutdown_connections 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x555d1f194df0 0x555d1f1971e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d1f128c00 0x555d1f18d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555d1f120080 0x555d1f12ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 --2- 172.21.15.119:0/3999311892 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x555d1f11f6d0 0x555d1f12c950 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 >> 172.21.15.119:0/3999311892 conn(0x555d1f119da0 msgr2=0x555d1f118620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f05df400640 0 client.27458 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 shutdown_connections 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 1 -- 172.21.15.119:0/3999311892 wait complete. 2024-03-28T18:51:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f06075e7740 20 client.27458 trim_cache size 0 max 16384 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 msgr2=0x7f05e40a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 0x7f05e40a4ee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f05f8005fd0 tx=0x7f05f8009500 comp rx=0 tx=0).stop 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 msgr2=0x7f05fc0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.470+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f05f0005ab0 tx=0x7f05f0005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:16.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 shutdown_connections 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f05e40a2a20 0x7f05e40a4ee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05fc107640 0x7f05fc074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05fc101ff0 0x7f05fc073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 --2- 172.21.15.119:0/2243038977 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f05fc101630 0x7f05fc0739a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 >> 172.21.15.119:0/2243038977 conn(0x7f05fc078aa0 msgr2=0x7f05fc104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 shutdown_connections 2024-03-28T18:51:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.471+0000 7f0604800640 1 -- 172.21.15.119:0/2243038977 wait complete. 2024-03-28T18:51:16.490 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:16.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 -- 172.21.15.119:0/2256328229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40101630 msgr2=0x7fac40101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 --2- 172.21.15.119:0/2256328229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40101630 0x7fac40101ab0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009900 tx=0x7fac2c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 -- 172.21.15.119:0/2256328229 shutdown_connections 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 --2- 172.21.15.119:0/2256328229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101ff0 0x7fac4010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 --2- 172.21.15.119:0/2256328229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40101630 0x7fac40101ab0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 --2- 172.21.15.119:0/2256328229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40107640 0x7fac40107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 -- 172.21.15.119:0/2256328229 >> 172.21.15.119:0/2256328229 conn(0x7fac40078aa0 msgr2=0x7fac40078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 -- 172.21.15.119:0/2256328229 shutdown_connections 2024-03-28T18:51:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.700+0000 7fac46000640 1 -- 172.21.15.119:0/2256328229 wait complete. 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.701+0000 7fac46000640 1 Processor -- start 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.701+0000 7fac46000640 1 -- start start 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 0x7fac4019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40107640 0x7fac4019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40111a60 con 0x7fac40101ff0 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401118e0 con 0x7fac40107640 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac46000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40111be0 con 0x7fac40101630 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49858/0 (socket says 172.21.15.119:49858) 2024-03-28T18:51:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 -- 172.21.15.119:0/1922514322 learned_addr learned my addr 172.21.15.119:0/1922514322 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3fe00640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40107640 0x7fac4019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3ea00640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 0x7fac4019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 -- 172.21.15.119:0/1922514322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40107640 msgr2=0x7fac4019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40107640 0x7fac4019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 -- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 msgr2=0x7fac4019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 0x7fac4019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3f400640 1 -- 172.21.15.119:0/1922514322 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4019f0e0 con 0x7fac40101630 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.702+0000 7fac3ea00640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 0x7fac4019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac3f400640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac3000ebc0 tx=0x7fac3000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac37e00640 1 -- 172.21.15.119:0/1922514322 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30010630 con 0x7fac40101630 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac37e00640 1 -- 172.21.15.119:0/1922514322 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac30010cb0 con 0x7fac40101630 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4019f3d0 con 0x7fac40101630 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac37e00640 1 -- 172.21.15.119:0/1922514322 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30018bd0 con 0x7fac40101630 2024-03-28T18:51:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.703+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac401a6c60 con 0x7fac40101630 2024-03-28T18:51:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.707+0000 7fac48ddf740 1 Processor -- start 2024-03-28T18:51:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.707+0000 7fac48ddf740 1 -- start start 2024-03-28T18:51:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac37e00640 1 -- 172.21.15.119:0/1922514322 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fac300107d0 con 0x7fac40101630 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2be050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b2162c0 con 0x55ec4b2bbc00 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b1bec90 con 0x55ec4b2b26d0 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac48ddf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b212360 con 0x55ec4b2b3080 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52756/0 (socket says 172.21.15.119:52756) 2024-03-28T18:51:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 -- 172.21.15.119:0/3338609882 learned_addr learned my addr 172.21.15.119:0/3338609882 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3ea00640 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2be050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 -- 172.21.15.119:0/3338609882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 msgr2=0x55ec4b2bb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 -- 172.21.15.119:0/3338609882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2bbc00 msgr2=0x55ec4b2be050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.708+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2be050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac3fe00640 1 -- 172.21.15.119:0/3338609882 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec4b1e3080 con 0x55ec4b2b26d0 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ca0 tx=0x7fac38011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76f091b97c1dce5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac2be00640 1 -- 172.21.15.119:0/3338609882 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3801e070 con 0x55ec4b2b26d0 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac2be00640 1 -- 172.21.15.119:0/3338609882 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac38009e50 con 0x55ec4b2b26d0 2024-03-28T18:51:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac2be00640 1 -- 172.21.15.119:0/3338609882 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3800ea20 con 0x55ec4b2b26d0 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.709+0000 7fac37e00640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 0x7fac240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- 172.21.15.119:0/3338609882 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 msgr2=0x55ec4b2b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ca0 tx=0x7fac38011810 comp rx=0 tx=0).stop 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac37e00640 1 -- 172.21.15.119:0/1922514322 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fac300cf540 con 0x7fac40101630 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- 172.21.15.119:0/3338609882 shutdown_connections 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2be050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3338609882 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2b2ab0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- 172.21.15.119:0/3338609882 >> 172.21.15.119:0/3338609882 conn(0x55ec4b2acda0 msgr2=0x55ec4b2ad180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac3f400640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 0x7fac240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- 172.21.15.119:0/3338609882 shutdown_connections 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- 172.21.15.119:0/3338609882 wait complete. 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac3f400640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 0x7fac240a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac300cccb0 tx=0x7fac300d5110 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.710+0000 7fac48ddf740 1 -- start start 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 Processor -- start 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b2beb70 con 0x55ec4b2b3080 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b1d20d0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec4b2becf0 con 0x55ec4b2bbc00 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac48ddf740 10 client.0 _ll_register_callbacks cb 0x55ec4b1d6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac3ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52772/0 (socket says 172.21.15.119:52772) 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.711+0000 7fac3fe00640 1 -- 172.21.15.119:0/227273505 learned_addr learned my addr 172.21.15.119:0/227273505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 -- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 msgr2=0x55ec4b2b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 -- 172.21.15.119:0/227273505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 msgr2=0x55ec4b2b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 -- 172.21.15.119:0/227273505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec4af2a3f0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac3fe00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e6c0 tx=0x7fac3800e5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59f742bb88a09f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac28c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3802e060 con 0x55ec4b2b26d0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac38009e50 con 0x55ec4b2b26d0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.712+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fac38004260 con 0x55ec4b2b26d0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac3801e030 con 0x55ec4b2b26d0 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 10 client.27470 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ec4aff47d0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac38009e50 con 0x55ec4b2b26d0 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fac38019430 con 0x55ec4b2b26d0 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 10 client.27470 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 10 client.27470 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 20 client.27470 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b2cb910 0x55ec4b2cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 4 client.27470 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec4b223800 con 0x55ec4b2cb910 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.713+0000 7fac3ea00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b2cb910 0x55ec4b2cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.714+0000 7fac3ea00640 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b2cb910 0x55ec4b2cdd00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b003b27ac575162 server_cookie=d61d3c665a5f553f in_seq=0 out_seq=0 2024-03-28T18:51:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.714+0000 7fac28c00640 10 client.27470 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac28c00640 1 -- 172.21.15.119:0/227273505 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec4b223800 con 0x55ec4b2cb910 2024-03-28T18:51:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac28c00640 10 client.27470 handle_command_reply: tid=0 2024-03-28T18:51:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 client.27470 shutdown 2024-03-28T18:51:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 msgr2=0x55ec4b2bb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e6c0 tx=0x7fac3800e5c0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 shutdown_connections 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b2cb910 0x55ec4b2cdd00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac28c00640 0 client.27470 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 --2- 172.21.15.119:0/227273505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bb4a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 >> 172.21.15.119:0/227273505 conn(0x55ec4b2acda0 msgr2=0x55ec4b2aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 shutdown_connections 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 1 -- 172.21.15.119:0/227273505 wait complete. 2024-03-28T18:51:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.715+0000 7fac48ddf740 20 client.27470 trim_cache size 0 max 16384 2024-03-28T18:51:16.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 Processor -- start 2024-03-28T18:51:16.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 -- start start 2024-03-28T18:51:16.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2baf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401af610 con 0x55ec4b2b26d0 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401b0f90 con 0x55ec4b2bbc00 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac48ddf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401afc70 con 0x55ec4b2b3080 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac3ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2baf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39084/0 (socket says 172.21.15.119:39084) 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.729+0000 7fac3fe00640 1 -- 172.21.15.119:0/893066027 learned_addr learned my addr 172.21.15.119:0/893066027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3fe00640 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 -- 172.21.15.119:0/893066027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2bbc00 msgr2=0x55ec4b2bb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 -- 172.21.15.119:0/893066027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 msgr2=0x55ec4b2ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 -- 172.21.15.119:0/893066027 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec4b2b4400 con 0x55ec4b2b3080 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3fe00640 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac3ea00640 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2baf60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fac2c02c170 tx=0x7fac2c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2609de718baf0ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac2b400640 1 -- 172.21.15.119:0/893066027 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0462d0 con 0x55ec4b2b3080 2024-03-28T18:51:16.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac2b400640 1 -- 172.21.15.119:0/893066027 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac2c04c070 con 0x55ec4b2b3080 2024-03-28T18:51:16.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.730+0000 7fac2b400640 1 -- 172.21.15.119:0/893066027 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c005460 con 0x55ec4b2b3080 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- 172.21.15.119:0/893066027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 msgr2=0x55ec4b2baf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2baf60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fac2c02c170 tx=0x7fac2c02e040 comp rx=0 tx=0).stop 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- 172.21.15.119:0/893066027 shutdown_connections 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2b3080 0x55ec4b2baf60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 --2- 172.21.15.119:0/893066027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- 172.21.15.119:0/893066027 >> 172.21.15.119:0/893066027 conn(0x55ec4b2acda0 msgr2=0x55ec4b2b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- 172.21.15.119:0/893066027 shutdown_connections 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- 172.21.15.119:0/893066027 wait complete. 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.731+0000 7fac48ddf740 1 -- start start 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 Processor -- start 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401b0930 con 0x55ec4b2b3080 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401b02d0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401ad7f0 con 0x55ec4b2bbc00 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac48ddf740 10 client.0 _ll_register_callbacks cb 0x7fac3801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac3fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52786/0 (socket says 172.21.15.119:52786) 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac3fe00640 1 -- 172.21.15.119:0/3305280809 learned_addr learned my addr 172.21.15.119:0/3305280809 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac3ea00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.732+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 -- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 msgr2=0x55ec4b2bff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 -- 172.21.15.119:0/3305280809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 msgr2=0x55ec4b2bfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 -- 172.21.15.119:0/3305280809 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec4b325fa0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3ea00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac3fe00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fac38019800 tx=0x7fac3800e630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536f450a8e7fb0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac28c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3802e040 con 0x55ec4b2b26d0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac2c04c070 con 0x55ec4b2b26d0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fac3801f040 con 0x55ec4b2b26d0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac3800ea70 con 0x55ec4b2b26d0 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac48ddf740 10 client.27476 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.733+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ec4b3272a0 con 0x55ec4b2b26d0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac2c04c070 con 0x55ec4b2b26d0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fac38003b60 con 0x55ec4b2b26d0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 10 client.27476 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 10 client.27476 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 20 client.27476 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b327890 0x55ec4b329cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 4 client.27476 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec4b21db50 con 0x55ec4b327890 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.734+0000 7fac3ea00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b327890 0x55ec4b329cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac28c00640 10 client.27476 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac3ea00640 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b327890 0x55ec4b329cb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4267c90a4e1af6e server_cookie=6021d5fd1a4a3eb in_seq=0 out_seq=0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac28c00640 1 -- 172.21.15.119:0/3305280809 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec4b21db50 con 0x55ec4b327890 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac28c00640 10 client.27476 handle_command_reply: tid=0 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 client.27476 shutdown 2024-03-28T18:51:16.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 msgr2=0x55ec4b2bf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fac38019800 tx=0x7fac3800e630 comp rx=0 tx=0).stop 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 shutdown_connections 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ec4b327890 0x55ec4b329cb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ec4b2bbc00 0x55ec4b2bff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac28c00640 0 client.27476 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ec4b2b3080 0x55ec4b2bfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 --2- 172.21.15.119:0/3305280809 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ec4b2b26d0 0x55ec4b2bf500 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 >> 172.21.15.119:0/3305280809 conn(0x55ec4b2acda0 msgr2=0x55ec4b2b8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 shutdown_connections 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 1 -- 172.21.15.119:0/3305280809 wait complete. 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.735+0000 7fac48ddf740 20 client.27476 trim_cache size 0 max 16384 2024-03-28T18:51:16.741 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:16.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 msgr2=0x7fac240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 0x7fac240a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fac300cccb0 tx=0x7fac300d5110 comp rx=0 tx=0).stop 2024-03-28T18:51:16.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 msgr2=0x7fac4019a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fac3000ebc0 tx=0x7fac3000c6a0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 shutdown_connections 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fac240a2970 0x7fac240a4e30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fac40107640 0x7fac4019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac40101ff0 0x7fac4019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 --2- 172.21.15.119:0/1922514322 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac40101630 0x7fac4019a0f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 >> 172.21.15.119:0/1922514322 conn(0x7fac40078aa0 msgr2=0x7fac4010ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 shutdown_connections 2024-03-28T18:51:16.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.736+0000 7fac46000640 1 -- 172.21.15.119:0/1922514322 wait complete. 2024-03-28T18:51:16.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:16 smithi119 ceph-mon[31813]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:51:16.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:16 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3676784567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:16.755 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:16.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:51:16.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:16 smithi196 ceph-mon[36300]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:51:16.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:16 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3676784567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:16 smithi131 ceph-mon[36065]: pgmap v221: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:51:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:16 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3676784567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 -- 172.21.15.119:0/1740340792 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 msgr2=0x7fe9e410d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1740340792 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e410d970 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4009900 tx=0x7fe9d402bcc0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 -- 172.21.15.119:0/1740340792 shutdown_connections 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1740340792 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 0x7fe9e41102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1740340792 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e410d970 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1740340792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e4073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 -- 172.21.15.119:0/1740340792 >> 172.21.15.119:0/1740340792 conn(0x7fe9e40fd100 msgr2=0x7fe9e40ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 -- 172.21.15.119:0/1740340792 shutdown_connections 2024-03-28T18:51:16.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.972+0000 7fe9ec600640 1 -- 172.21.15.119:0/1740340792 wait complete. 2024-03-28T18:51:16.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 Processor -- start 2024-03-28T18:51:16.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 -- start start 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 0x7fe9e419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4113c60 con 0x7fe9e4073a70 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4113ae0 con 0x7fe9e410deb0 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.974+0000 7fe9ec600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4113de0 con 0x7fe9e4074390 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39112/0 (socket says 172.21.15.119:39112) 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1195419887 learned_addr learned my addr 172.21.15.119:0/1195419887 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 0x7fe9e419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 -- 172.21.15.119:0/1195419887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 msgr2=0x7fe9e419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 0x7fe9e419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 -- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 msgr2=0x7fe9e419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 -- 172.21.15.119:0/1195419887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e419f1e0 con 0x7fe9e4074390 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.975+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e419a780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe9d402c200 tx=0x7fe9d4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.976+0000 7fe9e2000640 1 -- 172.21.15.119:0/1195419887 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4042020 con 0x7fe9e4074390 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.976+0000 7fe9e2000640 1 -- 172.21.15.119:0/1195419887 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d402cdf0 con 0x7fe9e4074390 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.976+0000 7fe9e2000640 1 -- 172.21.15.119:0/1195419887 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4003a50 con 0x7fe9e4074390 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.976+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e419f470 con 0x7fe9e4074390 2024-03-28T18:51:16.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.976+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e41a6d60 con 0x7fe9e4074390 2024-03-28T18:51:16.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.979+0000 7fe9ef3ad740 1 Processor -- start 2024-03-28T18:51:16.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.979+0000 7fe9ef3ad740 1 -- start start 2024-03-28T18:51:16.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.979+0000 7fe9e2000640 1 -- 172.21.15.119:0/1195419887 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe9d4045020 con 0x7fe9e4074390 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ee3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99eecc00 0x561d99eef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99e472c0 con 0x561d99ee36d0 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99defc90 con 0x561d99ee4080 2024-03-28T18:51:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99e43360 con 0x561d99eecc00 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9ea600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ee3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52814/0 (socket says 172.21.15.119:52814) 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 -- 172.21.15.119:0/4186159883 learned_addr learned my addr 172.21.15.119:0/4186159883 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9200640 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99eecc00 0x561d99eef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 -- 172.21.15.119:0/4186159883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99eecc00 msgr2=0x561d99eef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99eecc00 0x561d99eef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 -- 172.21.15.119:0/4186159883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 msgr2=0x561d99ee3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ee3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.980+0000 7fe9e9c00640 1 -- 172.21.15.119:0/4186159883 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d99e14080 con 0x561d99ee4080 2024-03-28T18:51:16.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8011650 tx=0x7fe9d8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7c477b1280ec17 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9d2a00640 1 -- 172.21.15.119:0/4186159883 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d801e070 con 0x561d99ee4080 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9d2a00640 1 -- 172.21.15.119:0/4186159883 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d800fba0 con 0x561d99ee4080 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9d2a00640 1 -- 172.21.15.119:0/4186159883 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8019d80 con 0x561d99ee4080 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9e2000640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 0x7fe9cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9e2000640 1 -- 172.21.15.119:0/1195419887 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fe9d40f0830 con 0x7fe9e4074390 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 0x7fe9cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/4186159883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 msgr2=0x561d99eec630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.981+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8011650 tx=0x7fe9d8009f90 comp rx=0 tx=0).stop 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/4186159883 shutdown_connections 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99eecc00 0x561d99eef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec630 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/4186159883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ee3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/4186159883 >> 172.21.15.119:0/4186159883 conn(0x561d99eddda0 msgr2=0x561d99ede180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/4186159883 shutdown_connections 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 0x7fe9cc0a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe9e406ad50 tx=0x7fe9dc009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/4186159883 wait complete. 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.982+0000 7fe9ef3ad740 1 -- start start 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 Processor -- start 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee4080 0x561d99ee98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99ee9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99eefb70 con 0x561d99ee4080 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99e030d0 con 0x561d99eecc00 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d99eefcf0 con 0x561d99ee36d0 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ef3ad740 10 client.0 _ll_register_callbacks cb 0x561d99e07e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49932/0 (socket says 172.21.15.119:49932) 2024-03-28T18:51:16.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1761060641 learned_addr learned my addr 172.21.15.119:0/1761060641 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee4080 0x561d99ee98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.983+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99ee9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 msgr2=0x561d99ee9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99ee9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1761060641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee4080 msgr2=0x561d99ee98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee4080 0x561d99ee98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1761060641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d99b5b3f0 con 0x561d99ee36d0 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c00 tx=0x7fe9d8009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34cef99bad8fb490 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:16.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9c7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:16.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8027060 con 0x561d99ee36d0 2024-03-28T18:51:16.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fe9d800fd50 con 0x561d99ee36d0 2024-03-28T18:51:16.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9d800fba0 con 0x561d99ee36d0 2024-03-28T18:51:16.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9d802e050 con 0x561d99ee36d0 2024-03-28T18:51:16.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.984+0000 7fe9ef3ad740 10 client.27460 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d99c257d0 con 0x561d99ee36d0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d800fba0 con 0x561d99ee36d0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe9d8021070 con 0x561d99ee36d0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 10 client.27460 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 10 client.27460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 20 client.27460 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99efc9c0 0x561d99efedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 4 client.27460 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d99e54800 con 0x561d99efc9c0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.985+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99efc9c0 0x561d99efedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.986+0000 7fe9e9200640 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99efc9c0 0x561d99efedb0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=950f7c36830c2fb2 server_cookie=6db61e32e81a8c86 in_seq=0 out_seq=0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.986+0000 7fe9c7400640 10 client.27460 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9c7400640 1 -- 172.21.15.119:0/1761060641 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d99e54800 con 0x561d99efc9c0 2024-03-28T18:51:16.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9c7400640 10 client.27460 handle_command_reply: tid=0 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 client.27460 shutdown 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 msgr2=0x561d99eec510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c00 tx=0x7fe9d8009ef0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 shutdown_connections 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99efc9c0 0x561d99efedb0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99ee9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee4080 0x561d99ee98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1761060641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eec510 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 >> 172.21.15.119:0/1761060641 conn(0x561d99eddda0 msgr2=0x561d99edfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9c7400640 0 client.27460 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 shutdown_connections 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1761060641 wait complete. 2024-03-28T18:51:16.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:16.987+0000 7fe9ef3ad740 20 client.27460 trim_cache size 0 max 16384 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 Processor -- start 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 -- start start 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 0x561d99edcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41af6d0 con 0x561d99eecc00 2024-03-28T18:51:17.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41b1050 con 0x561d99ee4080 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41afd30 con 0x561d99ee36d0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49948/0 (socket says 172.21.15.119:49948) 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9e9200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.001+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1152309872 learned_addr learned my addr 172.21.15.119:0/1152309872 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 0x561d99edcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1152309872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 msgr2=0x561d99eec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1152309872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 msgr2=0x561d99edcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 0x561d99edcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 -- 172.21.15.119:0/1152309872 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d99eefa50 con 0x561d99ee36d0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9ea600640 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 0x561d99edcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80198b0 tx=0x7fe9d80059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9380cf348d462585 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9d2000640 1 -- 172.21.15.119:0/1152309872 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8015020 con 0x561d99ee36d0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9d2000640 1 -- 172.21.15.119:0/1152309872 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8005040 con 0x561d99ee36d0 2024-03-28T18:51:17.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.002+0000 7fe9d2000640 1 -- 172.21.15.119:0/1152309872 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8027070 con 0x561d99ee36d0 2024-03-28T18:51:17.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1152309872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 msgr2=0x561d99eebfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:17.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80198b0 tx=0x7fe9d80059b0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1152309872 shutdown_connections 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99eecc00 0x561d99edcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99ee4080 0x561d99eec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/1152309872 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee36d0 0x561d99eebfd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1152309872 >> 172.21.15.119:0/1152309872 conn(0x561d99eddda0 msgr2=0x561d99edc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1152309872 shutdown_connections 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/1152309872 wait complete. 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.003+0000 7fe9ef3ad740 1 -- start start 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 Processor -- start 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ef2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99f50fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41b09f0 con 0x561d99ee36d0 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41b0390 con 0x561d99eecc00 2024-03-28T18:51:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41ad8b0 con 0x561d99ee4080 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ef3ad740 10 client.0 _ll_register_callbacks cb 0x7fe9d801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9e9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9e9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49950/0 (socket says 172.21.15.119:49950) 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9e9200640 1 -- 172.21.15.119:0/3414213083 learned_addr learned my addr 172.21.15.119:0/3414213083 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ef2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.004+0000 7fe9ea600640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99f50fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 -- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 msgr2=0x561d99f50fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99f50fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 -- 172.21.15.119:0/3414213083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 msgr2=0x561d99ef2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ef2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 -- 172.21.15.119:0/3414213083 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d99f515d0 con 0x561d99ee4080 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9e9200640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40b9310 tx=0x7fe9d40f2c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e72bc30e377a4b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9c7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d40bc810 con 0x561d99ee4080 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9d8005040 con 0x561d99ee4080 2024-03-28T18:51:17.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fe9d40b9520 con 0x561d99ee4080 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9d40bd5b0 con 0x561d99ee4080 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9ef3ad740 10 client.27466 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561d99f589b0 con 0x561d99ee4080 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.005+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8005040 con 0x561d99ee4080 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe9d40ee110 con 0x561d99ee4080 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 10 client.27466 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 10 client.27466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 20 client.27466 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99f59030 0x561d99f5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 4 client.27466 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d99e4eb50 con 0x561d99f59030 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99f59030 0x561d99f5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9e9c00640 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99f59030 0x561d99f5b420 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48b3a7397282144b server_cookie=d6d0128191a10ac4 in_seq=0 out_seq=0 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9c7400640 10 client.27466 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9c7400640 1 -- 172.21.15.119:0/3414213083 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d99e4eb50 con 0x561d99f59030 2024-03-28T18:51:17.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.006+0000 7fe9c7400640 10 client.27466 handle_command_reply: tid=0 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 client.27466 shutdown 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 msgr2=0x561d99ef3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40b9310 tx=0x7fe9d40f2c80 comp rx=0 tx=0).stop 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 shutdown_connections 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561d99f59030 0x561d99f5b420 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561d99eecc00 0x561d99f50fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561d99ee4080 0x561d99ef3120 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 --2- 172.21.15.119:0/3414213083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d99ee36d0 0x561d99ef2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9c7400640 0 client.27466 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 >> 172.21.15.119:0/3414213083 conn(0x561d99eddda0 msgr2=0x561d99edc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 shutdown_connections 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 1 -- 172.21.15.119:0/3414213083 wait complete. 2024-03-28T18:51:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.007+0000 7fe9ef3ad740 20 client.27466 trim_cache size 0 max 16384 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 msgr2=0x7fe9cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 0x7fe9cc0a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe9e406ad50 tx=0x7fe9dc009500 comp rx=0 tx=0).stop 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 msgr2=0x7fe9e419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e419a780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe9d402c200 tx=0x7fe9d4005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 shutdown_connections 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe9cc0a29f0 0x7fe9cc0a4eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe9e410deb0 0x7fe9e419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4074390 0x7fe9e419a780 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 --2- 172.21.15.119:0/1195419887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4073a70 0x7fe9e419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 >> 172.21.15.119:0/1195419887 conn(0x7fe9e40fd100 msgr2=0x7fe9e40fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 shutdown_connections 2024-03-28T18:51:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:17.008+0000 7fe9ec600640 1 -- 172.21.15.119:0/1195419887 wait complete. 2024-03-28T18:51:18.027 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:51:18.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:51:18.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:51:18.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 -- 172.21.15.119:0/97844261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e0010de60 msgr2=0x7f2e00110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 --2- 172.21.15.119:0/97844261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e0010de60 0x7f2e00110250 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2df8009900 tx=0x7f2df802bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:18.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 -- 172.21.15.119:0/97844261 shutdown_connections 2024-03-28T18:51:18.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 --2- 172.21.15.119:0/97844261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e0010de60 0x7f2e00110250 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 --2- 172.21.15.119:0/97844261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e00100c60 0x7f2e0010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 --2- 172.21.15.119:0/97844261 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e001002b0 0x7f2e00100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 -- 172.21.15.119:0/97844261 >> 172.21.15.119:0/97844261 conn(0x7f2e000fbef0 msgr2=0x7f2e000fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 -- 172.21.15.119:0/97844261 shutdown_connections 2024-03-28T18:51:18.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.255+0000 7f2e04e00640 1 -- 172.21.15.119:0/97844261 wait complete. 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.256+0000 7f2e04e00640 1 Processor -- start 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.256+0000 7f2e04e00640 1 -- start start 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e0010de60 0x7f2e0019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00113b60 con 0x7f2e001002b0 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001139e0 con 0x7f2e0010de60 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2e04e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00113ce0 con 0x7f2e00100c60 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49972/0 (socket says 172.21.15.119:49972) 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39180/0 (socket says 172.21.15.119:39180) 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 -- 172.21.15.119:0/2294806770 learned_addr learned my addr 172.21.15.119:0/2294806770 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dffe00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e0010de60 0x7f2e0019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 -- 172.21.15.119:0/2294806770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e0010de60 msgr2=0x7f2e0019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e0010de60 0x7f2e0019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 -- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 msgr2=0x7f2e0019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dfea00640 1 -- 172.21.15.119:0/2294806770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0019efa0 con 0x7f2e00100c60 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.257+0000 7f2dff400640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2dfea00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ebc0 tx=0x7f2de800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2df7e00640 1 -- 172.21.15.119:0/2294806770 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8010630 con 0x7f2e00100c60 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2df7e00640 1 -- 172.21.15.119:0/2294806770 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de8010cb0 con 0x7f2e00100c60 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2df7e00640 1 -- 172.21.15.119:0/2294806770 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8018bd0 con 0x7f2e00100c60 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0019f290 con 0x7f2e00100c60 2024-03-28T18:51:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.258+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e001a6b20 con 0x7f2e00100c60 2024-03-28T18:51:18.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.261+0000 7f2e0922f740 1 Processor -- start 2024-03-28T18:51:18.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.261+0000 7f2df7e00640 1 -- 172.21.15.119:0/2294806770 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2de80107d0 con 0x7f2e00100c60 2024-03-28T18:51:18.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.261+0000 7f2e0922f740 1 -- start start 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.261+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57be0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be0bb0 0x560f57be9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57bebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2e0922f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57b44eb0 con 0x560f57be0290 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2e0922f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57aeda60 con 0x560f57be9790 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2e0922f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57b40f50 con 0x560f57be0bb0 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dffe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57be0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57bebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be0bb0 0x560f57be9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dffe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57be0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39186/0 (socket says 172.21.15.119:39186) 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dffe00640 1 -- 172.21.15.119:0/3115062505 learned_addr learned my addr 172.21.15.119:0/3115062505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 -- 172.21.15.119:0/3115062505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be0bb0 msgr2=0x560f57be9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be0bb0 0x560f57be9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 -- 172.21.15.119:0/3115062505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 msgr2=0x560f57be0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57be0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.262+0000 7f2dfea00640 1 -- 172.21.15.119:0/3115062505 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f579217e0 con 0x560f57be9790 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57bebb80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2de80ccaa0 tx=0x7f2de80d5a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904f004823fd4c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2df7e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 0x7f2de00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2df4c00640 1 -- 172.21.15.119:0/3115062505 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de80e2070 con 0x560f57be9790 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2df4c00640 1 -- 172.21.15.119:0/3115062505 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de8096350 con 0x560f57be9790 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2df4c00640 1 -- 172.21.15.119:0/3115062505 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de809fb00 con 0x560f57be9790 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2dffe00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 0x7f2de00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.263+0000 7f2df7e00640 1 -- 172.21.15.119:0/2294806770 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2de80cf3e0 con 0x7f2e00100c60 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- 172.21.15.119:0/3115062505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 msgr2=0x560f57bebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57bebb80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2de80ccaa0 tx=0x7f2de80d5a50 comp rx=0 tx=0).stop 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- 172.21.15.119:0/3115062505 shutdown_connections 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57bebb80 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be0bb0 0x560f57be9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3115062505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57be0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- 172.21.15.119:0/3115062505 >> 172.21.15.119:0/3115062505 conn(0x560f57bda9c0 msgr2=0x560f57bdada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2dffe00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 0x7f2de00a4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2df8006010 tx=0x7f2df802df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- 172.21.15.119:0/3115062505 shutdown_connections 2024-03-28T18:51:18.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- 172.21.15.119:0/3115062505 wait complete. 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.264+0000 7f2e0922f740 1 -- start start 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 Processor -- start 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57be65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57bec580 con 0x560f57be0bb0 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57b00eb0 con 0x560f57be0290 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57bec700 con 0x560f57be9790 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2e0922f740 10 client.0 _ll_register_callbacks cb 0x560f57af5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2dff400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2dffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2dff400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52876/0 (socket says 172.21.15.119:52876) 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2dff400640 1 -- 172.21.15.119:0/3133082114 learned_addr learned my addr 172.21.15.119:0/3133082114 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.265+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57be65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 -- 172.21.15.119:0/3133082114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 msgr2=0x560f57be6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 -- 172.21.15.119:0/3133082114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 msgr2=0x560f57be65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57be65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 -- 172.21.15.119:0/3133082114 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f57981f70 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dffe00640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2dff400640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2df0012d20 tx=0x7f2df000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf444edf3e5187e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2ded600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0012110 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2df0004050 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2de8096350 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df00122b0 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2e0922f740 10 client.27506 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.266+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560f57819170 con 0x560f57be0290 2024-03-28T18:51:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de8096350 con 0x560f57be0290 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2df0022030 con 0x560f57be0290 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 10 client.27506 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 10 client.27506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 20 client.27506 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57bf9920 0x560f57bfbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 4 client.27506 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f57b523f0 con 0x560f57bf9920 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.267+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57bf9920 0x560f57bfbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.268+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57bf9920 0x560f57bfbd30 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7568861a6d1101d server_cookie=b347df719a207065 in_seq=0 out_seq=0 2024-03-28T18:51:18.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.268+0000 7f2ded600640 10 client.27506 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2ded600640 1 -- 172.21.15.119:0/3133082114 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560f57b523f0 con 0x560f57bf9920 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2ded600640 10 client.27506 handle_command_reply: tid=0 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 client.27506 shutdown 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 msgr2=0x560f57be9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2df0012d20 tx=0x7f2df000f790 comp rx=0 tx=0).stop 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 shutdown_connections 2024-03-28T18:51:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57bf9920 0x560f57bfbd30 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57be65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2ded600640 0 client.27506 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3133082114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0290 0x560f57be9240 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 >> 172.21.15.119:0/3133082114 conn(0x560f57bda9c0 msgr2=0x560f57bdc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 shutdown_connections 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.269+0000 7f2e0922f740 1 -- 172.21.15.119:0/3133082114 wait complete. 2024-03-28T18:51:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.270+0000 7f2e0922f740 20 client.27506 trim_cache size 0 max 16384 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.287+0000 7f2e0922f740 1 Processor -- start 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 -- start start 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 0x560f57be8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001af450 con 0x560f57be0290 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001b0dd0 con 0x560f57be0bb0 2024-03-28T18:51:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2e0922f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001afab0 con 0x560f57be9790 2024-03-28T18:51:18.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39206/0 (socket says 172.21.15.119:39206) 2024-03-28T18:51:18.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2dff400640 1 -- 172.21.15.119:0/3719201963 learned_addr learned my addr 172.21.15.119:0/3719201963 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 0x560f57be8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.288+0000 7f2dffe00640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 -- 172.21.15.119:0/3719201963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 msgr2=0x560f57be9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 -- 172.21.15.119:0/3719201963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 msgr2=0x560f57be8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 0x560f57be8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 -- 172.21.15.119:0/3719201963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f57be08b0 con 0x560f57be0290 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dfea00640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 0x560f57be8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2dff400640 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009cb0 tx=0x7f2df00059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e726f5f1c3eef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2defe00640 1 -- 172.21.15.119:0/3719201963 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0010760 con 0x560f57be0290 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2defe00640 1 -- 172.21.15.119:0/3719201963 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2df0010900 con 0x560f57be0290 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.289+0000 7f2defe00640 1 -- 172.21.15.119:0/3719201963 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0003850 con 0x560f57be0290 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- 172.21.15.119:0/3719201963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 msgr2=0x560f57bd9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009cb0 tx=0x7f2df00059b0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- 172.21.15.119:0/3719201963 shutdown_connections 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57be9790 0x560f57be9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be0bb0 0x560f57be8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 --2- 172.21.15.119:0/3719201963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0290 0x560f57bd9900 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- 172.21.15.119:0/3719201963 >> 172.21.15.119:0/3719201963 conn(0x560f57bda9c0 msgr2=0x560f57bd91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- 172.21.15.119:0/3719201963 shutdown_connections 2024-03-28T18:51:18.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- 172.21.15.119:0/3719201963 wait complete. 2024-03-28T18:51:18.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.290+0000 7f2e0922f740 1 -- start start 2024-03-28T18:51:18.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 Processor -- start 2024-03-28T18:51:18.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57beee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 0x560f57c551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001b0770 con 0x560f57be0bb0 2024-03-28T18:51:18.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001b0110 con 0x560f57be9790 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001ad630 con 0x560f57bef3a0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2e0922f740 10 client.0 _ll_register_callbacks cb 0x7f2df0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2dffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 0x560f57c551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2dfea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57beee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2dff400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39222/0 (socket says 172.21.15.119:39222) 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.291+0000 7f2dff400640 1 -- 172.21.15.119:0/1060322597 learned_addr learned my addr 172.21.15.119:0/1060322597 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 -- 172.21.15.119:0/1060322597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 msgr2=0x560f57c551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 0x560f57c551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 -- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 msgr2=0x560f57beee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57beee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 -- 172.21.15.119:0/1060322597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f57c55710 con 0x560f57be0bb0 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dffe00640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 0x560f57c551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dfea00640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57beee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2dff400640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2df0010570 tx=0x7f2df0005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade7c0fc5cc1be10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2ded600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0031070 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2df00035f0 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.292+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2df0010900 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2df0010e20 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 10 client.17280 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560f57c569b0 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2df0010900 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2df0034050 con 0x560f57be0bb0 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 10 client.17280 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 20 client.17280 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57c56fb0 0x560f57c593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 4 client.17280 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:51:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.293+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560f57b4c740 con 0x560f57c56fb0 2024-03-28T18:51:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.294+0000 7f2dfea00640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57c56fb0 0x560f57c593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.294+0000 7f2dfea00640 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57c56fb0 0x560f57c593a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc90b5a8aecd1e87 server_cookie=1685070c150f8638 in_seq=0 out_seq=0 2024-03-28T18:51:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.294+0000 7f2ded600640 10 client.17280 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2ded600640 1 -- 172.21.15.119:0/1060322597 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560f57b4c740 con 0x560f57c56fb0 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2ded600640 10 client.17280 handle_command_reply: tid=0 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 client.17280 shutdown 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 msgr2=0x560f57bee920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2df0010570 tx=0x7f2df0005680 comp rx=0 tx=0).stop 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 shutdown_connections 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560f57c56fb0 0x560f57c593a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560f57bef3a0 0x560f57c551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560f57be9790 0x560f57beee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 --2- 172.21.15.119:0/1060322597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57be0bb0 0x560f57bee920 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2ded600640 0 client.17280 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 >> 172.21.15.119:0/1060322597 conn(0x560f57bda9c0 msgr2=0x560f57bd9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.295+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 shutdown_connections 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e0922f740 1 -- 172.21.15.119:0/1060322597 wait complete. 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e0922f740 20 client.17280 trim_cache size 0 max 16384 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "6cd9de29-c8b4-489f-9cb4-daad83e9ad99", 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:51:18.300 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:18.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 msgr2=0x7f2de00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 0x7f2de00a4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2df8006010 tx=0x7f2df802df90 comp rx=0 tx=0).stop 2024-03-28T18:51:18.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 msgr2=0x7f2e0019a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2de800ebc0 tx=0x7f2de800c6a0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 shutdown_connections 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2de00a29f0 0x7f2de00a4eb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2e0010de60 0x7f2e0019e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e00100c60 0x7f2e0019a540 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 --2- 172.21.15.119:0/2294806770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e001002b0 0x7f2e0019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 >> 172.21.15.119:0/2294806770 conn(0x7f2e000fbef0 msgr2=0x7f2e0010f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.296+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 shutdown_connections 2024-03-28T18:51:18.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.297+0000 7f2e04e00640 1 -- 172.21.15.119:0/2294806770 wait complete. 2024-03-28T18:51:18.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:51:18.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:51:18.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 -- 172.21.15.119:0/3886571698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890109850 msgr2=0x7f5890109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 --2- 172.21.15.119:0/3886571698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890109850 0x7f5890109c30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5880009930 tx=0x7f588002bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:18.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 -- 172.21.15.119:0/3886571698 shutdown_connections 2024-03-28T18:51:18.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 --2- 172.21.15.119:0/3886571698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890104200 0x7f5890110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 --2- 172.21.15.119:0/3886571698 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890103840 0x7f5890103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 --2- 172.21.15.119:0/3886571698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890109850 0x7f5890109c30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 -- 172.21.15.119:0/3886571698 >> 172.21.15.119:0/3886571698 conn(0x7f58900ff2f0 msgr2=0x7f5890101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 -- 172.21.15.119:0/3886571698 shutdown_connections 2024-03-28T18:51:18.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.522+0000 7f5896000640 1 -- 172.21.15.119:0/3886571698 wait complete. 2024-03-28T18:51:18.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.523+0000 7f5896000640 1 Processor -- start 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.523+0000 7f5896000640 1 -- start start 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 0x7f589019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 0x7f589019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890113cb0 con 0x7f5890103840 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890113b30 con 0x7f5890109850 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f5896000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5890113e30 con 0x7f5890104200 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39250/0 (socket says 172.21.15.119:39250) 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 -- 172.21.15.119:0/768709563 learned_addr learned my addr 172.21.15.119:0/768709563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588ea00640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 0x7f589019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588fe00640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 0x7f589019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 -- 172.21.15.119:0/768709563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 msgr2=0x7f589019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 0x7f589019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 -- 172.21.15.119:0/768709563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 msgr2=0x7f589019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 0x7f589019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588f400640 1 -- 172.21.15.119:0/768709563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589019f180 con 0x7f5890103840 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588ea00640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 0x7f589019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.524+0000 7f588fe00640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 0x7f589019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f588f400640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f5887e00640 1 -- 172.21.15.119:0/768709563 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880043020 con 0x7f5890103840 2024-03-28T18:51:18.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589019f410 con 0x7f5890103840 2024-03-28T18:51:18.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58901a6d00 con 0x7f5890103840 2024-03-28T18:51:18.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f5887e00640 1 -- 172.21.15.119:0/768709563 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58800054e0 con 0x7f5890103840 2024-03-28T18:51:18.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.525+0000 7f5887e00640 1 -- 172.21.15.119:0/768709563 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5880003b80 con 0x7f5890103840 2024-03-28T18:51:18.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.528+0000 7f5887e00640 1 -- 172.21.15.119:0/768709563 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5880046020 con 0x7f5890103840 2024-03-28T18:51:18.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.528+0000 7f5898d84740 1 Processor -- start 2024-03-28T18:51:18.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 -- start start 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c50290 0x5568e1c52680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1bab250 con 0x5568e1c46f20 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1b53a30 con 0x5568e1c50290 2024-03-28T18:51:18.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f5898d84740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1ba72f0 con 0x5568e1c47c00 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f588fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f588fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:50032/0 (socket says 172.21.15.119:50032) 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f588fe00640 1 -- 172.21.15.119:0/3470616160 learned_addr learned my addr 172.21.15.119:0/3470616160 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f588ea00640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c50290 0x5568e1c52680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.529+0000 7f588f400640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 -- 172.21.15.119:0/3470616160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c50290 msgr2=0x5568e1c52680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c50290 0x5568e1c52680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 -- 172.21.15.119:0/3470616160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 msgr2=0x5568e1c473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 -- 172.21.15.119:0/3470616160 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568e1b3e690 con 0x5568e1c47c00 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588f400640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f5887e00640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 0x7f58740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588fe00640 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5888010710 tx=0x7f5888010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c101222cded3fcd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f5887e00640 1 -- 172.21.15.119:0/768709563 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f58800f0770 con 0x7f5890103840 2024-03-28T18:51:18.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f588f400640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 0x7f58740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f587be00640 1 -- 172.21.15.119:0/3470616160 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800e8d0 con 0x5568e1c47c00 2024-03-28T18:51:18.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f587be00640 1 -- 172.21.15.119:0/3470616160 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5888012450 con 0x5568e1c47c00 2024-03-28T18:51:18.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.530+0000 7f587be00640 1 -- 172.21.15.119:0/3470616160 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588801c650 con 0x5568e1c47c00 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 -- 172.21.15.119:0/3470616160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 msgr2=0x5568e1c46950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5888010710 tx=0x7f5888010bd0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f588f400640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 0x7f58740a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5568e1c48150 tx=0x7f5880009260 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 -- 172.21.15.119:0/3470616160 shutdown_connections 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c50290 0x5568e1c52680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 --2- 172.21.15.119:0/3470616160 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c47c00 0x5568e1c46950 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 -- 172.21.15.119:0/3470616160 >> 172.21.15.119:0/3470616160 conn(0x5568e1c0e200 msgr2=0x5568e1c41970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 -- 172.21.15.119:0/3470616160 shutdown_connections 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.531+0000 7f5898d84740 1 -- 172.21.15.119:0/3470616160 wait complete. 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 -- start start 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 Processor -- start 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c4f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1c53100 con 0x5568e1c46f20 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1b66e80 con 0x5568e1c47c00 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568e1c53280 con 0x5568e1c50290 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f5898d84740 10 client.0 _ll_register_callbacks cb 0x5568e1b79130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.532+0000 7f588ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c4f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52922/0 (socket says 172.21.15.119:52922) 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 -- 172.21.15.119:0/3188213617 learned_addr learned my addr 172.21.15.119:0/3188213617 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588fe00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 -- 172.21.15.119:0/3188213617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 msgr2=0x5568e1c4d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 -- 172.21.15.119:0/3188213617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 msgr2=0x5568e1c4f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c4f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 -- 172.21.15.119:0/3188213617 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568e19897d0 con 0x5568e1c47c00 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f588ea00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f587c012d20 tx=0x7f587c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8774e7727a3cd516 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f5878c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.533+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c012110 con 0x5568e1c47c00 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5888012450 con 0x5568e1c47c00 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f587c004050 con 0x5568e1c47c00 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f587c0122b0 con 0x5568e1c47c00 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5898d84740 10 client.27524 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:18.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568e18bf3f0 con 0x5568e1c47c00 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5888012450 con 0x5568e1c47c00 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.534+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f587c022030 con 0x5568e1c47c00 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 10 client.27524 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 10 client.27524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 20 client.27524 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1c601e0 0x5568e1c625d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 4 client.27524 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568e1bb8790 con 0x5568e1c601e0 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f588fe00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1c601e0 0x5568e1c625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f588fe00640 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1c601e0 0x5568e1c625d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=707c6c9baf8cb313 server_cookie=272f57a88d8e913 in_seq=0 out_seq=0 2024-03-28T18:51:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.535+0000 7f5878c00640 10 client.27524 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5878c00640 1 -- 172.21.15.119:0/3188213617 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5568e1bb8790 con 0x5568e1c601e0 2024-03-28T18:51:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5878c00640 10 client.27524 handle_command_reply: tid=0 2024-03-28T18:51:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 client.27524 shutdown 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 msgr2=0x5568e1c4cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f587c012d20 tx=0x7f587c00f790 comp rx=0 tx=0).stop 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 shutdown_connections 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1c601e0 0x5568e1c625d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4cc90 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 --2- 172.21.15.119:0/3188213617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c4f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 >> 172.21.15.119:0/3188213617 conn(0x5568e1c0e200 msgr2=0x5568e1c436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5878c00640 0 client.27524 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 shutdown_connections 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.536+0000 7f5898d84740 1 -- 172.21.15.119:0/3188213617 wait complete. 2024-03-28T18:51:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.537+0000 7f5898d84740 20 client.27524 trim_cache size 0 max 16384 2024-03-28T18:51:18.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 Processor -- start 2024-03-28T18:51:18.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 -- start start 2024-03-28T18:51:18.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587000e7a0 con 0x5568e1c46f20 2024-03-28T18:51:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.556+0000 7f5898d84740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5870010120 con 0x5568e1c47c00 2024-03-28T18:51:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f5898d84740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587000ee00 con 0x5568e1c50290 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39274/0 (socket says 172.21.15.119:39274) 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588fe00640 1 -- 172.21.15.119:0/3744833622 learned_addr learned my addr 172.21.15.119:0/3744833622 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588fe00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 -- 172.21.15.119:0/3744833622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 msgr2=0x5568e1c4f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 -- 172.21.15.119:0/3744833622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 msgr2=0x5568e1c51680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588ea00640 1 -- 172.21.15.119:0/3744833622 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568e1c62940 con 0x5568e1c47c00 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.557+0000 7f588fe00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.558+0000 7f588ea00640 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4f3c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f587c010510 tx=0x7f587c00fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12dc23d319f7a8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.558+0000 7f587b400640 1 -- 172.21.15.119:0/3744833622 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c010780 con 0x5568e1c47c00 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.558+0000 7f587b400640 1 -- 172.21.15.119:0/3744833622 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c010e00 con 0x5568e1c47c00 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.558+0000 7f587b400640 1 -- 172.21.15.119:0/3744833622 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c0047d0 con 0x5568e1c47c00 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- 172.21.15.119:0/3744833622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 msgr2=0x5568e1c4f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4f3c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f587c010510 tx=0x7f587c00fd10 comp rx=0 tx=0).stop 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- 172.21.15.119:0/3744833622 shutdown_connections 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1c4f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c47c00 0x5568e1c4f3c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 --2- 172.21.15.119:0/3744833622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c46f20 0x5568e1c51680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- 172.21.15.119:0/3744833622 >> 172.21.15.119:0/3744833622 conn(0x5568e1c0e200 msgr2=0x5568e1c414a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- 172.21.15.119:0/3744833622 shutdown_connections 2024-03-28T18:51:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- 172.21.15.119:0/3744833622 wait complete. 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.559+0000 7f5898d84740 1 -- start start 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 Processor -- start 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c46f20 0x5568e1c54210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1cb4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587000fac0 con 0x5568e1c47c00 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587000f460 con 0x5568e1c46f20 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587000c980 con 0x5568e1c50290 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f5898d84740 10 client.0 _ll_register_callbacks cb 0x7f587c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f588ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f588fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c46f20 0x5568e1c54210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f588ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39280/0 (socket says 172.21.15.119:39280) 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f588ea00640 1 -- 172.21.15.119:0/491486028 learned_addr learned my addr 172.21.15.119:0/491486028 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.560+0000 7f588fe00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1cb4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 -- 172.21.15.119:0/491486028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 msgr2=0x5568e1cb4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1cb4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 -- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c46f20 msgr2=0x5568e1c54210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c46f20 0x5568e1c54210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 -- 172.21.15.119:0/491486028 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568e187f0f0 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f588ea00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f587c012cf0 tx=0x7f587c01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f94429f1ef35310 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f5878c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c010c20 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f587c005020 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.561+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f587c010e00 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f587c005220 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 10 client.17298 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5568e1cbc200 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c010e00 con 0x5568e1c47c00 2024-03-28T18:51:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f587c03b030 con 0x5568e1c47c00 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 10 client.17298 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 20 client.17298 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1cbc7c0 0x5568e1cbebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 4 client.17298 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5568e1bb2ae0 con 0x5568e1cbc7c0 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.562+0000 7f588fe00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1cbc7c0 0x5568e1cbebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f588fe00640 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1cbc7c0 0x5568e1cbebe0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a84b786d45e727 server_cookie=2add882bc0ba19f in_seq=0 out_seq=0 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5878c00640 10 client.17298 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5878c00640 1 -- 172.21.15.119:0/491486028 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5568e1bb2ae0 con 0x5568e1cbc7c0 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5878c00640 10 client.17298 handle_command_reply: tid=0 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 client.17298 shutdown 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 msgr2=0x5568e1c54750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f587c012cf0 tx=0x7f587c01c650 comp rx=0 tx=0).stop 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 shutdown_connections 2024-03-28T18:51:18.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5568e1cbc7c0 0x5568e1cbebe0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5568e1c50290 0x5568e1cb4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5568e1c47c00 0x5568e1c54750 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5878c00640 0 client.17298 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 --2- 172.21.15.119:0/491486028 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5568e1c46f20 0x5568e1c54210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 >> 172.21.15.119:0/491486028 conn(0x5568e1c0e200 msgr2=0x5568e1c42c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 shutdown_connections 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.563+0000 7f5898d84740 1 -- 172.21.15.119:0/491486028 wait complete. 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5898d84740 20 client.17298 trim_cache size 0 max 16384 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:51:18.568 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:51:18.569 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 msgr2=0x7f58740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 0x7f58740a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5568e1c48150 tx=0x7f5880009260 comp rx=0 tx=0).stop 2024-03-28T18:51:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 msgr2=0x7f589019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880004360 comp rx=0 tx=0).stop 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 shutdown_connections 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f58740a2970 0x7f58740a4e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5890109850 0x7f589019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5890104200 0x7f589019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 --2- 172.21.15.119:0/768709563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5890103840 0x7f589019a190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 >> 172.21.15.119:0/768709563 conn(0x7f58900ff2f0 msgr2=0x7f58900ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.564+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 shutdown_connections 2024-03-28T18:51:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.565+0000 7f5896000640 1 -- 172.21.15.119:0/768709563 wait complete. 2024-03-28T18:51:18.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:51:18.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: pgmap v222: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.660 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:18 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 -- 172.21.15.119:0/3317307738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 msgr2=0x7f510c114930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 --2- 172.21.15.119:0/3317307738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c114930 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5104009cd0 tx=0x7f510402c0e0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 -- 172.21.15.119:0/3317307738 shutdown_connections 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 --2- 172.21.15.119:0/3317307738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c114930 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 --2- 172.21.15.119:0/3317307738 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 0x7f510c107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 --2- 172.21.15.119:0/3317307738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 -- 172.21.15.119:0/3317307738 >> 172.21.15.119:0/3317307738 conn(0x7f510c0771c0 msgr2=0x7f510c0795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:18.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 -- 172.21.15.119:0/3317307738 shutdown_connections 2024-03-28T18:51:18.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.792+0000 7f5110e00640 1 -- 172.21.15.119:0/3317307738 wait complete. 2024-03-28T18:51:18.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.793+0000 7f5110e00640 1 Processor -- start 2024-03-28T18:51:18.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 -- start start 2024-03-28T18:51:18.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 0x7f510c19e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c1a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c117e80 con 0x7f510c10da70 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c117d00 con 0x7f510c107a60 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f5110e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c118000 con 0x7f510c108420 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f510aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f510aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:50102/0 (socket says 172.21.15.119:50102) 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f510aa00640 1 -- 172.21.15.119:0/491265858 learned_addr learned my addr 172.21.15.119:0/491265858 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.794+0000 7f510b400640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 0x7f510c19e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510be00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c1a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 -- 172.21.15.119:0/491265858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 msgr2=0x7f510c19e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 0x7f510c19e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 -- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 msgr2=0x7f510c1a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c1a2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 -- 172.21.15.119:0/491265858 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c1a32b0 con 0x7f510c108420 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510be00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c1a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f510aa00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f50f400e9f0 tx=0x7f50f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f400cd80 con 0x7f510c108420 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50f4010910 con 0x7f510c108420 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f4018990 con 0x7f510c108420 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.795+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c1a35a0 con 0x7f510c108420 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.796+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c1aae30 con 0x7f510c108420 2024-03-28T18:51:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.797+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c068800 con 0x7f510c108420 2024-03-28T18:51:18.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.799+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f50f4010ac0 con 0x7f510c108420 2024-03-28T18:51:18.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.800+0000 7f5103e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 0x7f50f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:18.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.800+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f50f40cf480 con 0x7f510c108420 2024-03-28T18:51:18.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.800+0000 7f510b400640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 0x7f50f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:18.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.801+0000 7f510b400640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 0x7f50f00a4e30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009b50 tx=0x7f50fc005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:18.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:18.807+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f50f409c150 con 0x7f510c108420 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: pgmap v222: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:18 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:51:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: pgmap v222: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:51:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:51:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:51:18.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:18.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:18 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:51:19.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.046+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f510c19f610 con 0x7f510c108420 2024-03-28T18:51:19.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.048+0000 7f5103e00640 1 -- 172.21.15.119:0/491265858 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f510c19f610 con 0x7f510c108420 2024-03-28T18:51:19.049 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:51:19.049 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:51:19.050 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:51:19.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 msgr2=0x7f50f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 0x7f50f00a4e30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009b50 tx=0x7f50fc005f90 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 msgr2=0x7f510c19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f50f400e9f0 tx=0x7f50f400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 shutdown_connections 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50f00a2970 0x7f50f00a4e30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f510c10da70 0x7f510c1a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f510c108420 0x7f510c19e800 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 --2- 172.21.15.119:0/491265858 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f510c107a60 0x7f510c19e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.050+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 >> 172.21.15.119:0/491265858 conn(0x7f510c0771c0 msgr2=0x7f510c077850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.051+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 shutdown_connections 2024-03-28T18:51:19.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.051+0000 7f5110e00640 1 -- 172.21.15.119:0/491265858 wait complete. 2024-03-28T18:51:19.073 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:19.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 -- 172.21.15.119:0/2963637818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6381075f0 msgr2=0x7fc6381079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 --2- 172.21.15.119:0/2963637818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6381075f0 0x7fc6381079d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc628009930 tx=0x7fc62802bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 -- 172.21.15.119:0/2963637818 shutdown_connections 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 --2- 172.21.15.119:0/2963637818 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101ef0 0x7fc63810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 --2- 172.21.15.119:0/2963637818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc638101530 0x7fc6381019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 --2- 172.21.15.119:0/2963637818 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6381075f0 0x7fc6381079d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 -- 172.21.15.119:0/2963637818 >> 172.21.15.119:0/2963637818 conn(0x7fc6380fd000 msgr2=0x7fc6380ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.284+0000 7fc63d800640 1 -- 172.21.15.119:0/2963637818 shutdown_connections 2024-03-28T18:51:19.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.285+0000 7fc63d800640 1 -- 172.21.15.119:0/2963637818 wait complete. 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 Processor -- start 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 -- start start 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101530 0x7fc638199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 0x7fc63819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381118b0 con 0x7fc6381075f0 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc638111730 con 0x7fc638101530 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.286+0000 7fc63d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc638111a30 con 0x7fc638101ef0 2024-03-28T18:51:19.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc637e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101530 0x7fc638199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37190/0 (socket says 172.21.15.119:37190) 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 -- 172.21.15.119:0/794977309 learned_addr learned my addr 172.21.15.119:0/794977309 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc63ce00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 0x7fc63819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 -- 172.21.15.119:0/794977309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101530 msgr2=0x7fc638199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101530 0x7fc638199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 -- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 msgr2=0x7fc63819e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 0x7fc63819e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 -- 172.21.15.119:0/794977309 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63819efe0 con 0x7fc638101ef0 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc63ce00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 0x7fc63819e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:19.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc62fe00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.287+0000 7fc635600640 1 -- 172.21.15.119:0/794977309 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62000cd80 con 0x7fc638101ef0 2024-03-28T18:51:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.288+0000 7fc635600640 1 -- 172.21.15.119:0/794977309 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc620010910 con 0x7fc638101ef0 2024-03-28T18:51:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.288+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63819f2d0 con 0x7fc638101ef0 2024-03-28T18:51:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.288+0000 7fc635600640 1 -- 172.21.15.119:0/794977309 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620018990 con 0x7fc638101ef0 2024-03-28T18:51:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.288+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6381a6b60 con 0x7fc638101ef0 2024-03-28T18:51:19.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 Processor -- start 2024-03-28T18:51:19.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 -- start start 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d02810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d05230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965c5d2c0 con 0x562965cfa260 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965c05c90 con 0x562965d02de0 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc641b81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965c59360 con 0x562965cf98b0 2024-03-28T18:51:19.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc635600640 1 -- 172.21.15.119:0/794977309 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc620010ac0 con 0x7fc638101ef0 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.291+0000 7fc63ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37200/0 (socket says 172.21.15.119:37200) 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 -- 172.21.15.119:0/3619703381 learned_addr learned my addr 172.21.15.119:0/3619703381 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc637e00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d02810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc62fe00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d05230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 -- 172.21.15.119:0/3619703381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 msgr2=0x562965d05230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d05230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 -- 172.21.15.119:0/3619703381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 msgr2=0x562965d02810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d02810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 -- 172.21.15.119:0/3619703381 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562965c2a080 con 0x562965cf98b0 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc637e00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d02810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.292+0000 7fc63ce00640 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc630010450 tx=0x7fc630010910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23fcd9044a246f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc62d600640 1 -- 172.21.15.119:0/3619703381 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630017070 con 0x562965cf98b0 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc62d600640 1 -- 172.21.15.119:0/3619703381 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc630011c40 con 0x562965cf98b0 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc62d600640 1 -- 172.21.15.119:0/3619703381 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63001adb0 con 0x562965cf98b0 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc635600640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 0x7fc61c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc641b81740 1 -- 172.21.15.119:0/3619703381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 msgr2=0x562965cf9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.293+0000 7fc641b81740 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc630010450 tx=0x7fc630010910 comp rx=0 tx=0).stop 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 -- 172.21.15.119:0/3619703381 shutdown_connections 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d05230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d02810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 --2- 172.21.15.119:0/3619703381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf9c90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 -- 172.21.15.119:0/3619703381 >> 172.21.15.119:0/3619703381 conn(0x562965cf3f80 msgr2=0x562965cf4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc635600640 1 -- 172.21.15.119:0/794977309 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc6200cf340 con 0x7fc638101ef0 2024-03-28T18:51:19.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc637e00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 0x7fc61c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 -- 172.21.15.119:0/3619703381 shutdown_connections 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 -- 172.21.15.119:0/3619703381 wait complete. 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.294+0000 7fc641b81740 1 -- start start 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 Processor -- start 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc637e00640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 0x7fc61c0a4e30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc628005ab0 tx=0x7fc628005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cf98b0 0x562965d02720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d000c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965d05d50 con 0x562965cf98b0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965c190d0 con 0x562965d02de0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562965d05ed0 con 0x562965cfa260 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc641b81740 10 client.0 _ll_register_callbacks cb 0x562965c1de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37210/0 (socket says 172.21.15.119:37210) 2024-03-28T18:51:19.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc62fe00640 1 -- 172.21.15.119:0/2494772257 learned_addr learned my addr 172.21.15.119:0/2494772257 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc63ce00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.295+0000 7fc637e00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cf98b0 0x562965d02720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 -- 172.21.15.119:0/2494772257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 msgr2=0x562965d000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 -- 172.21.15.119:0/2494772257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cf98b0 msgr2=0x562965d02720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cf98b0 0x562965d02720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 -- 172.21.15.119:0/2494772257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629659713f0 con 0x562965cfa260 2024-03-28T18:51:19.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc62fe00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc6200a3b20 tx=0x7fc62009b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6c82fff26790eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc626000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:19.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62009b8b0 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc6200ccc20 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.296+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc630011c40 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6200cce20 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 10 client.27520 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562965a3b7d0 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc630011c40 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc620018b30 con 0x562965cfa260 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 10 client.27520 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 10 client.27520 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 20 client.27520 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d12c80 0x562965d15070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 4 client.27520 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562965c6a800 con 0x562965d12c80 2024-03-28T18:51:19.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.297+0000 7fc637e00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d12c80 0x562965d15070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.298+0000 7fc626000640 10 client.27520 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.298+0000 7fc637e00640 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d12c80 0x562965d15070 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86f2b98c371e9c6f server_cookie=4b46194421326d77 in_seq=0 out_seq=0 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.298+0000 7fc626000640 1 -- 172.21.15.119:0/2494772257 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562965c6a800 con 0x562965d12c80 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.298+0000 7fc626000640 10 client.27520 handle_command_reply: tid=0 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 client.27520 shutdown 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 msgr2=0x562965cffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc6200a3b20 tx=0x7fc62009b770 comp rx=0 tx=0).stop 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 shutdown_connections 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d12c80 0x562965d15070 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965d02de0 0x562965d000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc626000640 0 client.27520 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cfa260 0x562965cffab0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 --2- 172.21.15.119:0/2494772257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cf98b0 0x562965d02720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 >> 172.21.15.119:0/2494772257 conn(0x562965cf3f80 msgr2=0x562965cf5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 shutdown_connections 2024-03-28T18:51:19.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 1 -- 172.21.15.119:0/2494772257 wait complete. 2024-03-28T18:51:19.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.299+0000 7fc641b81740 20 client.27520 trim_cache size 0 max 16384 2024-03-28T18:51:19.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 Processor -- start 2024-03-28T18:51:19.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 -- start start 2024-03-28T18:51:19.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cfa260 0x562965d021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965d02de0 0x562965d02720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381af480 con 0x562965d02de0 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381b0e00 con 0x562965cfa260 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc641b81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381afae0 con 0x562965cf98b0 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc637e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc63ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965d02de0 0x562965d02720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cfa260 0x562965d021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.313+0000 7fc637e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37222/0 (socket says 172.21.15.119:37222) 2024-03-28T18:51:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 -- 172.21.15.119:0/1116870404 learned_addr learned my addr 172.21.15.119:0/1116870404 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 -- 172.21.15.119:0/1116870404 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cfa260 msgr2=0x562965d021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cfa260 0x562965d021e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 -- 172.21.15.119:0/1116870404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965d02de0 msgr2=0x562965d02720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965d02de0 0x562965d02720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 -- 172.21.15.119:0/1116870404 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562965cee570 con 0x562965cf98b0 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc637e00640 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc628036040 tx=0x7fc628040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b5435553da2eb5f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc62cc00640 1 -- 172.21.15.119:0/1116870404 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280048c0 con 0x562965cf98b0 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc62cc00640 1 -- 172.21.15.119:0/1116870404 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62805a070 con 0x562965cf98b0 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.314+0000 7fc62cc00640 1 -- 172.21.15.119:0/1116870404 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc628055600 con 0x562965cf98b0 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 -- 172.21.15.119:0/1116870404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 msgr2=0x562965cf2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc628036040 tx=0x7fc628040000 comp rx=0 tx=0).stop 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 -- 172.21.15.119:0/1116870404 shutdown_connections 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965d02de0 0x562965d02720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cfa260 0x562965d021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 --2- 172.21.15.119:0/1116870404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965cf98b0 0x562965cf2f40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 -- 172.21.15.119:0/1116870404 >> 172.21.15.119:0/1116870404 conn(0x562965cf3f80 msgr2=0x562965cf5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 -- 172.21.15.119:0/1116870404 shutdown_connections 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.315+0000 7fc641b81740 1 -- 172.21.15.119:0/1116870404 wait complete. 2024-03-28T18:51:19.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 -- start start 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 Processor -- start 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d08f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 0x562965d09480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381b07a0 con 0x562965cfa260 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381b0140 con 0x562965cf98b0 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381ad660 con 0x562965d02de0 2024-03-28T18:51:19.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc641b81740 10 client.0 _ll_register_callbacks cb 0x7fc620009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.316+0000 7fc637e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53020/0 (socket says 172.21.15.119:53020) 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 -- 172.21.15.119:0/328260884 learned_addr learned my addr 172.21.15.119:0/328260884 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc62fe00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d08f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc63ce00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 0x562965d09480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 -- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 msgr2=0x562965d09480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 0x562965d09480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 -- 172.21.15.119:0/328260884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 msgr2=0x562965d08f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d08f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 -- 172.21.15.119:0/328260884 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562965d099c0 con 0x562965cf98b0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc63ce00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 0x562965d09480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc637e00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc628000c00 tx=0x7fc628005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41143370262a77ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc626000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:19.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.317+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc628004c90 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc62805a070 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc628004f50 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc628005150 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 10 client.27554 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562965d6f150 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62805a070 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc6280035f0 con 0x562965cf98b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 10 client.27554 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 10 client.27554 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 20 client.27554 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d6f7b0 0x562965d71bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 4 client.27554 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.318+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562965c64b50 con 0x562965d6f7b0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc62fe00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d6f7b0 0x562965d71bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc62fe00640 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d6f7b0 0x562965d71bd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=21ef936be7b145ae server_cookie=31510b27ba2325cd in_seq=0 out_seq=0 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc626000640 10 client.27554 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc626000640 1 -- 172.21.15.119:0/328260884 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562965c64b50 con 0x562965d6f7b0 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc626000640 10 client.27554 handle_command_reply: tid=0 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 client.27554 shutdown 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 msgr2=0x562965d0aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc628000c00 tx=0x7fc628005dc0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 shutdown_connections 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x562965d6f7b0 0x562965d71bd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562965d02de0 0x562965d09480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562965cfa260 0x562965d08f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 --2- 172.21.15.119:0/328260884 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562965cf98b0 0x562965d0aba0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 >> 172.21.15.119:0/328260884 conn(0x562965cf3f80 msgr2=0x562965cf5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.319+0000 7fc626000640 0 client.27554 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 shutdown_connections 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc641b81740 1 -- 172.21.15.119:0/328260884 wait complete. 2024-03-28T18:51:19.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc641b81740 20 client.27554 trim_cache size 0 max 16384 2024-03-28T18:51:19.325 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:19.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 msgr2=0x7fc61c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 0x7fc61c0a4e30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc628005ab0 tx=0x7fc628005fd0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 msgr2=0x7fc63819a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 shutdown_connections 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc61c0a2970 0x7fc61c0a4e30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6381075f0 0x7fc63819e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc638101ef0 0x7fc63819a530 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 --2- 172.21.15.119:0/794977309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc638101530 0x7fc638199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.320+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 >> 172.21.15.119:0/794977309 conn(0x7fc6380fd000 msgr2=0x7fc6380fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.321+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 shutdown_connections 2024-03-28T18:51:19.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.321+0000 7fc63d800640 1 -- 172.21.15.119:0/794977309 wait complete. 2024-03-28T18:51:19.339 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:19.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 -- 172.21.15.119:0/4181545569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc23810cf80 msgr2=0x7fc238114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 --2- 172.21.15.119:0/4181545569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc23810cf80 0x7fc238114b50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc230009ca0 tx=0x7fc23002c0f0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 -- 172.21.15.119:0/4181545569 shutdown_connections 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 --2- 172.21.15.119:0/4181545569 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc23810cf80 0x7fc238114b50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 --2- 172.21.15.119:0/4181545569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc238108860 0x7fc23810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 --2- 172.21.15.119:0/4181545569 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc238107eb0 0x7fc238108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 -- 172.21.15.119:0/4181545569 >> 172.21.15.119:0/4181545569 conn(0x7fc238075be0 msgr2=0x7fc238078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.538+0000 7fc23d200640 1 -- 172.21.15.119:0/4181545569 shutdown_connections 2024-03-28T18:51:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.539+0000 7fc23d200640 1 -- 172.21.15.119:0/4181545569 wait complete. 2024-03-28T18:51:19.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.539+0000 7fc23d200640 1 Processor -- start 2024-03-28T18:51:19.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 -- start start 2024-03-28T18:51:19.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc23810cf80 0x7fc23819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2381a2e00 0x7fc2381a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238118030 con 0x7fc23810cf80 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238117eb0 con 0x7fc238107eb0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc23d200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381181b0 con 0x7fc2381a2e00 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc235e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc235e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53036/0 (socket says 172.21.15.119:53036) 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc235e00640 1 -- 172.21.15.119:0/3968063153 learned_addr learned my addr 172.21.15.119:0/3968063153 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc235400640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc23810cf80 0x7fc23819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.540+0000 7fc236800640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2381a2e00 0x7fc2381a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 -- 172.21.15.119:0/3968063153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2381a2e00 msgr2=0x7fc2381a32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2381a2e00 0x7fc2381a32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 -- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc23810cf80 msgr2=0x7fc23819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc23810cf80 0x7fc23819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 -- 172.21.15.119:0/3968063153 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2381a38f0 con 0x7fc238107eb0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc235e00640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc22800e9f0 tx=0x7fc22800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc22ea00640 1 -- 172.21.15.119:0/3968063153 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22800cd80 con 0x7fc238107eb0 2024-03-28T18:51:19.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc22ea00640 1 -- 172.21.15.119:0/3968063153 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc228010910 con 0x7fc238107eb0 2024-03-28T18:51:19.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc22ea00640 1 -- 172.21.15.119:0/3968063153 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc228018990 con 0x7fc238107eb0 2024-03-28T18:51:19.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2381a3be0 con 0x7fc238107eb0 2024-03-28T18:51:19.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.541+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381ab3d0 con 0x7fc238107eb0 2024-03-28T18:51:19.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.544+0000 7fc22ea00640 1 -- 172.21.15.119:0/3968063153 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc228018b30 con 0x7fc238107eb0 2024-03-28T18:51:19.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.545+0000 7fc23ff77740 1 Processor -- start 2024-03-28T18:51:19.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.545+0000 7fc23ff77740 1 -- start start 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f28ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 0x5634e5f31630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f34050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5e8c2c0 con 0x5634e5f286d0 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5e34c90 con 0x5634e5f29080 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc23ff77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5e88360 con 0x5634e5f31c00 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc236800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f28ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f34050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc236800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f28ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39960/0 (socket says 172.21.15.119:39960) 2024-03-28T18:51:19.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc236800640 1 -- 172.21.15.119:0/2563549242 learned_addr learned my addr 172.21.15.119:0/2563549242 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235e00640 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 0x5634e5f31630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc22ea00640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 0x7fc21c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 -- 172.21.15.119:0/2563549242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 msgr2=0x5634e5f31630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 0x5634e5f31630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 -- 172.21.15.119:0/2563549242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 msgr2=0x5634e5f28ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f28ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235400640 1 -- 172.21.15.119:0/2563549242 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634e5e59080 con 0x5634e5f31c00 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.546+0000 7fc235e00640 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 0x5634e5f31630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc22ea00640 1 -- 172.21.15.119:0/3968063153 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc2280cf1d0 con 0x7fc238107eb0 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc236800640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 0x7fc21c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc235400640 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f34050 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc224005d00 tx=0x7fc2240107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a047423fcb825673 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc223400640 1 -- 172.21.15.119:0/2563549242 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22401c480 con 0x5634e5f31c00 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc223400640 1 -- 172.21.15.119:0/2563549242 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc22401cb00 con 0x5634e5f31c00 2024-03-28T18:51:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.547+0000 7fc223400640 1 -- 172.21.15.119:0/2563549242 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224011af0 con 0x5634e5f31c00 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 -- 172.21.15.119:0/2563549242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 msgr2=0x5634e5f34050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f34050 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc224005d00 tx=0x7fc2240107a0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc236800640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 0x7fc21c0a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc230009a60 tx=0x7fc23002e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 -- 172.21.15.119:0/2563549242 shutdown_connections 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f34050 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f29080 0x5634e5f31630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 --2- 172.21.15.119:0/2563549242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f28ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 -- 172.21.15.119:0/2563549242 >> 172.21.15.119:0/2563549242 conn(0x5634e5f22da0 msgr2=0x5634e5f23180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 -- 172.21.15.119:0/2563549242 shutdown_connections 2024-03-28T18:51:19.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.548+0000 7fc23ff77740 1 -- 172.21.15.119:0/2563549242 wait complete. 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 -- start start 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 Processor -- start 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f2e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5f34b70 con 0x5634e5f29080 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5e480d0 con 0x5634e5f286d0 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634e5f34cf0 con 0x5634e5f31c00 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc23ff77740 10 client.0 _ll_register_callbacks cb 0x5634e5e4ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc236800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc235e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.549+0000 7fc235400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f2e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53052/0 (socket says 172.21.15.119:53052) 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc236800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37260/0 (socket says 172.21.15.119:37260) 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 -- 172.21.15.119:0/4025582769 learned_addr learned my addr 172.21.15.119:0/4025582769 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 -- 172.21.15.119:0/4025582769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 msgr2=0x5634e5f2edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 -- 172.21.15.119:0/4025582769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 msgr2=0x5634e5f2e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f2e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 -- 172.21.15.119:0/4025582769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634e5ba03f0 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc236800640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc235e00640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc22800e9c0 tx=0x7fc22809b3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1575658946e821c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc217e00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22809b890 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc228010ac0 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.550+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc22401cb00 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc22809f930 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 10 client.27566 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5634e5c6a7d0 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc22401cb00 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc2280d2bb0 con 0x5634e5f286d0 2024-03-28T18:51:19.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 10 client.27566 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 10 client.27566 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 20 client.27566 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f41990 0x5634e5f43d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 4 client.27566 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634e5e99800 con 0x5634e5f41990 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.551+0000 7fc235400640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f41990 0x5634e5f43d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.552+0000 7fc235400640 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f41990 0x5634e5f43d80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e8384af76ea61261 server_cookie=634ed3b80b22094c in_seq=0 out_seq=0 2024-03-28T18:51:19.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.552+0000 7fc217e00640 10 client.27566 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc217e00640 1 -- 172.21.15.119:0/4025582769 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634e5e99800 con 0x5634e5f41990 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc217e00640 10 client.27566 handle_command_reply: tid=0 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 client.27566 shutdown 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 msgr2=0x5634e5f314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc22800e9c0 tx=0x7fc22809b3d0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 shutdown_connections 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f31c00 0x5634e5f2edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f41990 0x5634e5f43d80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f2e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 --2- 172.21.15.119:0/4025582769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f286d0 0x5634e5f314e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc217e00640 0 client.27566 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:19.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 >> 172.21.15.119:0/4025582769 conn(0x5634e5f22da0 msgr2=0x5634e5f24af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 shutdown_connections 2024-03-28T18:51:19.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 1 -- 172.21.15.119:0/4025582769 wait complete. 2024-03-28T18:51:19.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.553+0000 7fc23ff77740 20 client.27566 trim_cache size 0 max 16384 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 Processor -- start 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 -- start start 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f24890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f314e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b3a60 con 0x5634e5f286d0 2024-03-28T18:51:19.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b53e0 con 0x5634e5f31c00 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc23ff77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b40c0 con 0x5634e5f29080 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc235400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.567+0000 7fc235e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f24890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37266/0 (socket says 172.21.15.119:37266) 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 -- 172.21.15.119:0/1367873638 learned_addr learned my addr 172.21.15.119:0/1367873638 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc236800640 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 -- 172.21.15.119:0/1367873638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 msgr2=0x5634e5f314e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f314e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 -- 172.21.15.119:0/1367873638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 msgr2=0x5634e5f24890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f24890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 -- 172.21.15.119:0/1367873638 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634e5f2f140 con 0x5634e5f29080 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235e00640 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f24890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc235400640 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc224000c00 tx=0x7fc22400b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=116e9ca924241762 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc222a00640 1 -- 172.21.15.119:0/1367873638 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22402e020 con 0x5634e5f29080 2024-03-28T18:51:19.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc222a00640 1 -- 172.21.15.119:0/1367873638 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc22402a0d0 con 0x5634e5f29080 2024-03-28T18:51:19.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.568+0000 7fc222a00640 1 -- 172.21.15.119:0/1367873638 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2240054a0 con 0x5634e5f29080 2024-03-28T18:51:19.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- 172.21.15.119:0/1367873638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 msgr2=0x5634e5f30fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc224000c00 tx=0x7fc22400b670 comp rx=0 tx=0).stop 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- 172.21.15.119:0/1367873638 shutdown_connections 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f314e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f29080 0x5634e5f30fa0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 --2- 172.21.15.119:0/1367873638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f286d0 0x5634e5f24890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- 172.21.15.119:0/1367873638 >> 172.21.15.119:0/1367873638 conn(0x5634e5f22da0 msgr2=0x5634e5f24180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- 172.21.15.119:0/1367873638 shutdown_connections 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- 172.21.15.119:0/1367873638 wait complete. 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.569+0000 7fc23ff77740 1 -- start start 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 Processor -- start 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f35a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f35f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b4d80 con 0x5634e5f29080 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b4720 con 0x5634e5f31c00 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381b1c40 con 0x5634e5f286d0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc23ff77740 10 client.0 _ll_register_callbacks cb 0x7fc22800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc235e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc235400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f35a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc235e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37282/0 (socket says 172.21.15.119:37282) 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc236800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f35f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.570+0000 7fc235e00640 1 -- 172.21.15.119:0/3021571836 learned_addr learned my addr 172.21.15.119:0/3021571836 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 -- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 msgr2=0x5634e5f35f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f35f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 -- 172.21.15.119:0/3021571836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 msgr2=0x5634e5f35a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f35a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 -- 172.21.15.119:0/3021571836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634e5f9c270 con 0x5634e5f286d0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc236800640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f35f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc235e00640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc22809b590 tx=0x7fc228010cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a2a9b6b5de38f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc217e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2280d2d30 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc2280afbc0 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc22402a0d0 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc22809b890 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc23ff77740 10 client.27544 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5634e5f9d570 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.571+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc22402a0d0 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc22809ace0 con 0x5634e5f286d0 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 10 client.27544 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 10 client.27544 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 20 client.27544 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f9db60 0x5634e5f9ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 4 client.27544 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5634e5e93b50 con 0x5634e5f9db60 2024-03-28T18:51:19.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc235400640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f9db60 0x5634e5f9ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc235400640 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f9db60 0x5634e5f9ff80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4fb6682ce20a7c5f server_cookie=396de6ae56d0b826 in_seq=0 out_seq=0 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.572+0000 7fc217e00640 10 client.27544 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc217e00640 1 -- 172.21.15.119:0/3021571836 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5634e5e93b50 con 0x5634e5f9db60 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc217e00640 10 client.27544 handle_command_reply: tid=0 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 client.27544 shutdown 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 msgr2=0x5634e5f35500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc22809b590 tx=0x7fc228010cf0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 shutdown_connections 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5634e5f31c00 0x5634e5f35f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x5634e5f9db60 0x5634e5f9ff80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634e5f29080 0x5634e5f35a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc217e00640 0 client.27544 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 --2- 172.21.15.119:0/3021571836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5634e5f286d0 0x5634e5f35500 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 >> 172.21.15.119:0/3021571836 conn(0x5634e5f22da0 msgr2=0x5634e5f21eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 shutdown_connections 2024-03-28T18:51:19.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 1 -- 172.21.15.119:0/3021571836 wait complete. 2024-03-28T18:51:19.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.573+0000 7fc23ff77740 20 client.27544 trim_cache size 0 max 16384 2024-03-28T18:51:19.578 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 msgr2=0x7fc21c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 0x7fc21c0a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc230009a60 tx=0x7fc23002e040 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 msgr2=0x7fc23819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc22800e9f0 tx=0x7fc22800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 shutdown_connections 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc21c0a2970 0x7fc21c0a4e30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc2381a2e00 0x7fc2381a32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc23810cf80 0x7fc23819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 --2- 172.21.15.119:0/3968063153 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc238107eb0 0x7fc23819e590 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 >> 172.21.15.119:0/3968063153 conn(0x7fc238075be0 msgr2=0x7fc2380767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 shutdown_connections 2024-03-28T18:51:19.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.574+0000 7fc23d200640 1 -- 172.21.15.119:0/3968063153 wait complete. 2024-03-28T18:51:19.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:19 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/491265858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:19.592 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:19.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.796+0000 7fabeda00640 1 -- 172.21.15.119:0/3362428233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 msgr2=0x7fabe8110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.796+0000 7fabeda00640 1 --2- 172.21.15.119:0/3362428233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe8110700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009cd0 tx=0x7fabe002c0e0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 -- 172.21.15.119:0/3362428233 shutdown_connections 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 --2- 172.21.15.119:0/3362428233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe8110700 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 --2- 172.21.15.119:0/3362428233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8103830 0x7fabe8103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 --2- 172.21.15.119:0/3362428233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8109840 0x7fabe8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 -- 172.21.15.119:0/3362428233 >> 172.21.15.119:0/3362428233 conn(0x7fabe80ff320 msgr2=0x7fabe8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 -- 172.21.15.119:0/3362428233 shutdown_connections 2024-03-28T18:51:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.797+0000 7fabeda00640 1 -- 172.21.15.119:0/3362428233 wait complete. 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 Processor -- start 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 -- start start 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8103830 0x7fabe819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 0x7fabe819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8113c10 con 0x7fabe8103830 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8113a90 con 0x7fabe8109840 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.798+0000 7fabeda00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8113d90 con 0x7fabe81041f0 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37306/0 (socket says 172.21.15.119:37306) 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 0x7fabe819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 -- 172.21.15.119:0/186455890 learned_addr learned my addr 172.21.15.119:0/186455890 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6a00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8103830 0x7fabe819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 -- 172.21.15.119:0/186455890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 msgr2=0x7fabe819e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 0x7fabe819e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 -- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8103830 msgr2=0x7fabe819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8103830 0x7fabe819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 -- 172.21.15.119:0/186455890 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe819f100 con 0x7fabe81041f0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe7400640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 0x7fabe819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabe6000640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fabd400d950 tx=0x7fabd400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.799+0000 7fabdf400640 1 -- 172.21.15.119:0/186455890 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd4017030 con 0x7fabe81041f0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.800+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe819f3f0 con 0x7fabe81041f0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.800+0000 7fabdf400640 1 -- 172.21.15.119:0/186455890 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd4004020 con 0x7fabe81041f0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.800+0000 7fabdf400640 1 -- 172.21.15.119:0/186455890 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd40047a0 con 0x7fabe81041f0 2024-03-28T18:51:19.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.800+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe81a6c80 con 0x7fabe81041f0 2024-03-28T18:51:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.804+0000 7fabf064f740 1 Processor -- start 2024-03-28T18:51:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.804+0000 7fabf064f740 1 -- start start 2024-03-28T18:51:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.804+0000 7fabf064f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.804+0000 7fabf064f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 0x55f4cf700630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabf064f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf700c00 0x55f4cf703050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabf064f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf65b2c0 con 0x55f4cf6f76d0 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabf064f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf603c90 con 0x55f4cf700c00 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabf064f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf657360 con 0x55f4cf6f8080 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40014/0 (socket says 172.21.15.119:40014) 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 0x55f4cf700630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe7400640 1 -- 172.21.15.119:0/2066006471 learned_addr learned my addr 172.21.15.119:0/2066006471 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabdf400640 1 -- 172.21.15.119:0/186455890 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fabd40041d0 con 0x7fabe81041f0 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 -- 172.21.15.119:0/2066006471 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf700c00 msgr2=0x55f4cf703050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf700c00 0x55f4cf703050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 -- 172.21.15.119:0/2066006471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 msgr2=0x55f4cf6f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe6a00640 1 -- 172.21.15.119:0/2066006471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4cf628080 con 0x55f4cf6f8080 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.805+0000 7fabe7400640 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabe6a00640 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 0x55f4cf700630 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009c20 tx=0x7fabd8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f86128bd846a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabd3e00640 1 -- 172.21.15.119:0/2066006471 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8004050 con 0x55f4cf6f8080 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabd3e00640 1 -- 172.21.15.119:0/2066006471 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd8005500 con 0x55f4cf6f8080 2024-03-28T18:51:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabd3e00640 1 -- 172.21.15.119:0/2066006471 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd8004d50 con 0x55f4cf6f8080 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 -- 172.21.15.119:0/2066006471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 msgr2=0x55f4cf700630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 0x55f4cf700630 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fabd8009c20 tx=0x7fabd8011810 comp rx=0 tx=0).stop 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 -- 172.21.15.119:0/2066006471 shutdown_connections 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf700c00 0x55f4cf703050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f8080 0x55f4cf700630 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 --2- 172.21.15.119:0/2066006471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.806+0000 7fabf064f740 1 -- 172.21.15.119:0/2066006471 >> 172.21.15.119:0/2066006471 conn(0x55f4cf6f1da0 msgr2=0x55f4cf6f2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabdf400640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 0x7fabcc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabf064f740 1 -- 172.21.15.119:0/2066006471 shutdown_connections 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabdf400640 1 -- 172.21.15.119:0/186455890 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fabd40ce890 con 0x7fabe81041f0 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabe7400640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 0x7fabcc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabf064f740 1 -- 172.21.15.119:0/2066006471 wait complete. 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.807+0000 7fabf064f740 1 -- start start 2024-03-28T18:51:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabe7400640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 0x7fabcc0a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fabe0037040 tx=0x7fabe002e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 Processor -- start 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf700520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf6fde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf703b70 con 0x55f4cf6f76d0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf6170d0 con 0x55f4cf6f8080 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4cf703cf0 con 0x55f4cf700c00 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabf064f740 10 client.0 _ll_register_callbacks cb 0x55f4cf61be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf700520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabe6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabe7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf6fde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.808+0000 7fabe6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53122/0 (socket says 172.21.15.119:53122) 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 -- 172.21.15.119:0/2668403411 learned_addr learned my addr 172.21.15.119:0/2668403411 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 -- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 msgr2=0x55f4cf6fde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf6fde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 -- 172.21.15.119:0/2668403411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 msgr2=0x55f4cf700520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf700520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 -- 172.21.15.119:0/2668403411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4cf36f3f0 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabe6000640 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fabd40cbfb0 tx=0x7fabd409a880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ec7571e0750b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd409efb0 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fabd40a3170 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.809+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabd8005500 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabd40b8070 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 10 client.27581 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f4cf4397d0 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd8005500 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fabd4099d20 con 0x55f4cf6f8080 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 10 client.27581 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 10 client.27581 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 20 client.27581 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf7109d0 0x55f4cf712dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 4 client.27581 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:19.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4cf668800 con 0x55f4cf7109d0 2024-03-28T18:51:19.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.810+0000 7fabe6a00640 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf7109d0 0x55f4cf712dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.811+0000 7fabe6a00640 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf7109d0 0x55f4cf712dc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3092581c6d64c2dc server_cookie=10db5a719eb08c40 in_seq=0 out_seq=0 2024-03-28T18:51:19.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.811+0000 7fabd0c00640 10 client.27581 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabd0c00640 1 -- 172.21.15.119:0/2668403411 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f4cf668800 con 0x55f4cf7109d0 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabd0c00640 10 client.27581 handle_command_reply: tid=0 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 client.27581 shutdown 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 msgr2=0x55f4cf6fd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fabd40cbfb0 tx=0x7fabd409a880 comp rx=0 tx=0).stop 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 shutdown_connections 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf7109d0 0x55f4cf712dc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf6fde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fd8b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabd0c00640 0 client.27581 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 --2- 172.21.15.119:0/2668403411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f76d0 0x55f4cf700520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 >> 172.21.15.119:0/2668403411 conn(0x55f4cf6f1da0 msgr2=0x55f4cf6f3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.812+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 shutdown_connections 2024-03-28T18:51:19.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.813+0000 7fabf064f740 1 -- 172.21.15.119:0/2668403411 wait complete. 2024-03-28T18:51:19.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.813+0000 7fabf064f740 20 client.27581 trim_cache size 0 max 16384 2024-03-28T18:51:19.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.826+0000 7fabf064f740 1 Processor -- start 2024-03-28T18:51:19.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.826+0000 7fabf064f740 1 -- start start 2024-03-28T18:51:19.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81af630 con 0x55f4cf6f8080 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81b0fb0 con 0x55f4cf6f76d0 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabf064f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81afc90 con 0x55f4cf700c00 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37320/0 (socket says 172.21.15.119:37320) 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 -- 172.21.15.119:0/3872604784 learned_addr learned my addr 172.21.15.119:0/3872604784 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe6a00640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe6000640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 -- 172.21.15.119:0/3872604784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 msgr2=0x55f4cf6f3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 -- 172.21.15.119:0/3872604784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f8080 msgr2=0x55f4cf6fffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.827+0000 7fabe7400640 1 -- 172.21.15.119:0/3872604784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4cf6f7290 con 0x55f4cf700c00 2024-03-28T18:51:19.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.828+0000 7fabe6a00640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:19.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.828+0000 7fabe7400640 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009ca0 tx=0x7fabe0040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4d806c531ec80d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.828+0000 7fabd3400640 1 -- 172.21.15.119:0/3872604784 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0039040 con 0x55f4cf700c00 2024-03-28T18:51:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.828+0000 7fabd3400640 1 -- 172.21.15.119:0/3872604784 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabe0053070 con 0x55f4cf700c00 2024-03-28T18:51:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.828+0000 7fabd3400640 1 -- 172.21.15.119:0/3872604784 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0040b30 con 0x55f4cf700c00 2024-03-28T18:51:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- 172.21.15.119:0/3872604784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 msgr2=0x55f4cf700520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fabe0009ca0 tx=0x7fabe0040000 comp rx=0 tx=0).stop 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- 172.21.15.119:0/3872604784 shutdown_connections 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf700c00 0x55f4cf700520 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf6f8080 0x55f4cf6fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 --2- 172.21.15.119:0/3872604784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f76d0 0x55f4cf6f3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- 172.21.15.119:0/3872604784 >> 172.21.15.119:0/3872604784 conn(0x55f4cf6f1da0 msgr2=0x55f4cf6f0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- 172.21.15.119:0/3872604784 shutdown_connections 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- 172.21.15.119:0/3872604784 wait complete. 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 -- start start 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.829+0000 7fabf064f740 1 Processor -- start 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf708e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 0x55f4cf706e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81b0950 con 0x55f4cf700c00 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81b02f0 con 0x55f4cf6f8080 2024-03-28T18:51:19.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe81ad810 con 0x55f4cf6f76d0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabf064f740 10 client.0 _ll_register_callbacks cb 0x7fabd409a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf708e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37322/0 (socket says 172.21.15.119:37322) 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 -- 172.21.15.119:0/556434064 learned_addr learned my addr 172.21.15.119:0/556434064 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe7400640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 0x55f4cf706e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 -- 172.21.15.119:0/556434064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 msgr2=0x55f4cf708e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf708e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.830+0000 7fabe6a00640 1 -- 172.21.15.119:0/556434064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 msgr2=0x55f4cf706e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabe6a00640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 0x55f4cf706e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabe6a00640 1 -- 172.21.15.119:0/556434064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4cf707680 con 0x55f4cf6f76d0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabe6000640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf708e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabe7400640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 0x55f4cf706e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabe6a00640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fabd801b040 tx=0x7fabd80097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b193308da80f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd802e5a0 con 0x55f4cf6f76d0 2024-03-28T18:51:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabe0053070 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fabd802e740 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabd803b050 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabf064f740 10 client.27562 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.831+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f4cf76cee0 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabe0053070 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fabd8019980 con 0x55f4cf6f76d0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 10 client.27562 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 10 client.27562 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 20 client.27562 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf76d410 0x55f4cf76f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 4 client.27562 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f4cf662b50 con 0x55f4cf76d410 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabe6000640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf76d410 0x55f4cf76f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabe6000640 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf76d410 0x55f4cf76f800 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d86100ca2f8b14a server_cookie=5eeef6d33962fb9d in_seq=0 out_seq=0 2024-03-28T18:51:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.832+0000 7fabd0c00640 10 client.27562 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabd0c00640 1 -- 172.21.15.119:0/556434064 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f4cf662b50 con 0x55f4cf76d410 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabd0c00640 10 client.27562 handle_command_reply: tid=0 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 client.27562 shutdown 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 msgr2=0x55f4cf7088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fabd801b040 tx=0x7fabd80097a0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 shutdown_connections 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55f4cf76d410 0x55f4cf76f800 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f4cf700c00 0x55f4cf706e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f4cf6f8080 0x55f4cf708e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabd0c00640 0 client.27562 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 --2- 172.21.15.119:0/556434064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4cf6f76d0 0x55f4cf7088e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 >> 172.21.15.119:0/556434064 conn(0x55f4cf6f1da0 msgr2=0x55f4cf6f84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 shutdown_connections 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 1 -- 172.21.15.119:0/556434064 wait complete. 2024-03-28T18:51:19.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.833+0000 7fabf064f740 20 client.27562 trim_cache size 0 max 16384 2024-03-28T18:51:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:19.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 msgr2=0x7fabcc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 0x7fabcc0a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fabe0037040 tx=0x7fabe002e040 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 msgr2=0x7fabe819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fabd400d950 tx=0x7fabd400de10 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 shutdown_connections 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fabcc0a2970 0x7fabcc0a4e30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fabe8109840 0x7fabe819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fabe81041f0 0x7fabe819a650 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 --2- 172.21.15.119:0/186455890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabe8103830 0x7fabe819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 >> 172.21.15.119:0/186455890 conn(0x7fabe80ff320 msgr2=0x7fabe80ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 shutdown_connections 2024-03-28T18:51:19.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:19.834+0000 7fabeda00640 1 -- 172.21.15.119:0/186455890 wait complete. 2024-03-28T18:51:19.853 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:19.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:51:19.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:19 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/491265858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:19.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:19 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/491265858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 -- 172.21.15.119:0/2416566044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0103840 msgr2=0x7f64b0103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 --2- 172.21.15.119:0/2416566044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0103840 0x7f64b0103cc0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6498009930 tx=0x7f649802bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 -- 172.21.15.119:0/2416566044 shutdown_connections 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 --2- 172.21.15.119:0/2416566044 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0104200 0x7f64b0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 --2- 172.21.15.119:0/2416566044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0103840 0x7f64b0103cc0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 --2- 172.21.15.119:0/2416566044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 0x7f64b0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 -- 172.21.15.119:0/2416566044 >> 172.21.15.119:0/2416566044 conn(0x7f64b00ff350 msgr2=0x7f64b0101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 -- 172.21.15.119:0/2416566044 shutdown_connections 2024-03-28T18:51:20.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.070+0000 7f64b7a00640 1 -- 172.21.15.119:0/2416566044 wait complete. 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 Processor -- start 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 -- start start 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 0x7f64b019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 0x7f64b019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 0x7f64b019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0113cb0 con 0x7f64b0109850 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0113b30 con 0x7f64b0103840 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.071+0000 7f64b7a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0113e30 con 0x7f64b0104200 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64b5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 0x7f64b019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64b5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 0x7f64b019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53142/0 (socket says 172.21.15.119:53142) 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 0x7f64b019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64b5000640 1 -- 172.21.15.119:0/33030180 learned_addr learned my addr 172.21.15.119:0/33030180 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64b5a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 0x7f64b019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 -- 172.21.15.119:0/33030180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 msgr2=0x7f64b019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 0x7f64b019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 -- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 msgr2=0x7f64b019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 0x7f64b019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 -- 172.21.15.119:0/33030180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b019f1d0 con 0x7f64b0104200 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.072+0000 7f64a7e00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 0x7f64b019a770 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6498036040 tx=0x7f6498005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.073+0000 7f64ae000640 1 -- 172.21.15.119:0/33030180 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498043020 con 0x7f64b0104200 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.073+0000 7f64ae000640 1 -- 172.21.15.119:0/33030180 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6498005000 con 0x7f64b0104200 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.073+0000 7f64ae000640 1 -- 172.21.15.119:0/33030180 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498003af0 con 0x7f64b0104200 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.073+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b019f460 con 0x7f64b0104200 2024-03-28T18:51:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.073+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b01a6d50 con 0x7f64b0104200 2024-03-28T18:51:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 Processor -- start 2024-03-28T18:51:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ae000640 1 -- 172.21.15.119:0/33030180 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6498046020 con 0x7f64b0104200 2024-03-28T18:51:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 -- start start 2024-03-28T18:51:20.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf52080 0x557cbdf5a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf5d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbdeb52c0 con 0x557cbdf52080 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbde5dc90 con 0x557cbdf516d0 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.076+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbdeb1360 con 0x557cbdf5ac00 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53150/0 (socket says 172.21.15.119:53150) 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 -- 172.21.15.119:0/1745678593 learned_addr learned my addr 172.21.15.119:0/1745678593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5000640 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf52080 0x557cbdf5a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64a7e00640 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf5d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 -- 172.21.15.119:0/1745678593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 msgr2=0x557cbdf5d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf5d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 -- 172.21.15.119:0/1745678593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf52080 msgr2=0x557cbdf5a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf52080 0x557cbdf5a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 -- 172.21.15.119:0/1745678593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cbde82080 con 0x557cbdf516d0 2024-03-28T18:51:20.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.077+0000 7f64b5a00640 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009ca0 tx=0x7f64a8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b17966a06cb6ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64a6000640 1 -- 172.21.15.119:0/1745678593 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a801e070 con 0x557cbdf516d0 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64a6000640 1 -- 172.21.15.119:0/1745678593 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a8019910 con 0x557cbdf516d0 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64a6000640 1 -- 172.21.15.119:0/1745678593 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800ea50 con 0x557cbdf516d0 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ae000640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 0x7f64940a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ae000640 1 -- 172.21.15.119:0/33030180 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f64980f0cd0 con 0x7f64b0104200 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64b5000640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 0x7f64940a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 -- 172.21.15.119:0/1745678593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 msgr2=0x557cbdf51ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009ca0 tx=0x7f64a8011810 comp rx=0 tx=0).stop 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 -- 172.21.15.119:0/1745678593 shutdown_connections 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf5d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf52080 0x557cbdf5a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/1745678593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf51ab0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 -- 172.21.15.119:0/1745678593 >> 172.21.15.119:0/1745678593 conn(0x557cbdf4bda0 msgr2=0x557cbdf4c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.078+0000 7f64ba7e3740 1 -- 172.21.15.119:0/1745678593 shutdown_connections 2024-03-28T18:51:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.079+0000 7f64ba7e3740 1 -- 172.21.15.119:0/1745678593 wait complete. 2024-03-28T18:51:20.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.079+0000 7f64b5000640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 0x7f64940a4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x557cbdf4ce10 tx=0x7f64a0009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.079+0000 7f64ba7e3740 1 -- start start 2024-03-28T18:51:20.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.079+0000 7f64ba7e3740 1 Processor -- start 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf578d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbdf5db70 con 0x557cbdf516d0 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbde710d0 con 0x557cbdf52080 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbdf5dcf0 con 0x557cbdf5ac00 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64ba7e3740 10 client.0 _ll_register_callbacks cb 0x557cbde75e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40076/0 (socket says 172.21.15.119:40076) 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53164/0 (socket says 172.21.15.119:53164) 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5000640 1 -- 172.21.15.119:0/2483960094 learned_addr learned my addr 172.21.15.119:0/2483960094 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37366/0 (socket says 172.21.15.119:37366) 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 -- 172.21.15.119:0/2483960094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 msgr2=0x557cbdf578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 -- 172.21.15.119:0/2483960094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 msgr2=0x557cbdf5a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5a00640 1 -- 172.21.15.119:0/2483960094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cbdbc93f0 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.080+0000 7f64b5000640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f64b5a00640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009ca0 tx=0x7f64a800e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e07309c0bf420b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f649ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8026030 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64a8019910 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f64a8019430 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64a8030050 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f64ba7e3740 10 client.27574 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.081+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557cbdc937d0 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a8019910 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f64a80052c0 con 0x557cbdf5ac00 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 10 client.27574 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:20.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 10 client.27574 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 20 client.27574 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdf6aa50 0x557cbdf6ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 4 client.27574 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cbdec2800 con 0x557cbdf6aa50 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.082+0000 7f64b5000640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdf6aa50 0x557cbdf6ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.083+0000 7f64b5000640 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdf6aa50 0x557cbdf6ce40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3351ab3fa079fa3 server_cookie=c7b19b849bef1f20 in_seq=0 out_seq=0 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.083+0000 7f649ea00640 10 client.27574 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.083+0000 7f649ea00640 1 -- 172.21.15.119:0/2483960094 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557cbdec2800 con 0x557cbdf6aa50 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.083+0000 7f649ea00640 10 client.27574 handle_command_reply: tid=0 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.083+0000 7f64ba7e3740 1 client.27574 shutdown 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 msgr2=0x557cbdf58010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009ca0 tx=0x7f64a800e5f0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 shutdown_connections 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdf6aa50 0x557cbdf6ce40 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf58010 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/2483960094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf5a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f649ea00640 0 client.27574 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 >> 172.21.15.119:0/2483960094 conn(0x557cbdf4bda0 msgr2=0x557cbdf4daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 shutdown_connections 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 1 -- 172.21.15.119:0/2483960094 wait complete. 2024-03-28T18:51:20.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.084+0000 7f64ba7e3740 20 client.27574 trim_cache size 0 max 16384 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 Processor -- start 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 -- start start 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf54b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01af6b0 con 0x557cbdf516d0 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01b1030 con 0x557cbdf52080 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01afd10 con 0x557cbdf5ac00 2024-03-28T18:51:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64b5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64b5000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40092/0 (socket says 172.21.15.119:40092) 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53174/0 (socket says 172.21.15.119:53174) 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64b5000640 1 -- 172.21.15.119:0/266403371 learned_addr learned my addr 172.21.15.119:0/266403371 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.098+0000 7f64b5a00640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf54b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 -- 172.21.15.119:0/266403371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 msgr2=0x557cbdf54b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf54b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 -- 172.21.15.119:0/266403371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 msgr2=0x557cbdf4d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 -- 172.21.15.119:0/266403371 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cbdf5da50 con 0x557cbdf52080 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64b5000640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64b5a00640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf54b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a7e00640 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f64980ee440 tx=0x7f649802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9cfe8989a225304 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a5600640 1 -- 172.21.15.119:0/266403371 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980bda50 con 0x557cbdf52080 2024-03-28T18:51:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a5600640 1 -- 172.21.15.119:0/266403371 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64980dbe50 con 0x557cbdf52080 2024-03-28T18:51:20.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.099+0000 7f64a5600640 1 -- 172.21.15.119:0/266403371 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64980b99c0 con 0x557cbdf52080 2024-03-28T18:51:20.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- 172.21.15.119:0/266403371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 msgr2=0x557cbdf5a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f64980ee440 tx=0x7f649802bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- 172.21.15.119:0/266403371 shutdown_connections 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf5ac00 0x557cbdf54b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf52080 0x557cbdf5a110 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/266403371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf516d0 0x557cbdf4d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- 172.21.15.119:0/266403371 >> 172.21.15.119:0/266403371 conn(0x557cbdf4bda0 msgr2=0x557cbdf4c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- 172.21.15.119:0/266403371 shutdown_connections 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- 172.21.15.119:0/266403371 wait complete. 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.100+0000 7f64ba7e3740 1 -- start start 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 Processor -- start 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf52080 0x557cbdf608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf5ac00 0x557cbdf60e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01b09d0 con 0x557cbdf5ac00 2024-03-28T18:51:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01b0370 con 0x557cbdf516d0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b01ad890 con 0x557cbdf52080 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64ba7e3740 10 client.0 _ll_register_callbacks cb 0x7f64a801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64b5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64b5000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53176/0 (socket says 172.21.15.119:53176) 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf52080 0x557cbdf608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64b5000640 1 -- 172.21.15.119:0/381930121 learned_addr learned my addr 172.21.15.119:0/381930121 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.101+0000 7f64b5a00640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf5ac00 0x557cbdf60e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 -- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf52080 msgr2=0x557cbdf608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf52080 0x557cbdf608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 -- 172.21.15.119:0/381930121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf5ac00 msgr2=0x557cbdf60e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf5ac00 0x557cbdf60e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 -- 172.21.15.119:0/381930121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cbdfc5180 con 0x557cbdf516d0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64b5000640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64a0004440 tx=0x7f64a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b184b9e83b8b73b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f649ea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a002b740 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64980dbe50 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f64a002b8e0 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.102+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64a002bae0 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 10 client.27605 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557cbdfc6480 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64980dbe50 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f64a0039030 con 0x557cbdf516d0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 10 client.27605 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 10 client.27605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 20 client.27605 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdfc6be0 0x557cbdfc9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 4 client.27605 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557cbdebcb50 con 0x557cbdfc6be0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.103+0000 7f64a7e00640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdfc6be0 0x557cbdfc9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64a7e00640 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdfc6be0 0x557cbdfc9000 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ee0dade3d00ea35 server_cookie=f03b7bc3e18adb36 in_seq=0 out_seq=0 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f649ea00640 10 client.27605 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f649ea00640 1 -- 172.21.15.119:0/381930121 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557cbdebcb50 con 0x557cbdfc6be0 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f649ea00640 10 client.27605 handle_command_reply: tid=0 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 client.27605 shutdown 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 msgr2=0x557cbdf603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64a0004440 tx=0x7f64a0004a40 comp rx=0 tx=0).stop 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 shutdown_connections 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x557cbdfc6be0 0x557cbdfc9000 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cbdf5ac00 0x557cbdf60e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557cbdf52080 0x557cbdf608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 --2- 172.21.15.119:0/381930121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557cbdf516d0 0x557cbdf603a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 >> 172.21.15.119:0/381930121 conn(0x557cbdf4bda0 msgr2=0x557cbdf52cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f649ea00640 0 client.27605 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 shutdown_connections 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 1 -- 172.21.15.119:0/381930121 wait complete. 2024-03-28T18:51:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.104+0000 7f64ba7e3740 20 client.27605 trim_cache size 0 max 16384 2024-03-28T18:51:20.109 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 msgr2=0x7f64940a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 0x7f64940a4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x557cbdf4ce10 tx=0x7f64a0009500 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 msgr2=0x7f64b019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 0x7f64b019a770 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6498036040 tx=0x7f6498005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 shutdown_connections 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f64940a29f0 0x7f64940a4eb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64b0109850 0x7f64b019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b0104200 0x7f64b019a770 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 --2- 172.21.15.119:0/33030180 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f64b0103840 0x7f64b019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 >> 172.21.15.119:0/33030180 conn(0x7f64b00ff350 msgr2=0x7f64b00ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 shutdown_connections 2024-03-28T18:51:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.105+0000 7f64b7a00640 1 -- 172.21.15.119:0/33030180 wait complete. 2024-03-28T18:51:20.124 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:20.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:51:20.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 -- 172.21.15.119:0/3381643502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0101630 msgr2=0x7f75f0101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 --2- 172.21.15.119:0/3381643502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0101630 0x7f75f0101ab0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009930 tx=0x7f75dc02bc20 comp rx=0 tx=0).stop 2024-03-28T18:51:20.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 -- 172.21.15.119:0/3381643502 shutdown_connections 2024-03-28T18:51:20.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 --2- 172.21.15.119:0/3381643502 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101ff0 0x7f75f010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 --2- 172.21.15.119:0/3381643502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0101630 0x7f75f0101ab0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 --2- 172.21.15.119:0/3381643502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0107640 0x7f75f0107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 -- 172.21.15.119:0/3381643502 >> 172.21.15.119:0/3381643502 conn(0x7f75f0078aa0 msgr2=0x7f75f0078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 -- 172.21.15.119:0/3381643502 shutdown_connections 2024-03-28T18:51:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.336+0000 7f75f6200640 1 -- 172.21.15.119:0/3381643502 wait complete. 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.337+0000 7f75f6200640 1 Processor -- start 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.337+0000 7f75f6200640 1 -- start start 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0101ff0 0x7f75f0073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0107640 0x7f75f0074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0111a60 con 0x7f75f0101ff0 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01118e0 con 0x7f75f0101630 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75f6200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0111be0 con 0x7f75f0107640 2024-03-28T18:51:20.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53194/0 (socket says 172.21.15.119:53194) 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 -- 172.21.15.119:0/738344737 learned_addr learned my addr 172.21.15.119:0/738344737 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75efe00640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0107640 0x7f75f0074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75eea00640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0101ff0 0x7f75f0073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 -- 172.21.15.119:0/738344737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0107640 msgr2=0x7f75f0074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0107640 0x7f75f0074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 -- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0101ff0 msgr2=0x7f75f0073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0101ff0 0x7f75f0073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.338+0000 7f75ef400640 1 -- 172.21.15.119:0/738344737 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f01a6480 con 0x7f75f0101630 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75ef400640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f75e000ea60 tx=0x7f75e000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75e7e00640 1 -- 172.21.15.119:0/738344737 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e000cd20 con 0x7f75f0101630 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75e7e00640 1 -- 172.21.15.119:0/738344737 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e00035f0 con 0x7f75f0101630 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f01a6770 con 0x7f75f0101630 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75e7e00640 1 -- 172.21.15.119:0/738344737 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0010690 con 0x7f75f0101630 2024-03-28T18:51:20.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.339+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f01a6ce0 con 0x7f75f0101630 2024-03-28T18:51:20.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.342+0000 7f75e7e00640 1 -- 172.21.15.119:0/738344737 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f75e00037a0 con 0x7f75f0101630 2024-03-28T18:51:20.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.342+0000 7f75f8fcd740 1 Processor -- start 2024-03-28T18:51:20.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 -- start start 2024-03-28T18:51:20.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c4080 0x5618cb5cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5ccc00 0x5618cb5cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb5272c0 con 0x5618cb5ccc00 2024-03-28T18:51:20.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb4cfc90 con 0x5618cb5c36d0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb523360 con 0x5618cb5c4080 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c4080 0x5618cb5cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75efe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75efe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53208/0 (socket says 172.21.15.119:53208) 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75efe00640 1 -- 172.21.15.119:0/3409266323 learned_addr learned my addr 172.21.15.119:0/3409266323 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.343+0000 7f75eea00640 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5ccc00 0x5618cb5cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 -- 172.21.15.119:0/3409266323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c4080 msgr2=0x5618cb5cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c4080 0x5618cb5cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 -- 172.21.15.119:0/3409266323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5ccc00 msgr2=0x5618cb5cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5ccc00 0x5618cb5cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 -- 172.21.15.119:0/3409266323 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618cb4f4080 con 0x5618cb5c36d0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75e7e00640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 0x7f75d40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75efe00640 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011440 tx=0x7f75e8011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=636d07d30886ea39 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75e7e00640 1 -- 172.21.15.119:0/738344737 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75e00cf150 con 0x7f75f0101630 2024-03-28T18:51:20.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75ef400640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 0x7f75d40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75dbe00640 1 -- 172.21.15.119:0/3409266323 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8003c30 con 0x5618cb5c36d0 2024-03-28T18:51:20.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75dbe00640 1 -- 172.21.15.119:0/3409266323 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e8003dd0 con 0x5618cb5c36d0 2024-03-28T18:51:20.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.344+0000 7f75dbe00640 1 -- 172.21.15.119:0/3409266323 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e800fb50 con 0x5618cb5c36d0 2024-03-28T18:51:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 -- 172.21.15.119:0/3409266323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 msgr2=0x5618cb5c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75ef400640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 0x7f75d40a4eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5618cb5c31a0 tx=0x7f75e00d2c00 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011440 tx=0x7f75e8011900 comp rx=0 tx=0).stop 2024-03-28T18:51:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 -- 172.21.15.119:0/3409266323 shutdown_connections 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5ccc00 0x5618cb5cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c4080 0x5618cb5cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/3409266323 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c36d0 0x5618cb5c3ab0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 -- 172.21.15.119:0/3409266323 >> 172.21.15.119:0/3409266323 conn(0x5618cb5bdda0 msgr2=0x5618cb5be180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 -- 172.21.15.119:0/3409266323 shutdown_connections 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.345+0000 7f75f8fcd740 1 -- 172.21.15.119:0/3409266323 wait complete. 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 -- start start 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 Processor -- start 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5c9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb5cfb70 con 0x5618cb5c4080 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb4e30d0 con 0x5618cb5ccc00 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618cb5cfcf0 con 0x5618cb5c36d0 2024-03-28T18:51:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.346+0000 7f75f8fcd740 10 client.0 _ll_register_callbacks cb 0x5618cb4e7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37434/0 (socket says 172.21.15.119:37434) 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75efe00640 1 -- 172.21.15.119:0/1944459013 learned_addr learned my addr 172.21.15.119:0/1944459013 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75efe00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 -- 172.21.15.119:0/1944459013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 msgr2=0x5618cb5cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 -- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 msgr2=0x5618cb5c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 -- 172.21.15.119:0/1944459013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618cb23b3f0 con 0x5618cb5c4080 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75eea00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5c9850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9aafe9fabbd4c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.347+0000 7f75d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc02ca80 con 0x5618cb5c4080 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75dc02cc20 con 0x5618cb5c4080 2024-03-28T18:51:20.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75e8003dd0 con 0x5618cb5c4080 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75dc0046c0 con 0x5618cb5c4080 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75f8fcd740 10 client.17388 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5618cb3057d0 con 0x5618cb5c4080 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e8003dd0 con 0x5618cb5c4080 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75dc0035f0 con 0x5618cb5c4080 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75f8fcd740 10 client.17388 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.348+0000 7f75f8fcd740 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75f8fcd740 20 client.17388 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb5dc990 0x5618cb5ded80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75f8fcd740 4 client.17388 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618cb534800 con 0x5618cb5dc990 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75efe00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb5dc990 0x5618cb5ded80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75efe00640 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb5dc990 0x5618cb5ded80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c5b61da757f8496 server_cookie=7a7fc5934b0b5d09 in_seq=0 out_seq=0 2024-03-28T18:51:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.349+0000 7f75d8c00640 10 client.17388 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75d8c00640 1 -- 172.21.15.119:0/1944459013 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5618cb534800 con 0x5618cb5dc990 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75d8c00640 10 client.17388 handle_command_reply: tid=0 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 client.17388 shutdown 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 msgr2=0x5618cb5c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5c9850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc0092f0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 shutdown_connections 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb5dc990 0x5618cb5ded80 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5c9850 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/1944459013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75d8c00640 0 client.17388 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 >> 172.21.15.119:0/1944459013 conn(0x5618cb5bdda0 msgr2=0x5618cb5bfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 shutdown_connections 2024-03-28T18:51:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.350+0000 7f75f8fcd740 1 -- 172.21.15.119:0/1944459013 wait complete. 2024-03-28T18:51:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.351+0000 7f75f8fcd740 20 client.17388 trim_cache size 0 max 16384 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.364+0000 7f75f8fcd740 1 Processor -- start 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.364+0000 7f75f8fcd740 1 -- start start 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.364+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.364+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01af720 con 0x5618cb5c4080 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01b10a0 con 0x5618cb5ccc00 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01afd80 con 0x5618cb5c36d0 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37442/0 (socket says 172.21.15.119:37442) 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 -- 172.21.15.119:0/2188074365 learned_addr learned my addr 172.21.15.119:0/2188074365 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75eea00640 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 -- 172.21.15.119:0/2188074365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 msgr2=0x5618cb5c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 -- 172.21.15.119:0/2188074365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 msgr2=0x5618cb5cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 -- 172.21.15.119:0/2188074365 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618cb5ca120 con 0x5618cb5c36d0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.365+0000 7f75efe00640 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75e80049d0 tx=0x7f75e8011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96d2d40e7b771df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75db400640 1 -- 172.21.15.119:0/2188074365 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8004c00 con 0x5618cb5c36d0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75db400640 1 -- 172.21.15.119:0/2188074365 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e8004da0 con 0x5618cb5c36d0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75db400640 1 -- 172.21.15.119:0/2188074365 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8004560 con 0x5618cb5c36d0 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2188074365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 msgr2=0x5618cb5cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75e80049d0 tx=0x7f75e8011900 comp rx=0 tx=0).stop 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2188074365 shutdown_connections 2024-03-28T18:51:20.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5ccc00 0x5618cb5c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c4080 0x5618cb5cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2188074365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5c36d0 0x5618cb5cbf80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2188074365 >> 172.21.15.119:0/2188074365 conn(0x5618cb5bdda0 msgr2=0x5618cb5bfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.366+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2188074365 shutdown_connections 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2188074365 wait complete. 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 -- start start 2024-03-28T18:51:20.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 Processor -- start 2024-03-28T18:51:20.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 0x5618cb5d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.367+0000 7f75f8fcd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5ccc00 0x5618cb5d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01b0a40 con 0x5618cb5c36d0 2024-03-28T18:51:20.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01b03e0 con 0x5618cb5c4080 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75f8fcd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f01ad900 con 0x5618cb5ccc00 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75f8fcd740 10 client.0 _ll_register_callbacks cb 0x7f75dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75efe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 0x5618cb5d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53226/0 (socket says 172.21.15.119:53226) 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 -- 172.21.15.119:0/2351783870 learned_addr learned my addr 172.21.15.119:0/2351783870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75efe00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5ccc00 0x5618cb5d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 -- 172.21.15.119:0/2351783870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5ccc00 msgr2=0x5618cb5d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5ccc00 0x5618cb5d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 -- 172.21.15.119:0/2351783870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 msgr2=0x5618cb5d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 0x5618cb5d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 -- 172.21.15.119:0/2351783870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618cb5d1820 con 0x5618cb5c4080 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75efe00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 0x5618cb5d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:20.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75eea00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f101c51d9264dc60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.368+0000 7f75d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc009520 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75dc004860 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75e8004da0 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75dc063050 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 10 client.27623 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5618cb638ee0 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e8004da0 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75dc004ba0 con 0x5618cb5c4080 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 10 client.27623 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 10 client.27623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 20 client.27623 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb6395e0 0x5618cb63ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 4 client.27623 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:20.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.369+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5618cb52eb50 con 0x5618cb6395e0 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75efe00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb6395e0 0x5618cb63ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75efe00640 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb6395e0 0x5618cb63ba00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7524b66ba9d559 server_cookie=89d8e06ad389b739 in_seq=0 out_seq=0 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75d8c00640 10 client.27623 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75d8c00640 1 -- 172.21.15.119:0/2351783870 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5618cb52eb50 con 0x5618cb6395e0 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75d8c00640 10 client.27623 handle_command_reply: tid=0 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 client.27623 shutdown 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 msgr2=0x5618cb5d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f75dc036040 tx=0x7f75dc005fd0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 shutdown_connections 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5618cb6395e0 0x5618cb63ba00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5618cb5ccc00 0x5618cb5d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5618cb5c4080 0x5618cb5d0da0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75d8c00640 0 client.27623 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 --2- 172.21.15.119:0/2351783870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5618cb5c36d0 0x5618cb5d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.370+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 >> 172.21.15.119:0/2351783870 conn(0x5618cb5bdda0 msgr2=0x5618cb5bc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 shutdown_connections 2024-03-28T18:51:20.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f8fcd740 1 -- 172.21.15.119:0/2351783870 wait complete. 2024-03-28T18:51:20.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f8fcd740 20 client.27623 trim_cache size 0 max 16384 2024-03-28T18:51:20.375 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 msgr2=0x7f75d40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 0x7f75d40a4eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5618cb5c31a0 tx=0x7f75e00d2c00 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 msgr2=0x7f75f00739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f75e000ea60 tx=0x7f75e000ee60 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 shutdown_connections 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d40a29f0 0x7f75d40a4eb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75f0107640 0x7f75f0074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75f0101ff0 0x7f75f0073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 --2- 172.21.15.119:0/738344737 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75f0101630 0x7f75f00739a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.371+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 >> 172.21.15.119:0/738344737 conn(0x7f75f0078aa0 msgr2=0x7f75f0104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.372+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 shutdown_connections 2024-03-28T18:51:20.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:20.372+0000 7f75f6200640 1 -- 172.21.15.119:0/738344737 wait complete. 2024-03-28T18:51:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:20 smithi119 ceph-mon[31813]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:51:20.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:20 smithi196 ceph-mon[36300]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:51:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:20 smithi131 ceph-mon[36065]: pgmap v223: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:51:21.391 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:51:21.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:51:21.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:51:21.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 -- 172.21.15.119:0/2627182124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 msgr2=0x7f30c4075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 --2- 172.21.15.119:0/2627182124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c4075fb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009930 tx=0x7f30b802bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:21.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 -- 172.21.15.119:0/2627182124 shutdown_connections 2024-03-28T18:51:21.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 --2- 172.21.15.119:0/2627182124 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 0x7f30c41148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 --2- 172.21.15.119:0/2627182124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 0x7f30c4111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 --2- 172.21.15.119:0/2627182124 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c4075fb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 -- 172.21.15.119:0/2627182124 >> 172.21.15.119:0/2627182124 conn(0x7f30c40ff310 msgr2=0x7f30c4101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 -- 172.21.15.119:0/2627182124 shutdown_connections 2024-03-28T18:51:21.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.623+0000 7f30cba00640 1 -- 172.21.15.119:0/2627182124 wait complete. 2024-03-28T18:51:21.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.624+0000 7f30cba00640 1 Processor -- start 2024-03-28T18:51:21.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.624+0000 7f30cba00640 1 -- start start 2024-03-28T18:51:21.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 0x7f30c419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 0x7f30c41a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4118090 con 0x7f30c40764f0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4117f10 con 0x7f30c4075bd0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30cba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4118210 con 0x7f30c41124e0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53246/0 (socket says 172.21.15.119:53246) 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 -- 172.21.15.119:0/116914655 learned_addr learned my addr 172.21.15.119:0/116914655 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9a00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 0x7f30c41a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c3e00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 0x7f30c419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 -- 172.21.15.119:0/116914655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 msgr2=0x7f30c41a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 0x7f30c41a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 -- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 msgr2=0x7f30c419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 0x7f30c419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.625+0000 7f30c9000640 1 -- 172.21.15.119:0/116914655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c41a3580 con 0x7f30c4075bd0 2024-03-28T18:51:21.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30c9000640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009900 tx=0x7f30b8004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30c1600640 1 -- 172.21.15.119:0/116914655 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8043020 con 0x7f30c4075bd0 2024-03-28T18:51:21.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c41a3810 con 0x7f30c4075bd0 2024-03-28T18:51:21.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30c1600640 1 -- 172.21.15.119:0/116914655 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b80054e0 con 0x7f30c4075bd0 2024-03-28T18:51:21.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30c1600640 1 -- 172.21.15.119:0/116914655 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8003b20 con 0x7f30c4075bd0 2024-03-28T18:51:21.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.626+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c41ab100 con 0x7f30c4075bd0 2024-03-28T18:51:21.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.630+0000 7f30ce62d740 1 Processor -- start 2024-03-28T18:51:21.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 -- start start 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c1600640 1 -- 172.21.15.119:0/116914655 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f30b8046020 con 0x7f30c4075bd0 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed0a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ed0c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ec65eb0 con 0x55e01ed0a580 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ec0ea60 con 0x55e01ed019a0 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30ce62d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ec61f50 con 0x55e01ed01080 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37468/0 (socket says 172.21.15.119:37468) 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed0a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c9a00640 1 -- 172.21.15.119:0/4128363485 learned_addr learned my addr 172.21.15.119:0/4128363485 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.631+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ed0c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 -- 172.21.15.119:0/4128363485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 msgr2=0x55e01ed0a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed0a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 -- 172.21.15.119:0/4128363485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 msgr2=0x55e01ed0c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ed0c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 -- 172.21.15.119:0/4128363485 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e01ea427e0 con 0x55e01ed01080 2024-03-28T18:51:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c9a00640 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009f40 tx=0x7f30bc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f54f912ea0e86e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30b6000640 1 -- 172.21.15.119:0/4128363485 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc004d20 con 0x55e01ed01080 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30b6000640 1 -- 172.21.15.119:0/4128363485 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30bc004050 con 0x55e01ed01080 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30b6000640 1 -- 172.21.15.119:0/4128363485 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc005280 con 0x55e01ed01080 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.632+0000 7f30c1600640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 0x7f30ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30c1600640 1 -- 172.21.15.119:0/116914655 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f30b80f0f90 con 0x7f30c4075bd0 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30c9000640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 0x7f30ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 -- 172.21.15.119:0/4128363485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 msgr2=0x55e01ed01460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f30bc009f40 tx=0x7f30bc0118d0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 -- 172.21.15.119:0/4128363485 shutdown_connections 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ed0c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed0a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 --2- 172.21.15.119:0/4128363485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed01460 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 -- 172.21.15.119:0/4128363485 >> 172.21.15.119:0/4128363485 conn(0x55e01ecfb7b0 msgr2=0x55e01ecfbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 -- 172.21.15.119:0/4128363485 shutdown_connections 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30ce62d740 1 -- 172.21.15.119:0/4128363485 wait complete. 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.633+0000 7f30c9000640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 0x7f30ac0a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e01ed00b50 tx=0x7f30b80f4a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 -- start start 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 Processor -- start 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed07780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ed0d370 con 0x55e01ed019a0 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ec21eb0 con 0x55e01ed0a580 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.634+0000 7f30ce62d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01ed0d4f0 con 0x55e01ed01080 2024-03-28T18:51:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30ce62d740 10 client.0 _ll_register_callbacks cb 0x55e01ec16150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40182/0 (socket says 172.21.15.119:40182) 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 -- 172.21.15.119:0/2050458750 learned_addr learned my addr 172.21.15.119:0/2050458750 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c9a00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed07780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 -- 172.21.15.119:0/2050458750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 msgr2=0x55e01ed09eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 -- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 msgr2=0x55e01ed07780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed07780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 -- 172.21.15.119:0/2050458750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e01eaa2f70 con 0x55e01ed019a0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c9a00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed07780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30c3e00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30b0004650 tx=0x7f30b00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1d0e01dc94c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.635+0000 7f30aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0011910 con 0x55e01ed019a0 2024-03-28T18:51:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f30b0011ab0 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30bc004050 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b0011cb0 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30ce62d740 10 client.17403 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e01e93a170 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30bc004050 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.636+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f30b0018070 con 0x55e01ed019a0 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 10 client.17403 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 20 client.17403 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed1a670 0x55e01ed1ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 4 client.17403 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e01ec733f0 con 0x55e01ed1a670 2024-03-28T18:51:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30c9a00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed1a670 0x55e01ed1ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30c9a00640 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed1a670 0x55e01ed1ca80 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=458cc6a3a45863b3 server_cookie=bf0f5bb716623ab7 in_seq=0 out_seq=0 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.637+0000 7f30aaa00640 10 client.17403 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30aaa00640 1 -- 172.21.15.119:0/2050458750 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e01ec733f0 con 0x55e01ed1a670 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30aaa00640 10 client.17403 handle_command_reply: tid=0 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 client.17403 shutdown 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 msgr2=0x55e01ed07240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30b0004650 tx=0x7f30b00039c0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 shutdown_connections 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed1a670 0x55e01ed1ca80 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed07780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30aaa00640 0 client.17403 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed019a0 0x55e01ed07240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 --2- 172.21.15.119:0/2050458750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.638+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 >> 172.21.15.119:0/2050458750 conn(0x55e01ecfb7b0 msgr2=0x55e01ecfd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.639+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 shutdown_connections 2024-03-28T18:51:21.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.639+0000 7f30ce62d740 1 -- 172.21.15.119:0/2050458750 wait complete. 2024-03-28T18:51:21.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.639+0000 7f30ce62d740 20 client.17403 trim_cache size 0 max 16384 2024-03-28T18:51:21.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 Processor -- start 2024-03-28T18:51:21.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 -- start start 2024-03-28T18:51:21.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ecfa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b3a50 con 0x55e01ed0a580 2024-03-28T18:51:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b53d0 con 0x55e01ed019a0 2024-03-28T18:51:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.656+0000 7f30ce62d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b40b0 con 0x55e01ed01080 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53276/0 (socket says 172.21.15.119:53276) 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 -- 172.21.15.119:0/1901214235 learned_addr learned my addr 172.21.15.119:0/1901214235 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c9a00640 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ecfa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 -- 172.21.15.119:0/1901214235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 msgr2=0x55e01ed09970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 -- 172.21.15.119:0/1901214235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 msgr2=0x55e01ecfa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ecfa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c3e00640 1 -- 172.21.15.119:0/1901214235 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e01ecf6c20 con 0x55e01ed019a0 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.657+0000 7f30c9a00640 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ecfa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30c3e00640 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f30b0001eb0 tx=0x7f30b000c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263aca692292e7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30b5600640 1 -- 172.21.15.119:0/1901214235 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0000e60 con 0x55e01ed019a0 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30b5600640 1 -- 172.21.15.119:0/1901214235 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b0005510 con 0x55e01ed019a0 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30b5600640 1 -- 172.21.15.119:0/1901214235 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0019bc0 con 0x55e01ed019a0 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 -- 172.21.15.119:0/1901214235 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 msgr2=0x55e01ed09eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f30b0001eb0 tx=0x7f30b000c6b0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 -- 172.21.15.119:0/1901214235 shutdown_connections 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed0a580 0x55e01ecfa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed019a0 0x55e01ed09eb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 --2- 172.21.15.119:0/1901214235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed01080 0x55e01ed09970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 -- 172.21.15.119:0/1901214235 >> 172.21.15.119:0/1901214235 conn(0x55e01ecfb7b0 msgr2=0x55e01ecf9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.658+0000 7f30ce62d740 1 -- 172.21.15.119:0/1901214235 shutdown_connections 2024-03-28T18:51:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.659+0000 7f30ce62d740 1 -- 172.21.15.119:0/1901214235 wait complete. 2024-03-28T18:51:21.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.659+0000 7f30ce62d740 1 -- start start 2024-03-28T18:51:21.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.659+0000 7f30ce62d740 1 Processor -- start 2024-03-28T18:51:21.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.659+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed0efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b4d70 con 0x55e01ed01080 2024-03-28T18:51:21.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b4710 con 0x55e01ed0a580 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c41b1c30 con 0x55e01ed019a0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30ce62d740 10 client.0 _ll_register_callbacks cb 0x7f30b0004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40200/0 (socket says 172.21.15.119:40200) 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37490/0 (socket says 172.21.15.119:37490) 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c9a00640 1 -- 172.21.15.119:0/665121563 learned_addr learned my addr 172.21.15.119:0/665121563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c9a00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed0efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 -- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 msgr2=0x55e01ed0efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed0efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 -- 172.21.15.119:0/665121563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 msgr2=0x55e01ed12630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c3e00640 1 -- 172.21.15.119:0/665121563 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e01ed0f7c0 con 0x55e01ed019a0 2024-03-28T18:51:21.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.660+0000 7f30c9a00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30c3e00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f30b0000c00 tx=0x7f30b0019a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd274f29eae2c311 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0000e60 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30b0005510 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f30b001a440 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30b000c430 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30ce62d740 10 client.27616 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e01ed77060 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30b0005510 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.661+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f30b0018070 con 0x55e01ed019a0 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 10 client.27616 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 10 client.27616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 20 client.27616 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed77660 0x55e01ed79a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 4 client.27616 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e01ec6d740 con 0x55e01ed77660 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30c9a00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed77660 0x55e01ed79a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30aaa00640 10 client.27616 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.662+0000 7f30c9a00640 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed77660 0x55e01ed79a50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3a7983011564987 server_cookie=790ceaaf99d569a4 in_seq=0 out_seq=0 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30aaa00640 1 -- 172.21.15.119:0/665121563 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e01ec6d740 con 0x55e01ed77660 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30aaa00640 10 client.27616 handle_command_reply: tid=0 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 client.27616 shutdown 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 msgr2=0x55e01ed0e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f30b0000c00 tx=0x7f30b0019a30 comp rx=0 tx=0).stop 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 shutdown_connections 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e01ed77660 0x55e01ed79a50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e01ed0a580 0x55e01ed0efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e01ed019a0 0x55e01ed0e9b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 --2- 172.21.15.119:0/665121563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e01ed01080 0x55e01ed12630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 >> 172.21.15.119:0/665121563 conn(0x55e01ecfb7b0 msgr2=0x55e01ecfa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30aaa00640 0 client.27616 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 shutdown_connections 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 1 -- 172.21.15.119:0/665121563 wait complete. 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.664+0000 7f30ce62d740 20 client.27616 trim_cache size 0 max 16384 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "ec02546b-bf8f-4495-bd91-9c49d95291e1", 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:51:21.668 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 msgr2=0x7f30ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 0x7f30ac0a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e01ed00b50 tx=0x7f30b80f4a40 comp rx=0 tx=0).stop 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 msgr2=0x7f30c419e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f30b8009900 tx=0x7f30b8004360 comp rx=0 tx=0).stop 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 shutdown_connections 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f30ac0a2970 0x7f30ac0a4e30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30c41124e0 0x7f30c41a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30c40764f0 0x7f30c419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 --2- 172.21.15.119:0/116914655 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f30c4075bd0 0x7f30c419e590 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 >> 172.21.15.119:0/116914655 conn(0x7f30c40ff310 msgr2=0x7f30c4113b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 shutdown_connections 2024-03-28T18:51:21.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.665+0000 7f30cba00640 1 -- 172.21.15.119:0/116914655 wait complete. 2024-03-28T18:51:21.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:51:21.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:51:21.764 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:21 smithi119 ceph-mon[31813]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 -- 172.21.15.119:0/740846915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 msgr2=0x7f2c5c112100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 --2- 172.21.15.119:0/740846915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c112100 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009900 tx=0x7f2c4402bcd0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 -- 172.21.15.119:0/740846915 shutdown_connections 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 --2- 172.21.15.119:0/740846915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c112640 0x7f2c5c114a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 --2- 172.21.15.119:0/740846915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c112100 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 --2- 172.21.15.119:0/740846915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c073a40 0x7f2c5c073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.903+0000 7f2c63000640 1 -- 172.21.15.119:0/740846915 >> 172.21.15.119:0/740846915 conn(0x7f2c5c078aa0 msgr2=0x7f2c5c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.904+0000 7f2c63000640 1 -- 172.21.15.119:0/740846915 shutdown_connections 2024-03-28T18:51:21.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.904+0000 7f2c63000640 1 -- 172.21.15.119:0/740846915 wait complete. 2024-03-28T18:51:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.904+0000 7f2c63000640 1 Processor -- start 2024-03-28T18:51:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 -- start start 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 0x7f2c5c19e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c118090 con 0x7f2c5c073a40 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c117f10 con 0x7f2c5c074360 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c63000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c118210 con 0x7f2c5c112640 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 0x7f2c5c19e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c61000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c53400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 0x7f2c5c19e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40228/0 (socket says 172.21.15.119:40228) 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c5be00640 1 -- 172.21.15.119:0/274329840 learned_addr learned my addr 172.21.15.119:0/274329840 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.905+0000 7f2c61000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37510/0 (socket says 172.21.15.119:37510) 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 -- 172.21.15.119:0/274329840 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 msgr2=0x7f2c5c1a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 -- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 msgr2=0x7f2c5c19e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 0x7f2c5c19e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 -- 172.21.15.119:0/274329840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c1a34c0 con 0x7f2c5c074360 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c61000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c53400640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c19ea10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2c4402c210 tx=0x7f2c44005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c59600640 1 -- 172.21.15.119:0/274329840 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44042020 con 0x7f2c5c074360 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c59600640 1 -- 172.21.15.119:0/274329840 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c4402ce00 con 0x7f2c5c074360 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c1a3750 con 0x7f2c5c074360 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c59600640 1 -- 172.21.15.119:0/274329840 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44003a50 con 0x7f2c5c074360 2024-03-28T18:51:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.906+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5c1ab040 con 0x7f2c5c074360 2024-03-28T18:51:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.909+0000 7f2c59600640 1 -- 172.21.15.119:0/274329840 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2c44045020 con 0x7f2c5c074360 2024-03-28T18:51:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 Processor -- start 2024-03-28T18:51:21.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 -- start start 2024-03-28T18:51:21.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e508b250 con 0x5581e5127c00 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5033a30 con 0x5581e5126f20 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c65be0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e50872f0 con 0x5581e5130290 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.910+0000 7f2c61000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c53400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c61000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40236/0 (socket says 172.21.15.119:40236) 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c61000640 1 -- 172.21.15.119:0/1899269379 learned_addr learned my addr 172.21.15.119:0/1899269379 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 -- 172.21.15.119:0/1899269379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 msgr2=0x5581e5132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 -- 172.21.15.119:0/1899269379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 msgr2=0x5581e5126950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 -- 172.21.15.119:0/1899269379 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e501e690 con 0x5581e5126f20 2024-03-28T18:51:21.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c61000640 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c59600640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 0x7f2c400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c59600640 1 -- 172.21.15.119:0/274329840 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2c440f0880 con 0x7f2c5c074360 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51273a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009c80 tx=0x7f2c4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee8b25cc28af04e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c61000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 0x7f2c400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.911+0000 7f2c51600640 1 -- 172.21.15.119:0/1899269379 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c004230 con 0x5581e5126f20 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c51600640 1 -- 172.21.15.119:0/1899269379 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c0043d0 con 0x5581e5126f20 2024-03-28T18:51:21.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c51600640 1 -- 172.21.15.119:0/1899269379 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c004df0 con 0x5581e5126f20 2024-03-28T18:51:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 -- 172.21.15.119:0/1899269379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 msgr2=0x5581e51273a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51273a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c009c80 tx=0x7f2c4c011810 comp rx=0 tx=0).stop 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c61000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 0x7f2c400a4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009fd0 tx=0x7f2c54009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 -- 172.21.15.119:0/1899269379 shutdown_connections 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51273a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1899269379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5126950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.912+0000 7f2c65be0740 1 -- 172.21.15.119:0/1899269379 >> 172.21.15.119:0/1899269379 conn(0x5581e50ee200 msgr2=0x5581e5121970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.913+0000 7f2c65be0740 1 -- 172.21.15.119:0/1899269379 shutdown_connections 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.913+0000 7f2c65be0740 1 -- 172.21.15.119:0/1899269379 wait complete. 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.913+0000 7f2c65be0740 1 -- start start 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.913+0000 7f2c65be0740 1 Processor -- start 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e512cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e512d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5133100 con 0x5581e5127c00 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5046e80 con 0x5581e5126f20 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581e5133280 con 0x5581e5130290 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c65be0740 10 client.0 _ll_register_callbacks cb 0x5581e5059130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53316/0 (socket says 172.21.15.119:53316) 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 -- 172.21.15.119:0/1166609824 learned_addr learned my addr 172.21.15.119:0/1166609824 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c53400640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e512cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c61000640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e512d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 -- 172.21.15.119:0/1166609824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 msgr2=0x5581e512d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e512d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 -- 172.21.15.119:0/1166609824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 msgr2=0x5581e512cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e512cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c5be00640 1 -- 172.21.15.119:0/1166609824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581e4e697d0 con 0x5581e5126f20 2024-03-28T18:51:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.914+0000 7f2c61000640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e512d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c5be00640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0048f0 tx=0x7f2c4c019a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b76071a55d4785 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c4a000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c028020 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2c4c024620 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c4c0043d0 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c4c033050 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c65be0740 10 client.27653 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5581e4d9f3f0 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.915+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c0043d0 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2c4c018070 con 0x5581e5126f20 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 10 client.27653 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 10 client.27653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 20 client.27653 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:21.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e51400c0 0x5581e51424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 4 client.27653 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581e5098790 con 0x5581e51400c0 2024-03-28T18:51:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c53400640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e51400c0 0x5581e51424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c4a000640 10 client.27653 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.916+0000 7f2c53400640 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e51400c0 0x5581e51424b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f58a9918e683304 server_cookie=2a57ecd53b9c234d in_seq=0 out_seq=0 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c4a000640 1 -- 172.21.15.119:0/1166609824 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581e5098790 con 0x5581e51400c0 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c4a000640 10 client.27653 handle_command_reply: tid=0 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 client.27653 shutdown 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 msgr2=0x5581e512f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0048f0 tx=0x7f2c4c019a70 comp rx=0 tx=0).stop 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 shutdown_connections 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e51400c0 0x5581e51424b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.917+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e512d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e512cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 1 --2- 172.21.15.119:0/1166609824 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512f8f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c4a000640 0 client.27653 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 >> 172.21.15.119:0/1166609824 conn(0x5581e50ee200 msgr2=0x5581e51236c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 shutdown_connections 2024-03-28T18:51:21.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 1 -- 172.21.15.119:0/1166609824 wait complete. 2024-03-28T18:51:21.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.918+0000 7f2c65be0740 20 client.27653 trim_cache size 0 max 16384 2024-03-28T18:51:21.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:21 smithi196 ceph-mon[36300]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:51:21.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 Processor -- start 2024-03-28T18:51:21.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 -- start start 2024-03-28T18:51:21.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5127c00 0x5581e51234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5130290 0x5581e512f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b39a0 con 0x5581e5130290 2024-03-28T18:51:21.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b5320 con 0x5581e5126f20 2024-03-28T18:51:21.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.935+0000 7f2c65be0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b4000 con 0x5581e5127c00 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c61000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5130290 0x5581e512f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53320/0 (socket says 172.21.15.119:53320) 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 -- 172.21.15.119:0/871403939 learned_addr learned my addr 172.21.15.119:0/871403939 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c53400640 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5127c00 0x5581e51234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 -- 172.21.15.119:0/871403939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5127c00 msgr2=0x5581e51234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5127c00 0x5581e51234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 -- 172.21.15.119:0/871403939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5130290 msgr2=0x5581e512f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5130290 0x5581e512f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 -- 172.21.15.119:0/871403939 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581e4d5f0f0 con 0x5581e5126f20 2024-03-28T18:51:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.936+0000 7f2c5be00640 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0059f0 tx=0x7f2c4c011e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=743ec4e9b8ea6920 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c50c00640 1 -- 172.21.15.119:0/871403939 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c028040 con 0x5581e5126f20 2024-03-28T18:51:21.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c50c00640 1 -- 172.21.15.119:0/871403939 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c01f040 con 0x5581e5126f20 2024-03-28T18:51:21.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c50c00640 1 -- 172.21.15.119:0/871403939 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0042d0 con 0x5581e5126f20 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 -- 172.21.15.119:0/871403939 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 msgr2=0x5581e512c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0059f0 tx=0x7f2c4c011e40 comp rx=0 tx=0).stop 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 -- 172.21.15.119:0/871403939 shutdown_connections 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5130290 0x5581e512f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5127c00 0x5581e51234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 --2- 172.21.15.119:0/871403939 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e512c560 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.937+0000 7f2c65be0740 1 -- 172.21.15.119:0/871403939 >> 172.21.15.119:0/871403939 conn(0x5581e50ee200 msgr2=0x5581e5122dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.938+0000 7f2c65be0740 1 -- 172.21.15.119:0/871403939 shutdown_connections 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.938+0000 7f2c65be0740 1 -- 172.21.15.119:0/871403939 wait complete. 2024-03-28T18:51:21.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.938+0000 7f2c65be0740 1 -- start start 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.938+0000 7f2c65be0740 1 Processor -- start 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5133e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b4cc0 con 0x5581e5127c00 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b4660 con 0x5581e5126f20 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c1b1b80 con 0x5581e5130290 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c65be0740 10 client.0 _ll_register_callbacks cb 0x7f2c54009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:53334/0 (socket says 172.21.15.119:53334) 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 -- 172.21.15.119:0/3246225553 learned_addr learned my addr 172.21.15.119:0/3246225553 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c61000640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c53400640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5133e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 -- 172.21.15.119:0/3246225553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 msgr2=0x5581e5134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 -- 172.21.15.119:0/3246225553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 msgr2=0x5581e5133e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5133e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 -- 172.21.15.119:0/3246225553 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581e519a790 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c5be00640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c011ef0 tx=0x7f2c4c011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9ba3272b7a6de11 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.939+0000 7f2c4a000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c028040 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2c4c005290 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c4c01f040 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c4c004040 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 10 client.27659 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5581e519b9b0 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c4c01f040 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2c4c045030 con 0x5581e5126f20 2024-03-28T18:51:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 10 client.27659 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 10 client.27659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 20 client.27659 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.940+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e519c0e0 0x5581e519e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c65be0740 4 client.27659 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5581e5092ae0 con 0x5581e519c0e0 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c53400640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e519c0e0 0x5581e519e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c53400640 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e519c0e0 0x5581e519e500 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb387eb88737064 server_cookie=e4a03de2160045fa in_seq=0 out_seq=0 2024-03-28T18:51:21.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c4a000640 10 client.27659 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:21 smithi131 ceph-mon[36065]: pgmap v224: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:51:21.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c4a000640 1 -- 172.21.15.119:0/3246225553 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5581e5092ae0 con 0x5581e519c0e0 2024-03-28T18:51:21.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c4a000640 10 client.27659 handle_command_reply: tid=0 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.941+0000 7f2c65be0740 1 client.27659 shutdown 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 msgr2=0x5581e51338d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c011ef0 tx=0x7f2c4c011d10 comp rx=0 tx=0).stop 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 shutdown_connections 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5581e519c0e0 0x5581e519e500 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5581e5130290 0x5581e5134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5581e5127c00 0x5581e5133e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c4a000640 0 client.27659 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 --2- 172.21.15.119:0/3246225553 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5581e5126f20 0x5581e51338d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 >> 172.21.15.119:0/3246225553 conn(0x5581e50ee200 msgr2=0x5581e5120a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 shutdown_connections 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 1 -- 172.21.15.119:0/3246225553 wait complete. 2024-03-28T18:51:21.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.942+0000 7f2c65be0740 20 client.27659 trim_cache size 0 max 16384 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "ec02546b-bf8f-4495-bd91-9c49d95291e1": { 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "ec02546b-bf8f-4495-bd91-9c49d95291e1", 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:51:21.947 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 msgr2=0x7f2c400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 0x7f2c400a4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2c54009fd0 tx=0x7f2c54009500 comp rx=0 tx=0).stop 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 msgr2=0x7f2c5c19ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c19ea10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2c4402c210 tx=0x7f2c44005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 shutdown_connections 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2c400a29f0 0x7f2c400a4eb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c5c112640 0x7f2c5c1a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2c5c074360 0x7f2c5c19ea10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 --2- 172.21.15.119:0/274329840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c5c073a40 0x7f2c5c19e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:21.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 >> 172.21.15.119:0/274329840 conn(0x7f2c5c078aa0 msgr2=0x7f2c5c10e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:21.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 shutdown_connections 2024-03-28T18:51:21.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:21.943+0000 7f2c63000640 1 -- 172.21.15.119:0/274329840 wait complete. 2024-03-28T18:51:21.961 INFO:tasks.cephfs.filesystem:scrub status for tag:ec02546b-bf8f-4495-bd91-9c49d95291e1 - {'path': '/', 'tag': 'ec02546b-bf8f-4495-bd91-9c49d95291e1', 'options': 'recursive,force'} 2024-03-28T18:51:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:22 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:51:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:22 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:22 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:51:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:22 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:22 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:22.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:22 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:51:22.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:22 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:22 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:51:22.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:22 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:22 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:22 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:51:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:22 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:22 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:51:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:22 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:22 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:51:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:23 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:23 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:23 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:51:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:23 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:51:23.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:23 smithi119 ceph-mon[31813]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:51:23.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:23 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:23.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:23 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:23.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:23 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:51:23.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:23 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:51:23.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:23 smithi196 ceph-mon[36300]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:51:23.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:23 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:51:23.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:23 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:51:23.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:23 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:51:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:23 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:51:23.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:23 smithi131 ceph-mon[36065]: pgmap v225: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:51:26.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:26 smithi196 ceph-mon[36300]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:51:26.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:26 smithi131 ceph-mon[36065]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:51:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:26 smithi119 ceph-mon[31813]: pgmap v226: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:51:28.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:28 smithi131 ceph-mon[36065]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:51:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:28 smithi119 ceph-mon[31813]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:51:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:28 smithi196 ceph-mon[36300]: pgmap v227: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-28T18:51:30.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:30 smithi131 ceph-mon[36065]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-03-28T18:51:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:30 smithi119 ceph-mon[31813]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-03-28T18:51:30.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:30 smithi196 ceph-mon[36300]: pgmap v228: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-03-28T18:51:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:32 smithi119 ceph-mon[31813]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:32.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:32 smithi196 ceph-mon[36300]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:32.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:32 smithi131 ceph-mon[36065]: pgmap v229: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:34 smithi119 ceph-mon[31813]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:34.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:34 smithi196 ceph-mon[36300]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:34 smithi131 ceph-mon[36065]: pgmap v230: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:51:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:36 smithi119 ceph-mon[31813]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:36.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:36 smithi196 ceph-mon[36300]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:36 smithi131 ceph-mon[36065]: pgmap v231: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:51:38.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:38 smithi196 ceph-mon[36300]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:38 smithi119 ceph-mon[31813]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:38 smithi131 ceph-mon[36065]: pgmap v232: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:40 smithi119 ceph-mon[31813]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:40 smithi196 ceph-mon[36300]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:40 smithi131 ceph-mon[36065]: pgmap v233: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:42 smithi119 ceph-mon[31813]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:42 smithi196 ceph-mon[36300]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:42.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:42 smithi131 ceph-mon[36065]: pgmap v234: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:44 smithi119 ceph-mon[31813]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:44 smithi196 ceph-mon[36300]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:44 smithi131 ceph-mon[36065]: pgmap v235: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:46 smithi119 ceph-mon[31813]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:46 smithi196 ceph-mon[36300]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:46 smithi131 ceph-mon[36065]: pgmap v236: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:48 smithi119 ceph-mon[31813]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:48 smithi196 ceph-mon[36300]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:48 smithi131 ceph-mon[36065]: pgmap v237: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:50 smithi119 ceph-mon[31813]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:50.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:50 smithi196 ceph-mon[36300]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:50 smithi131 ceph-mon[36065]: pgmap v238: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:51.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:51:51.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:51:52.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 -- 172.21.15.119:0/2536693974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708103840 msgr2=0x7f2708103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 --2- 172.21.15.119:0/2536693974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708103840 0x7f2708103cc0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009930 tx=0x7f26f002bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 -- 172.21.15.119:0/2536693974 shutdown_connections 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 --2- 172.21.15.119:0/2536693974 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708104200 0x7f2708110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 --2- 172.21.15.119:0/2536693974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708103840 0x7f2708103cc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 --2- 172.21.15.119:0/2536693974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 0x7f2708109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 -- 172.21.15.119:0/2536693974 >> 172.21.15.119:0/2536693974 conn(0x7f27080ff2f0 msgr2=0x7f2708101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.189+0000 7f270f000640 1 -- 172.21.15.119:0/2536693974 shutdown_connections 2024-03-28T18:51:52.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.190+0000 7f270f000640 1 -- 172.21.15.119:0/2536693974 wait complete. 2024-03-28T18:51:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.190+0000 7f270f000640 1 Processor -- start 2024-03-28T18:51:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 -- start start 2024-03-28T18:51:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 0x7f270819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 0x7f270819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2708113cb0 con 0x7f2708109850 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2708113b30 con 0x7f2708103840 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270f000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2708113e30 con 0x7f2708104200 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49292/0 (socket says 172.21.15.119:49292) 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f2707e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 0x7f270819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f26ffe00640 1 -- 172.21.15.119:0/2674115335 learned_addr learned my addr 172.21.15.119:0/2674115335 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.191+0000 7f270d000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 0x7f270819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 -- 172.21.15.119:0/2674115335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 msgr2=0x7f270819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 -- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 msgr2=0x7f270819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 0x7f270819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 -- 172.21.15.119:0/2674115335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270819f180 con 0x7f2708103840 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f26ffe00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:52.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2707e00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 0x7f270819a190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f26f800e9f0 tx=0x7f26f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2705600640 1 -- 172.21.15.119:0/2674115335 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f800ccc0 con 0x7f2708103840 2024-03-28T18:51:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270819f470 con 0x7f2708103840 2024-03-28T18:51:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2705600640 1 -- 172.21.15.119:0/2674115335 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26f8010910 con 0x7f2708103840 2024-03-28T18:51:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f2705600640 1 -- 172.21.15.119:0/2674115335 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8018960 con 0x7f2708103840 2024-03-28T18:51:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.192+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27081a6d00 con 0x7f2708103840 2024-03-28T18:51:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.195+0000 7f2705600640 1 -- 172.21.15.119:0/2674115335 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f26f8010ac0 con 0x7f2708103840 2024-03-28T18:51:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 Processor -- start 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 -- start start 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ae680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a8007250 con 0x55e1a80ac290 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a7fafa30 con 0x55e1a80a2f20 2024-03-28T18:51:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.196+0000 7f2711d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a80032f0 con 0x55e1a80a3c00 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49302/0 (socket says 172.21.15.119:49302) 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 -- 172.21.15.119:0/178142908 learned_addr learned my addr 172.21.15.119:0/178142908 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f2707e00640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f26ffe00640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 -- 172.21.15.119:0/178142908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 msgr2=0x55e1a80a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 -- 172.21.15.119:0/178142908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 msgr2=0x55e1a80ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 -- 172.21.15.119:0/178142908 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a7f9a690 con 0x55e1a80a3c00 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f26ffe00640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f270d000640 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2700009fd0 tx=0x7f27000118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f7f5f7128b279e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f2705600640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 0x7f26ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f26fd600640 1 -- 172.21.15.119:0/178142908 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700004d20 con 0x55e1a80a3c00 2024-03-28T18:51:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.197+0000 7f26fd600640 1 -- 172.21.15.119:0/178142908 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2700004050 con 0x55e1a80a3c00 2024-03-28T18:51:52.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f26fd600640 1 -- 172.21.15.119:0/178142908 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2700005260 con 0x55e1a80a3c00 2024-03-28T18:51:52.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2707e00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 0x7f26ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2705600640 1 -- 172.21.15.119:0/2674115335 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f26f80cf210 con 0x7f2708103840 2024-03-28T18:51:52.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 -- 172.21.15.119:0/178142908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 msgr2=0x55e1a80a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2707e00640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 0x7f26ec0a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26f80cc980 tx=0x7f26f80d2d60 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2700009fd0 tx=0x7f27000118d0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 -- 172.21.15.119:0/178142908 shutdown_connections 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 --2- 172.21.15.119:0/178142908 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80a2950 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.198+0000 7f2711d6a740 1 -- 172.21.15.119:0/178142908 >> 172.21.15.119:0/178142908 conn(0x55e1a806a200 msgr2=0x55e1a809d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.199+0000 7f2711d6a740 1 -- 172.21.15.119:0/178142908 shutdown_connections 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.199+0000 7f2711d6a740 1 -- 172.21.15.119:0/178142908 wait complete. 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.199+0000 7f2711d6a740 1 -- start start 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.199+0000 7f2711d6a740 1 Processor -- start 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 0x55e1a80ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 0x55e1a80a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80ac290 0x55e1a80a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a80af100 con 0x55e1a80a3c00 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a7fc2e80 con 0x55e1a80ac290 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a80af280 con 0x55e1a80a2f20 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f2711d6a740 10 client.0 _ll_register_callbacks cb 0x55e1a7fd5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 0x55e1a80ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 0x55e1a80a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 0x55e1a80ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49318/0 (socket says 172.21.15.119:49318) 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f270d000640 1 -- 172.21.15.119:0/3999620068 learned_addr learned my addr 172.21.15.119:0/3999620068 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f270d000640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80ac290 0x55e1a80a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 -- 172.21.15.119:0/3999620068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 msgr2=0x55e1a80ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 0x55e1a80ab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 -- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80ac290 msgr2=0x55e1a80a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80ac290 0x55e1a80a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.200+0000 7f26ffe00640 1 -- 172.21.15.119:0/3999620068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1a7de57d0 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f26ffe00640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 0x55e1a80a8d60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f26f0006010 tx=0x7f26f0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cab7b1669948e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f26f6000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0004440 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f26f002c880 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2700004050 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26f002ca80 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f2711d6a740 10 client.17439 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.201+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e1a7d1b3f0 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2700004050 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f26f0050030 con 0x55e1a80a3c00 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 10 client.17439 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 20 client.17439 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a80bc250 0x55e1a80be670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 4 client.17439 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1a8014790 con 0x55e1a80bc250 2024-03-28T18:51:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.202+0000 7f270d000640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a80bc250 0x55e1a80be670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.203+0000 7f270d000640 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a80bc250 0x55e1a80be670 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5692bd8b9d6a8c39 server_cookie=b2a8c37880d1eec8 in_seq=0 out_seq=0 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.203+0000 7f26f6000640 10 client.17439 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f26f6000640 1 -- 172.21.15.119:0/3999620068 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e1a8014790 con 0x55e1a80bc250 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f26f6000640 10 client.17439 handle_command_reply: tid=0 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 client.17439 shutdown 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 msgr2=0x55e1a80a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 0x55e1a80a8d60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f26f0006010 tx=0x7f26f0004040 comp rx=0 tx=0).stop 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 shutdown_connections 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a80bc250 0x55e1a80be670 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80ac290 0x55e1a80a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a3c00 0x55e1a80a8d60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 --2- 172.21.15.119:0/3999620068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a2f20 0x55e1a80ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f26f6000640 0 client.17439 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 >> 172.21.15.119:0/3999620068 conn(0x55e1a806a200 msgr2=0x55e1a809f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 shutdown_connections 2024-03-28T18:51:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 1 -- 172.21.15.119:0/3999620068 wait complete. 2024-03-28T18:51:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.204+0000 7f2711d6a740 20 client.17439 trim_cache size 0 max 16384 2024-03-28T18:51:52.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.221+0000 7f2711d6a740 1 Processor -- start 2024-03-28T18:51:52.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 -- start start 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a2f20 0x55e1a80a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a3c00 0x55e1a80ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081af640 con 0x55e1a80a2f20 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081b0fc0 con 0x55e1a80a3c00 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f2711d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081afca0 con 0x55e1a80ac290 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f270d000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a2f20 0x55e1a80a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f270d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49328/0 (socket says 172.21.15.119:49328) 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f270d000640 1 -- 172.21.15.119:0/4020185478 learned_addr learned my addr 172.21.15.119:0/4020185478 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.222+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a3c00 0x55e1a80ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 -- 172.21.15.119:0/4020185478 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a3c00 msgr2=0x55e1a80ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a3c00 0x55e1a80ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 -- 172.21.15.119:0/4020185478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a2f20 msgr2=0x55e1a80a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a2f20 0x55e1a80a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 -- 172.21.15.119:0/4020185478 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a809db80 con 0x55e1a80ac290 2024-03-28T18:51:52.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f270d000640 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2700009f70 tx=0x7f270002f4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a54286a8d0487b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f26fcc00640 1 -- 172.21.15.119:0/4020185478 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270002f720 con 0x55e1a80ac290 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f26fcc00640 1 -- 172.21.15.119:0/4020185478 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f270002fda0 con 0x55e1a80ac290 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.223+0000 7f26fcc00640 1 -- 172.21.15.119:0/4020185478 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270000fca0 con 0x55e1a80ac290 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 -- 172.21.15.119:0/4020185478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 msgr2=0x55e1a80ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2700009f70 tx=0x7f270002f4b0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 -- 172.21.15.119:0/4020185478 shutdown_connections 2024-03-28T18:51:52.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80ac290 0x55e1a80ab9d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a3c00 0x55e1a80ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 --2- 172.21.15.119:0/4020185478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80a2f20 0x55e1a80a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 -- 172.21.15.119:0/4020185478 >> 172.21.15.119:0/4020185478 conn(0x55e1a806a200 msgr2=0x55e1a80a7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 -- 172.21.15.119:0/4020185478 shutdown_connections 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.224+0000 7f2711d6a740 1 -- 172.21.15.119:0/4020185478 wait complete. 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 -- start start 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 Processor -- start 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081b0960 con 0x55e1a80ac290 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081b0300 con 0x55e1a80a2f20 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27081ad820 con 0x55e1a80a3c00 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f2711d6a740 10 client.0 _ll_register_callbacks cb 0x7f27000093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:52.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49344/0 (socket says 172.21.15.119:49344) 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.225+0000 7f26ffe00640 1 -- 172.21.15.119:0/115757600 learned_addr learned my addr 172.21.15.119:0/115757600 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f270d000640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f270d000640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 -- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 msgr2=0x55e1a80a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 -- 172.21.15.119:0/115757600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 msgr2=0x55e1a80ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ad6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 -- 172.21.15.119:0/115757600 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1a7cdb0f0 con 0x55e1a80a3c00 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f270d000640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26ffe00640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55e1a80aa330 tx=0x7f26f0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba3cbc6269f7a832 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26f6000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0005850 con 0x55e1a80a3c00 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f26f002cc20 con 0x55e1a80a3c00 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.226+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f270002fda0 con 0x55e1a80a3c00 2024-03-28T18:51:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26f0003c00 con 0x55e1a80a3c00 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 10 client.27652 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e1a8118620 con 0x55e1a80a3c00 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f270002fda0 con 0x55e1a80a3c00 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f26f005a070 con 0x55e1a80a3c00 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 10 client.27652 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 10 client.27652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 20 client.27652 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a8118d70 0x55e1a811b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 4 client.27652 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e1a800eae0 con 0x55e1a8118d70 2024-03-28T18:51:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.227+0000 7f270d000640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a8118d70 0x55e1a811b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f270d000640 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a8118d70 0x55e1a811b190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ba7caaddb37fe51 server_cookie=9faa2b293b4925d1 in_seq=0 out_seq=0 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f26f6000640 10 client.27652 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f26f6000640 1 -- 172.21.15.119:0/115757600 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e1a800eae0 con 0x55e1a8118d70 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f26f6000640 10 client.27652 handle_command_reply: tid=0 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 client.27652 shutdown 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 msgr2=0x55e1a80ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55e1a80aa330 tx=0x7f26f0009260 comp rx=0 tx=0).stop 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 shutdown_connections 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e1a8118d70 0x55e1a811b190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e1a80ac290 0x55e1a80ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1a80a3c00 0x55e1a80ad190 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f26f6000640 0 client.27652 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 --2- 172.21.15.119:0/115757600 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e1a80a2f20 0x55e1a80a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 >> 172.21.15.119:0/115757600 conn(0x55e1a806a200 msgr2=0x55e1a809c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 shutdown_connections 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.228+0000 7f2711d6a740 1 -- 172.21.15.119:0/115757600 wait complete. 2024-03-28T18:51:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f2711d6a740 20 client.27652 trim_cache size 0 max 16384 2024-03-28T18:51:52.233 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:52.233 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:51:52.233 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:51:52.233 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 msgr2=0x7f26ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 0x7f26ec0a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26f80cc980 tx=0x7f26f80d2d60 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 msgr2=0x7f270819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 0x7f270819a190 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f26f800e9f0 tx=0x7f26f800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 shutdown_connections 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f26ec0a2970 0x7f26ec0a4e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2708109850 0x7f270819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2708104200 0x7f270819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 --2- 172.21.15.119:0/2674115335 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2708103840 0x7f270819a190 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 >> 172.21.15.119:0/2674115335 conn(0x7f27080ff2f0 msgr2=0x7f27080ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 shutdown_connections 2024-03-28T18:51:52.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.229+0000 7f270f000640 1 -- 172.21.15.119:0/2674115335 wait complete. 2024-03-28T18:51:52.247 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:51:52.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.450+0000 7f9209a00640 1 -- 172.21.15.119:0/3505246714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f920406b8e0 msgr2=0x7f9204116230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.450+0000 7f9209a00640 1 --2- 172.21.15.119:0/3505246714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f920406b8e0 0x7f9204116230 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009900 tx=0x7f91ec02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 -- 172.21.15.119:0/3505246714 shutdown_connections 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 --2- 172.21.15.119:0/3505246714 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9204116770 0x7f9204118b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 --2- 172.21.15.119:0/3505246714 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f920406b8e0 0x7f9204116230 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 --2- 172.21.15.119:0/3505246714 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406afc0 0x7f920406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 -- 172.21.15.119:0/3505246714 >> 172.21.15.119:0/3505246714 conn(0x7f9204072c40 msgr2=0x7f9204073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 -- 172.21.15.119:0/3505246714 shutdown_connections 2024-03-28T18:51:52.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.451+0000 7f9209a00640 1 -- 172.21.15.119:0/3505246714 wait complete. 2024-03-28T18:51:52.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.452+0000 7f9209a00640 1 Processor -- start 2024-03-28T18:51:52.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.452+0000 7f9209a00640 1 -- start start 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.452+0000 7f9209a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.452+0000 7f9209a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 0x7f92041a8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9209a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9204116770 0x7f92041a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9209a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920411c520 con 0x7f920406b8e0 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9209a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920411c3a0 con 0x7f920406afc0 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9209a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920411c6a0 con 0x7f9204116770 2024-03-28T18:51:52.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51228/0 (socket says 172.21.15.119:51228) 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 -- 172.21.15.119:0/870642307 learned_addr learned my addr 172.21.15.119:0/870642307 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9203400640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9204116770 0x7f92041a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f91fa000640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 0x7f92041a8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 -- 172.21.15.119:0/870642307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9204116770 msgr2=0x7f92041a28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9204116770 0x7f92041a28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 -- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 msgr2=0x7f92041a8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 0x7f92041a8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 -- 172.21.15.119:0/870642307 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92041a3060 con 0x7f920406afc0 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f91fa000640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 0x7f92041a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:52.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.453+0000 7f9202a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f91f400ebc0 tx=0x7f91f400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.454+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4010630 con 0x7f920406afc0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.454+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91f4010cb0 con 0x7f920406afc0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.454+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f4018bd0 con 0x7f920406afc0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.454+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92041a3350 con 0x7f920406afc0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.454+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92041af5a0 con 0x7f920406afc0 2024-03-28T18:51:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.455+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9204068490 con 0x7f920406afc0 2024-03-28T18:51:52.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.457+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f91f40107d0 con 0x7f920406afc0 2024-03-28T18:51:52.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.458+0000 7f91fbe00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 0x7f91e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.458+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f91f40cf5f0 con 0x7f920406afc0 2024-03-28T18:51:52.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.458+0000 7f91fa000640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 0x7f91e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.459+0000 7f91fa000640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 0x7f91e80a4e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006010 tx=0x7f91ec005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.467+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f91f409c260 con 0x7f920406afc0 2024-03-28T18:51:52.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.706+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92041a3dc0 con 0x7f920406afc0 2024-03-28T18:51:52.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.708+0000 7f91fbe00640 1 -- 172.21.15.119:0/870642307 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f91f409bb60 con 0x7f920406afc0 2024-03-28T18:51:52.708 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:51:52.708 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:51:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.710+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 msgr2=0x7f91e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.710+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 0x7f91e80a4e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91ec006010 tx=0x7f91ec005a20 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.710+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 msgr2=0x7f92041a86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f91f400ebc0 tx=0x7f91f400c6a0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 shutdown_connections 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f91e80a2970 0x7f91e80a4e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9204116770 0x7f92041a28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f920406b8e0 0x7f92041a8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 --2- 172.21.15.119:0/870642307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f920406afc0 0x7f92041a86e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 >> 172.21.15.119:0/870642307 conn(0x7f9204072c40 msgr2=0x7f920406f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 shutdown_connections 2024-03-28T18:51:52.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.711+0000 7f9209a00640 1 -- 172.21.15.119:0/870642307 wait complete. 2024-03-28T18:51:52.726 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:52 smithi119 ceph-mon[31813]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:52.730 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:52.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:51:52.746 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:52 smithi131 ceph-mon[36065]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:52 smithi196 ceph-mon[36300]: pgmap v239: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:52.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 -- 172.21.15.119:0/2310446796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec104200 msgr2=0x7f75ec110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 --2- 172.21.15.119:0/2310446796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec104200 0x7f75ec110710 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009cd0 tx=0x7f75e402c0f0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 -- 172.21.15.119:0/2310446796 shutdown_connections 2024-03-28T18:51:52.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 --2- 172.21.15.119:0/2310446796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec104200 0x7f75ec110710 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 --2- 172.21.15.119:0/2310446796 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 --2- 172.21.15.119:0/2310446796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec109850 0x7f75ec109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 -- 172.21.15.119:0/2310446796 >> 172.21.15.119:0/2310446796 conn(0x7f75ec0ff2f0 msgr2=0x7f75ec101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 -- 172.21.15.119:0/2310446796 shutdown_connections 2024-03-28T18:51:52.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.937+0000 7f75f2a00640 1 -- 172.21.15.119:0/2310446796 wait complete. 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.938+0000 7f75f2a00640 1 Processor -- start 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.938+0000 7f75f2a00640 1 -- start start 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec104200 0x7f75ec19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 0x7f75ec19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec113c60 con 0x7f75ec104200 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec113ae0 con 0x7f75ec103840 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f2a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec113de0 con 0x7f75ec109850 2024-03-28T18:51:52.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51248/0 (socket says 172.21.15.119:51248) 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75ebe00640 1 -- 172.21.15.119:0/1371809000 learned_addr learned my addr 172.21.15.119:0/1371809000 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 0x7f75ec19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75eb400640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec104200 0x7f75ec19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 -- 172.21.15.119:0/1371809000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 msgr2=0x7f75ec19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 -- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec104200 msgr2=0x7f75ec19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec104200 0x7f75ec19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75f0a00640 1 -- 172.21.15.119:0/1371809000 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec19f140 con 0x7f75ec109850 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.939+0000 7f75ebe00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75f0a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 0x7f75ec19ea10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f75e4037040 tx=0x7f75e4032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75e8c00640 1 -- 172.21.15.119:0/1371809000 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4041020 con 0x7f75ec109850 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75e8c00640 1 -- 172.21.15.119:0/1371809000 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e4034cd0 con 0x7f75ec109850 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75e8c00640 1 -- 172.21.15.119:0/1371809000 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e4039040 con 0x7f75ec109850 2024-03-28T18:51:52.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ec19f3d0 con 0x7f75ec109850 2024-03-28T18:51:52.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.940+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec1a6cc0 con 0x7f75ec109850 2024-03-28T18:51:52.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.943+0000 7f75f579e740 1 Processor -- start 2024-03-28T18:51:52.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.943+0000 7f75e8c00640 1 -- 172.21.15.119:0/1371809000 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f75e4044020 con 0x7f75ec109850 2024-03-28T18:51:52.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.943+0000 7f75f579e740 1 -- start start 2024-03-28T18:51:52.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cb7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874c1b2c0 con 0x55e874cc0c00 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874bc3c90 con 0x55e874cb8080 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75f579e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874c17360 con 0x55e874cb76d0 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51258/0 (socket says 172.21.15.119:51258) 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 -- 172.21.15.119:0/3425735719 learned_addr learned my addr 172.21.15.119:0/3425735719 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75eb400640 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cb7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75eb400640 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 -- 172.21.15.119:0/3425735719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 msgr2=0x55e874cb7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cb7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 -- 172.21.15.119:0/3425735719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 msgr2=0x55e874cc3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.944+0000 7f75ebe00640 1 -- 172.21.15.119:0/3425735719 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e874be8080 con 0x55e874cb8080 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0115c0 tx=0x7f75dc011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e6c647f42c4b967 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75e8c00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 0x7f75d00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75e1200640 1 -- 172.21.15.119:0/3425735719 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00f660 con 0x55e874cb8080 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75e1200640 1 -- 172.21.15.119:0/3425735719 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75dc00fce0 con 0x55e874cb8080 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75e1200640 1 -- 172.21.15.119:0/3425735719 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc019db0 con 0x55e874cb8080 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75e8c00640 1 -- 172.21.15.119:0/1371809000 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75e40f0910 con 0x7f75ec109850 2024-03-28T18:51:52.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.945+0000 7f75eb400640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 0x7f75d00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 -- 172.21.15.119:0/3425735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 msgr2=0x55e874cc0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0115c0 tx=0x7f75dc011a80 comp rx=0 tx=0).stop 2024-03-28T18:51:52.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75eb400640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 0x7f75d00a4eb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009fd0 tx=0x7f75d8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 -- 172.21.15.119:0/3425735719 shutdown_connections 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc0630 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 --2- 172.21.15.119:0/3425735719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cb7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 -- 172.21.15.119:0/3425735719 >> 172.21.15.119:0/3425735719 conn(0x55e874cb1da0 msgr2=0x55e874cb2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 -- 172.21.15.119:0/3425735719 shutdown_connections 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.946+0000 7f75f579e740 1 -- 172.21.15.119:0/3425735719 wait complete. 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 -- start start 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 Processor -- start 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb8080 0x55e874cbd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 0x55e874cbde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874cc3b70 con 0x55e874cb8080 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874bd70d0 con 0x55e874cb76d0 2024-03-28T18:51:52.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874cc3cf0 con 0x55e874cc0c00 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75f579e740 10 client.0 _ll_register_callbacks cb 0x55e874bdbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.947+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51268/0 (socket says 172.21.15.119:51268) 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75f0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 0x55e874cbde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 -- 172.21.15.119:0/3479661514 learned_addr learned my addr 172.21.15.119:0/3479661514 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75eb400640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb8080 0x55e874cbd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 -- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 msgr2=0x55e874cbde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 0x55e874cbde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 -- 172.21.15.119:0/3479661514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb8080 msgr2=0x55e874cbd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb8080 0x55e874cbd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 -- 172.21.15.119:0/3479661514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e87492f3f0 con 0x55e874cb76d0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75f0a00640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 0x55e874cbde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75ebe00640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75dc011f80 tx=0x7f75dc00f890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95cfa2c80e53cb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75d5600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:52.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.948+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00fe90 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75dc005040 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75dc00fce0 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75dc030050 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 10 client.27695 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e8749f97d0 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75dc00fce0 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75dc0047e0 con 0x55e874cb76d0 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 10 client.27695 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 10 client.27695 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 20 client.27695 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874cd0a00 0x55e874cd2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 4 client.27695 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.949+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e874c28800 con 0x55e874cd0a00 2024-03-28T18:51:52.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.950+0000 7f75eb400640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874cd0a00 0x55e874cd2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.950+0000 7f75eb400640 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874cd0a00 0x55e874cd2df0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9cfd2516eb29ea39 server_cookie=d068b822d4828962 in_seq=0 out_seq=0 2024-03-28T18:51:52.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.950+0000 7f75d5600640 10 client.27695 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75d5600640 1 -- 172.21.15.119:0/3479661514 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e874c28800 con 0x55e874cd0a00 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75d5600640 10 client.27695 handle_command_reply: tid=0 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 client.27695 shutdown 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 msgr2=0x55e874cc0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75dc011f80 tx=0x7f75dc00f890 comp rx=0 tx=0).stop 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 shutdown_connections 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874cd0a00 0x55e874cd2df0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cc0c00 0x55e874cbde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb8080 0x55e874cbd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75d5600640 0 client.27695 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 --2- 172.21.15.119:0/3479661514 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb76d0 0x55e874cc0500 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.951+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 >> 172.21.15.119:0/3479661514 conn(0x55e874cb1da0 msgr2=0x55e874cb3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.952+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 shutdown_connections 2024-03-28T18:51:52.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.952+0000 7f75f579e740 1 -- 172.21.15.119:0/3479661514 wait complete. 2024-03-28T18:51:52.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.952+0000 7f75f579e740 20 client.27695 trim_cache size 0 max 16384 2024-03-28T18:51:52.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.965+0000 7f75f579e740 1 Processor -- start 2024-03-28T18:51:52.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.965+0000 7f75f579e740 1 -- start start 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb76d0 0x55e874cbffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 0x55e874cbab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1af5d0 con 0x55e874cb76d0 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1b0f50 con 0x55e874cc0c00 2024-03-28T18:51:52.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f579e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1afc30 con 0x55e874cb8080 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb76d0 0x55e874cbffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 0x55e874cbab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49418/0 (socket says 172.21.15.119:49418) 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 -- 172.21.15.119:0/223270827 learned_addr learned my addr 172.21.15.119:0/223270827 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 -- 172.21.15.119:0/223270827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 msgr2=0x55e874cbab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 0x55e874cbab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 -- 172.21.15.119:0/223270827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb76d0 msgr2=0x55e874cbffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb76d0 0x55e874cbffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75eb400640 1 -- 172.21.15.119:0/223270827 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e874cc3a50 con 0x55e874cb8080 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.966+0000 7f75f0a00640 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 0x55e874cbab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75eb400640 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009730 tx=0x7f75d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5c55ff06b50857 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75d7e00640 1 -- 172.21.15.119:0/223270827 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8036070 con 0x55e874cb8080 2024-03-28T18:51:52.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75d7e00640 1 -- 172.21.15.119:0/223270827 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75d8030dc0 con 0x55e874cb8080 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75d7e00640 1 -- 172.21.15.119:0/223270827 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d802d670 con 0x55e874cb8080 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 -- 172.21.15.119:0/223270827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 msgr2=0x55e874cc0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009730 tx=0x7f75d80046c0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 -- 172.21.15.119:0/223270827 shutdown_connections 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cc0c00 0x55e874cbab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb8080 0x55e874cc0500 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 --2- 172.21.15.119:0/223270827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cb76d0 0x55e874cbffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.967+0000 7f75f579e740 1 -- 172.21.15.119:0/223270827 >> 172.21.15.119:0/223270827 conn(0x55e874cb1da0 msgr2=0x55e874cb0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.968+0000 7f75f579e740 1 -- 172.21.15.119:0/223270827 shutdown_connections 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.968+0000 7f75f579e740 1 -- 172.21.15.119:0/223270827 wait complete. 2024-03-28T18:51:52.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.968+0000 7f75f579e740 1 -- start start 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.968+0000 7f75f579e740 1 Processor -- start 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1b08f0 con 0x55e874cc0c00 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1b0290 con 0x55e874cb8080 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec1ad7b0 con 0x55e874cb76d0 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f579e740 10 client.0 _ll_register_callbacks cb 0x7f75d8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75ebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49420/0 (socket says 172.21.15.119:49420) 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75ebe00640 1 -- 172.21.15.119:0/846237221 learned_addr learned my addr 172.21.15.119:0/846237221 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f0a00640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 -- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 msgr2=0x55e874cc8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 -- 172.21.15.119:0/846237221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 msgr2=0x55e874cc6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 -- 172.21.15.119:0/846237221 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e874cc7700 con 0x55e874cb8080 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75ebe00640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75f0a00640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75eb400640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc8ea0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f75d8034040 tx=0x7f75d8004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14206e28e9bf2926 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.969+0000 7f75d5600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d8036070 con 0x55e874cb8080 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75d8030dc0 con 0x55e874cb8080 2024-03-28T18:51:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f75d8011640 con 0x55e874cb8080 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75d8042050 con 0x55e874cb8080 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 10 client.27701 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e874d2cd60 con 0x55e874cb8080 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75d8030dc0 con 0x55e874cb8080 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f75d8004730 con 0x55e874cb8080 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 10 client.27701 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 10 client.27701 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 20 client.27701 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874d2d3b0 0x55e874d2f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.970+0000 7f75f579e740 4 client.27701 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e874c22b50 con 0x55e874d2d3b0 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75ebe00640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874d2d3b0 0x55e874d2f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75ebe00640 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874d2d3b0 0x55e874d2f7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a69b7ab5b8c88e1 server_cookie=81178708dae8b1ea in_seq=0 out_seq=0 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75d5600640 10 client.27701 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:52.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75d5600640 1 -- 172.21.15.119:0/846237221 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e874c22b50 con 0x55e874d2d3b0 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75d5600640 10 client.27701 handle_command_reply: tid=0 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.971+0000 7f75f579e740 1 client.27701 shutdown 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 msgr2=0x55e874cc8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc8ea0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f75d8034040 tx=0x7f75d8004480 comp rx=0 tx=0).stop 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 shutdown_connections 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55e874d2d3b0 0x55e874d2f7d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75d5600640 0 client.27701 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e874cc0c00 0x55e874cc6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e874cb8080 0x55e874cc8ea0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 --2- 172.21.15.119:0/846237221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e874cb76d0 0x55e874cc8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 >> 172.21.15.119:0/846237221 conn(0x55e874cb1da0 msgr2=0x55e874cb0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 shutdown_connections 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 1 -- 172.21.15.119:0/846237221 wait complete. 2024-03-28T18:51:52.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.972+0000 7f75f579e740 20 client.27701 trim_cache size 0 max 16384 2024-03-28T18:51:52.977 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:52.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 msgr2=0x7f75d00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 0x7f75d00a4eb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009fd0 tx=0x7f75d8005f90 comp rx=0 tx=0).stop 2024-03-28T18:51:52.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 msgr2=0x7f75ec19ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 0x7f75ec19ea10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f75e4037040 tx=0x7f75e4032ea0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 shutdown_connections 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f75d00a29f0 0x7f75d00a4eb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75ec109850 0x7f75ec19ea10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75ec104200 0x7f75ec19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 --2- 172.21.15.119:0/1371809000 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f75ec103840 0x7f75ec19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 >> 172.21.15.119:0/1371809000 conn(0x7f75ec0ff2f0 msgr2=0x7f75ec0ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 shutdown_connections 2024-03-28T18:51:52.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:52.973+0000 7f75f2a00640 1 -- 172.21.15.119:0/1371809000 wait complete. 2024-03-28T18:51:52.992 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:52.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:51:53.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 -- 172.21.15.119:0/3162880005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88108800 msgr2=0x7f5b8810c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/3162880005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88108800 0x7f5b8810c7c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009900 tx=0x7f5b7002bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:53.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 -- 172.21.15.119:0/3162880005 shutdown_connections 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/3162880005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b88114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/3162880005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88108800 0x7f5b8810c7c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/3162880005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b88107e50 0x7f5b88108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 -- 172.21.15.119:0/3162880005 >> 172.21.15.119:0/3162880005 conn(0x7f5b880774c0 msgr2=0x7f5b881064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 -- 172.21.15.119:0/3162880005 shutdown_connections 2024-03-28T18:51:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.201+0000 7f5b8ca00640 1 -- 172.21.15.119:0/3162880005 wait complete. 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.202+0000 7f5b8ca00640 1 Processor -- start 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 -- start start 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b8819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b881a2ce0 0x7f5b881a3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88117fa0 con 0x7f5b8810cea0 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88117e20 con 0x7f5b88107e50 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b8ca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88118120 con 0x7f5b881a2ce0 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b86800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b86800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51296/0 (socket says 172.21.15.119:51296) 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b86800640 1 -- 172.21.15.119:0/1569902172 learned_addr learned my addr 172.21.15.119:0/1569902172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b87200640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b881a2ce0 0x7f5b881a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.203+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b8819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 -- 172.21.15.119:0/1569902172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b881a2ce0 msgr2=0x7f5b881a3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b881a2ce0 0x7f5b881a3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 -- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 msgr2=0x7f5b8819e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b8819e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 -- 172.21.15.119:0/1569902172 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b881a37d0 con 0x7f5b88107e50 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b8819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:53.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b86800640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9f0 tx=0x7f5b7800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b7fe00640 1 -- 172.21.15.119:0/1569902172 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7800cd80 con 0x7f5b88107e50 2024-03-28T18:51:53.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881a3ac0 con 0x7f5b88107e50 2024-03-28T18:51:53.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b7fe00640 1 -- 172.21.15.119:0/1569902172 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b78010910 con 0x7f5b88107e50 2024-03-28T18:51:53.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b7fe00640 1 -- 172.21.15.119:0/1569902172 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780188f0 con 0x7f5b88107e50 2024-03-28T18:51:53.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.204+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881ab270 con 0x7f5b88107e50 2024-03-28T18:51:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.207+0000 7f5b7fe00640 1 -- 172.21.15.119:0/1569902172 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5b78010ac0 con 0x7f5b88107e50 2024-03-28T18:51:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.207+0000 7f5b901d4740 1 Processor -- start 2024-03-28T18:51:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 -- start start 2024-03-28T18:51:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa6ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6a9d22c0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6a97ac90 con 0x562f6aa6f260 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b901d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6a9ce360 con 0x562f6aa77de0 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b87200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa6ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.208+0000 7f5b86800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49436/0 (socket says 172.21.15.119:49436) 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b86800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51308/0 (socket says 172.21.15.119:51308) 2024-03-28T18:51:53.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 -- 172.21.15.119:0/1343816145 learned_addr learned my addr 172.21.15.119:0/1343816145 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 -- 172.21.15.119:0/1343816145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 msgr2=0x562f6aa77810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 -- 172.21.15.119:0/1343816145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 msgr2=0x562f6aa6ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa6ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 -- 172.21.15.119:0/1343816145 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f6a99f080 con 0x562f6aa77de0 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5b88108dc0 tx=0x7f5b70005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eab4dc2e64f9411 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.209+0000 7f5b7fe00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 0x7f5b6c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b7fe00640 1 -- 172.21.15.119:0/1569902172 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5b780cf310 con 0x7f5b88107e50 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b77400640 1 -- 172.21.15.119:0/1343816145 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70044020 con 0x562f6aa77de0 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b77400640 1 -- 172.21.15.119:0/1343816145 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b70004240 con 0x562f6aa77de0 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b87200640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 0x7f5b6c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b77400640 1 -- 172.21.15.119:0/1343816145 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b700046c0 con 0x562f6aa77de0 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 -- 172.21.15.119:0/1343816145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 msgr2=0x562f6aa7a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5b88108dc0 tx=0x7f5b70005dd0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b87200640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 0x7f5b6c0a4eb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b80005fd0 tx=0x7f5b80009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 -- 172.21.15.119:0/1343816145 shutdown_connections 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa77de0 0x562f6aa7a230 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1343816145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa6ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 -- 172.21.15.119:0/1343816145 >> 172.21.15.119:0/1343816145 conn(0x562f6aa68f80 msgr2=0x562f6aa69360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.210+0000 7f5b901d4740 1 -- 172.21.15.119:0/1343816145 shutdown_connections 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.211+0000 7f5b901d4740 1 -- 172.21.15.119:0/1343816145 wait complete. 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.211+0000 7f5b901d4740 1 -- start start 2024-03-28T18:51:53.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.211+0000 7f5b901d4740 1 Processor -- start 2024-03-28T18:51:53.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 0x562f6aa74a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 0x562f6aa74fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6aa7acf0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6a98e0d0 con 0x562f6aa77de0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6aa7ae70 con 0x562f6aa6f260 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b901d4740 10 client.0 _ll_register_callbacks cb 0x562f6a992e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54904/0 (socket says 172.21.15.119:54904) 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b87200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 0x562f6aa74fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 0x562f6aa74a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 -- 172.21.15.119:0/1888782004 learned_addr learned my addr 172.21.15.119:0/1888782004 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 -- 172.21.15.119:0/1888782004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 msgr2=0x562f6aa74a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 0x562f6aa74a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 -- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 msgr2=0x562f6aa74fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 0x562f6aa74fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b86800640 1 -- 172.21.15.119:0/1888782004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f6a6e63f0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 0x562f6aa74a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.212+0000 7f5b87200640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 0x562f6aa74fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b86800640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b780cc9d0 tx=0x7f5b78097be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec07b8a4adbb514b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780ccdf0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5b7809acd0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b70004240 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b7809bf80 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b901d4740 10 client.17478 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562f6a7b07d0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b70004240 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.213+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5b780975d0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 10 client.17478 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 10 client.17478 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 20 client.17478 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aa87be0 0x562f6aa89fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 4 client.17478 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:53.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f6a9df800 con 0x562f6aa87be0 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aa87be0 0x562f6aa89fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b7de00640 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aa87be0 0x562f6aa89fd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1a89df55b923edbc server_cookie=67ffca70f48a0a96 in_seq=0 out_seq=0 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.214+0000 7f5b6be00640 10 client.17478 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b6be00640 1 -- 172.21.15.119:0/1888782004 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f6a9df800 con 0x562f6aa87be0 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b6be00640 10 client.17478 handle_command_reply: tid=0 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b901d4740 1 client.17478 shutdown 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 msgr2=0x562f6aa776d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5b780cc9d0 tx=0x7f5b78097be0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.215+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 shutdown_connections 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa77de0 0x562f6aa74fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6f260 0x562f6aa74a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aa87be0 0x562f6aa89fd0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 --2- 172.21.15.119:0/1888782004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa6e8b0 0x562f6aa776d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 >> 172.21.15.119:0/1888782004 conn(0x562f6aa68f80 msgr2=0x562f6aa6acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b6be00640 0 client.17478 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:53.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 shutdown_connections 2024-03-28T18:51:53.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 1 -- 172.21.15.119:0/1888782004 wait complete. 2024-03-28T18:51:53.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.216+0000 7f5b901d4740 20 client.17478 trim_cache size 0 max 16384 2024-03-28T18:51:53.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.229+0000 7f5b901d4740 1 Processor -- start 2024-03-28T18:51:53.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.229+0000 7f5b901d4740 1 -- start start 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa63ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa776d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b3810 con 0x562f6aa77de0 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b5190 con 0x562f6aa6f260 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b901d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b3e70 con 0x562f6aa6e8b0 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b86800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa63ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51334/0 (socket says 172.21.15.119:51334) 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 -- 172.21.15.119:0/857626132 learned_addr learned my addr 172.21.15.119:0/857626132 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b87200640 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa776d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 -- 172.21.15.119:0/857626132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 msgr2=0x562f6aa63ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa63ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 -- 172.21.15.119:0/857626132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 msgr2=0x562f6aa776d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa776d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b7de00640 1 -- 172.21.15.119:0/857626132 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f6aa75350 con 0x562f6aa6f260 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.230+0000 7f5b86800640 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa63ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b7de00640 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002c170 tx=0x7f5b70042040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bbcb93e975a2de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b76a00640 1 -- 172.21.15.119:0/857626132 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70044040 con 0x562f6aa6f260 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b76a00640 1 -- 172.21.15.119:0/857626132 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b700535a0 con 0x562f6aa6f260 2024-03-28T18:51:53.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b76a00640 1 -- 172.21.15.119:0/857626132 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b700406d0 con 0x562f6aa6f260 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b901d4740 1 -- 172.21.15.119:0/857626132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 msgr2=0x562f6aa77190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.231+0000 7f5b901d4740 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002c170 tx=0x7f5b70042040 comp rx=0 tx=0).stop 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 -- 172.21.15.119:0/857626132 shutdown_connections 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa776d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa77190 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 --2- 172.21.15.119:0/857626132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa63ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 -- 172.21.15.119:0/857626132 >> 172.21.15.119:0/857626132 conn(0x562f6aa68f80 msgr2=0x562f6aa68160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 -- 172.21.15.119:0/857626132 shutdown_connections 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 -- 172.21.15.119:0/857626132 wait complete. 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 -- start start 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.232+0000 7f5b901d4740 1 Processor -- start 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa7e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa7f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b4b30 con 0x562f6aa77de0 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b44d0 con 0x562f6aa6f260 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881b19f0 con 0x562f6aa6e8b0 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b901d4740 10 client.0 _ll_register_callbacks cb 0x7f5b780d2ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51348/0 (socket says 172.21.15.119:51348) 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 -- 172.21.15.119:0/2607615499 learned_addr learned my addr 172.21.15.119:0/2607615499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b86800640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa7e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b87200640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa7f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 -- 172.21.15.119:0/2607615499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 msgr2=0x562f6aa7e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa7e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 -- 172.21.15.119:0/2607615499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 msgr2=0x562f6aa7f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa7f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.233+0000 7f5b7de00640 1 -- 172.21.15.119:0/2607615499 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f6aa7fc30 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b7de00640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002c170 tx=0x7f5b70005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8ce46ecd9ef46cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70058070 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b700535a0 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5b7002cc50 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b70064050 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b901d4740 10 client.27719 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562f6aae4010 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b700535a0 con 0x562f6aa6f260 2024-03-28T18:51:53.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.234+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5b70040a60 con 0x562f6aa6f260 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 10 client.27719 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 10 client.27719 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 20 client.27719 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aae4280 0x562f6aae6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 4 client.27719 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562f6a9d9b50 con 0x562f6aae4280 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b86800640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aae4280 0x562f6aae6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b86800640 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aae4280 0x562f6aae6670 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b6a0285abef503d3 server_cookie=9afdc9dd2bab1d0e in_seq=0 out_seq=0 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b6be00640 10 client.27719 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b6be00640 1 -- 172.21.15.119:0/2607615499 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562f6a9d9b50 con 0x562f6aae4280 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b6be00640 10 client.27719 handle_command_reply: tid=0 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.235+0000 7f5b901d4740 1 client.27719 shutdown 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 msgr2=0x562f6aa7e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002c170 tx=0x7f5b70005fd0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 shutdown_connections 2024-03-28T18:51:53.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x562f6aae4280 0x562f6aae6670 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562f6aa77de0 0x562f6aa7f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562f6aa6f260 0x562f6aa7e5c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 --2- 172.21.15.119:0/2607615499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562f6aa6e8b0 0x562f6aa7e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b6be00640 0 client.27719 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 >> 172.21.15.119:0/2607615499 conn(0x562f6aa68f80 msgr2=0x562f6aa68160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 shutdown_connections 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 1 -- 172.21.15.119:0/2607615499 wait complete. 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b901d4740 20 client.27719 trim_cache size 0 max 16384 2024-03-28T18:51:53.240 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.236+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 msgr2=0x7f5b6c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 0x7f5b6c0a4eb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b80005fd0 tx=0x7f5b80009500 comp rx=0 tx=0).stop 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 msgr2=0x7f5b8819e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9f0 tx=0x7f5b7800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 shutdown_connections 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5b6c0a29f0 0x7f5b6c0a4eb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b881a2ce0 0x7f5b881a3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b8810cea0 0x7f5b8819e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 --2- 172.21.15.119:0/1569902172 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5b88107e50 0x7f5b8819e420 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 >> 172.21.15.119:0/1569902172 conn(0x7f5b880774c0 msgr2=0x7f5b88079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 shutdown_connections 2024-03-28T18:51:53.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.237+0000 7f5b8ca00640 1 -- 172.21.15.119:0/1569902172 wait complete. 2024-03-28T18:51:53.253 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:53.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:51:53.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 -- 172.21.15.119:0/2760845664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a80764f0 msgr2=0x7fd7a8111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 --2- 172.21.15.119:0/2760845664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a80764f0 0x7fd7a8111fa0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd790009930 tx=0x7fd79002bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 -- 172.21.15.119:0/2760845664 shutdown_connections 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 --2- 172.21.15.119:0/2760845664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a81124e0 0x7fd7a81148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 --2- 172.21.15.119:0/2760845664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a80764f0 0x7fd7a8111fa0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 --2- 172.21.15.119:0/2760845664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 0x7fd7a8075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 -- 172.21.15.119:0/2760845664 >> 172.21.15.119:0/2760845664 conn(0x7fd7a80ff2f0 msgr2=0x7fd7a8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 -- 172.21.15.119:0/2760845664 shutdown_connections 2024-03-28T18:51:53.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.449+0000 7fd7ad400640 1 -- 172.21.15.119:0/2760845664 wait complete. 2024-03-28T18:51:53.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 Processor -- start 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 -- start start 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 0x7fd7a819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8118090 con 0x7fd7a8075bd0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8117f10 con 0x7fd7a80764f0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.451+0000 7fd7ad400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8118210 con 0x7fd7a81124e0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 0x7fd7a819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd7a7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd7a7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49500/0 (socket says 172.21.15.119:49500) 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51368/0 (socket says 172.21.15.119:51368) 2024-03-28T18:51:53.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd7a7200640 1 -- 172.21.15.119:0/404249183 learned_addr learned my addr 172.21.15.119:0/404249183 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 -- 172.21.15.119:0/404249183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 msgr2=0x7fd7a81a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 -- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 msgr2=0x7fd7a819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 0x7fd7a819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 -- 172.21.15.119:0/404249183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a81a35c0 con 0x7fd7a80764f0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd7a7200640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.452+0000 7fd79fe00640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd790036040 tx=0x7fd790005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.453+0000 7fd79f400640 1 -- 172.21.15.119:0/404249183 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790043020 con 0x7fd7a80764f0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.453+0000 7fd79f400640 1 -- 172.21.15.119:0/404249183 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd790005000 con 0x7fd7a80764f0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.453+0000 7fd79f400640 1 -- 172.21.15.119:0/404249183 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790003af0 con 0x7fd7a80764f0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.453+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a81a3850 con 0x7fd7a80764f0 2024-03-28T18:51:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.453+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a81ab140 con 0x7fd7a80764f0 2024-03-28T18:51:53.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 Processor -- start 2024-03-28T18:51:53.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 -- start start 2024-03-28T18:51:53.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd6334b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 0x55dd6334e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd632a62c0 con 0x55dd6334bc00 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6324ec90 con 0x55dd63343080 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.456+0000 7fd7b01db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd632a2360 con 0x55dd633426d0 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd79f400640 1 -- 172.21.15.119:0/404249183 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fd790046020 con 0x7fd7a80764f0 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49516/0 (socket says 172.21.15.119:49516) 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 -- 172.21.15.119:0/839815587 learned_addr learned my addr 172.21.15.119:0/839815587 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a6800640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd6334b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd79fe00640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 0x55dd6334e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 -- 172.21.15.119:0/839815587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 msgr2=0x55dd6334b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd6334b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 -- 172.21.15.119:0/839815587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 msgr2=0x55dd6334e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 0x55dd6334e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 -- 172.21.15.119:0/839815587 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd63273080 con 0x55dd633426d0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd79fe00640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 0x55dd6334e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.457+0000 7fd7a7200640 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0011840 tx=0x7fd7a0011d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ad30ff0c22b4bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd797400640 1 -- 172.21.15.119:0/839815587 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0011330 con 0x55dd633426d0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd797400640 1 -- 172.21.15.119:0/839815587 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7a000fe80 con 0x55dd633426d0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd797400640 1 -- 172.21.15.119:0/839815587 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00193e0 con 0x55dd633426d0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd79f400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 0x7fd7880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd79f400640 1 -- 172.21.15.119:0/404249183 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd7900f0c50 con 0x7fd7a80764f0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 -- 172.21.15.119:0/839815587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 msgr2=0x55dd63342ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0011840 tx=0x7fd7a0011d00 comp rx=0 tx=0).stop 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7a6800640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 0x7fd7880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 -- 172.21.15.119:0/839815587 shutdown_connections 2024-03-28T18:51:53.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd6334bc00 0x55dd6334e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd6334b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 --2- 172.21.15.119:0/839815587 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd633426d0 0x55dd63342ab0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.458+0000 7fd7b01db740 1 -- 172.21.15.119:0/839815587 >> 172.21.15.119:0/839815587 conn(0x55dd6333cda0 msgr2=0x55dd6333d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.459+0000 7fd7b01db740 1 -- 172.21.15.119:0/839815587 shutdown_connections 2024-03-28T18:51:53.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.459+0000 7fd7b01db740 1 -- 172.21.15.119:0/839815587 wait complete. 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.459+0000 7fd7a6800640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 0x7fd7880a4eb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55dd6333de10 tx=0x7fd798005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.459+0000 7fd7b01db740 1 -- start start 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.459+0000 7fd7b01db740 1 Processor -- start 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd63348810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63348d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6334eb70 con 0x55dd633426d0 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd632620d0 con 0x55dd6334bc00 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6334ecf0 con 0x55dd63343080 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7b01db740 10 client.0 _ll_register_callbacks cb 0x55dd63266e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd63348810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54956/0 (socket says 172.21.15.119:54956) 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7a6800640 1 -- 172.21.15.119:0/3052689554 learned_addr learned my addr 172.21.15.119:0/3052689554 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd7a7200640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63348d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 -- 172.21.15.119:0/3052689554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 msgr2=0x55dd63348d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63348d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 -- 172.21.15.119:0/3052689554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 msgr2=0x55dd6334b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.460+0000 7fd79fe00640 1 -- 172.21.15.119:0/3052689554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd62fba3f0 con 0x55dd63343080 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd63348810 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ee420 tx=0x7fd7900bce70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df649d1e8f7fbd99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd78fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900bd330 con 0x55dd63343080 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd7900b99a0 con 0x55dd63343080 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd632b3800 con 0x55dd63343080 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7900c1480 con 0x55dd63343080 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd7b01db740 10 client.27706 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dd630847d0 con 0x55dd63343080 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.461+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dd632b3800 con 0x55dd63343080 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd7900dd0b0 con 0x55dd63343080 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 10 client.27706 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 10 client.27706 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 20 client.27706 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd6335b8e0 0x55dd6335dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 4 client.27706 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd632adb50 con 0x55dd6335b8e0 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7a6800640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd6335b8e0 0x55dd6335dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd7a6800640 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd6335b8e0 0x55dd6335dcd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1364d9ca36cdb6a server_cookie=6f3d499e0464dc05 in_seq=0 out_seq=0 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.462+0000 7fd78fe00640 10 client.27706 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.463+0000 7fd78fe00640 1 -- 172.21.15.119:0/3052689554 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dd632adb50 con 0x55dd6335b8e0 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.463+0000 7fd78fe00640 10 client.27706 handle_command_reply: tid=0 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 client.27706 shutdown 2024-03-28T18:51:53.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 msgr2=0x55dd63348810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd63348810 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ee420 tx=0x7fd7900bce70 comp rx=0 tx=0).stop 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 shutdown_connections 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd6335b8e0 0x55dd6335dcd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63348d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd78fe00640 0 client.27706 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd63348810 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3052689554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 >> 172.21.15.119:0/3052689554 conn(0x55dd6333cda0 msgr2=0x55dd6333eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 shutdown_connections 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 1 -- 172.21.15.119:0/3052689554 wait complete. 2024-03-28T18:51:53.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.464+0000 7fd7b01db740 20 client.27706 trim_cache size 0 max 16384 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.477+0000 7fd7b01db740 1 Processor -- start 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.477+0000 7fd7b01db740 1 -- start start 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63345ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b3a30 con 0x55dd633426d0 2024-03-28T18:51:53.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b53b0 con 0x55dd6334bc00 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7b01db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b4090 con 0x55dd63343080 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49536/0 (socket says 172.21.15.119:49536) 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 -- 172.21.15.119:0/1074539915 learned_addr learned my addr 172.21.15.119:0/1074539915 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd7a7200640 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63345ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 -- 172.21.15.119:0/1074539915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 msgr2=0x55dd63345ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63345ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 -- 172.21.15.119:0/1074539915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 msgr2=0x55dd6334af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.478+0000 7fd79fe00640 1 -- 172.21.15.119:0/1074539915 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd63346d80 con 0x55dd63343080 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd79fe00640 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd7900b9d00 tx=0x7fd7900bcfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=303582bd515f7807 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd796a00640 1 -- 172.21.15.119:0/1074539915 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900bcc90 con 0x55dd63343080 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd796a00640 1 -- 172.21.15.119:0/1074539915 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd790003c90 con 0x55dd63343080 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd796a00640 1 -- 172.21.15.119:0/1074539915 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900dd0b0 con 0x55dd63343080 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd7b01db740 1 -- 172.21.15.119:0/1074539915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 msgr2=0x55dd6334b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.479+0000 7fd7b01db740 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd7900b9d00 tx=0x7fd7900bcfa0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 -- 172.21.15.119:0/1074539915 shutdown_connections 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd6334bc00 0x55dd63345ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd63343080 0x55dd6334b480 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 --2- 172.21.15.119:0/1074539915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd6334af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 -- 172.21.15.119:0/1074539915 >> 172.21.15.119:0/1074539915 conn(0x55dd6333cda0 msgr2=0x55dd63343cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 -- 172.21.15.119:0/1074539915 shutdown_connections 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 -- 172.21.15.119:0/1074539915 wait complete. 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 -- start start 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.480+0000 7fd7b01db740 1 Processor -- start 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd633af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 0x55dd6334fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b4d50 con 0x55dd633426d0 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b46f0 con 0x55dd63343080 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a81b1c10 con 0x55dd6334bc00 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7b01db740 10 client.0 _ll_register_callbacks cb 0x7fd7900b95d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51418/0 (socket says 172.21.15.119:51418) 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 -- 172.21.15.119:0/3045524061 learned_addr learned my addr 172.21.15.119:0/3045524061 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7a7200640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 0x55dd6334fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7a6800640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd633af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 -- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 msgr2=0x55dd6334fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 0x55dd6334fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 -- 172.21.15.119:0/3045524061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 msgr2=0x55dd633af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd633af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd79fe00640 1 -- 172.21.15.119:0/3045524061 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd633502f0 con 0x55dd63343080 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.481+0000 7fd7a7200640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 0x55dd6334fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd79fe00640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd7900f1000 tx=0x7fd790003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad322d62bddedcd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd78fe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900bcc90 con 0x55dd63343080 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd7900b99a0 con 0x55dd63343080 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd790003c90 con 0x55dd63343080 2024-03-28T18:51:53.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7900bd660 con 0x55dd63343080 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd7b01db740 10 client.27737 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dd633502f0 con 0x55dd63343080 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.482+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd790003c90 con 0x55dd63343080 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd7900bc1f0 con 0x55dd63343080 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 10 client.27737 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 10 client.27737 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 20 client.27737 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd633b7f50 0x55dd633ba340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 4 client.27737 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd6322b500 con 0x55dd633b7f50 2024-03-28T18:51:53.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7a6800640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd633b7f50 0x55dd633ba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd78fe00640 10 client.27737 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:53.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.483+0000 7fd7a6800640 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd633b7f50 0x55dd633ba340 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e030858fa27be3a server_cookie=7778c162d3df5269 in_seq=0 out_seq=0 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd78fe00640 1 -- 172.21.15.119:0/3045524061 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd6322b500 con 0x55dd633b7f50 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd78fe00640 10 client.27737 handle_command_reply: tid=0 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 client.27737 shutdown 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 msgr2=0x55dd633af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd7900f1000 tx=0x7fd790003f90 comp rx=0 tx=0).stop 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 shutdown_connections 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dd633b7f50 0x55dd633ba340 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dd6334bc00 0x55dd6334fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dd63343080 0x55dd633af920 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 --2- 172.21.15.119:0/3045524061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dd633426d0 0x55dd633af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd78fe00640 0 client.27737 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 >> 172.21.15.119:0/3045524061 conn(0x55dd6333cda0 msgr2=0x55dd6333b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 shutdown_connections 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 1 -- 172.21.15.119:0/3045524061 wait complete. 2024-03-28T18:51:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.484+0000 7fd7b01db740 20 client.27737 trim_cache size 0 max 16384 2024-03-28T18:51:53.490 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 msgr2=0x7fd7880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 0x7fd7880a4eb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55dd6333de10 tx=0x7fd798005f90 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 msgr2=0x7fd7a819eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd790036040 tx=0x7fd790005e80 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 shutdown_connections 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd7880a29f0 0x7fd7880a4eb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7a81124e0 0x7fd7a81a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd7a80764f0 0x7fd7a819eb10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 --2- 172.21.15.119:0/404249183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7a8075bd0 0x7fd7a819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 >> 172.21.15.119:0/404249183 conn(0x7fd7a80ff2f0 msgr2=0x7fd7a8113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 shutdown_connections 2024-03-28T18:51:53.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.485+0000 7fd7ad400640 1 -- 172.21.15.119:0/404249183 wait complete. 2024-03-28T18:51:53.502 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/870642307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:53.506 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:53.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 -- 172.21.15.119:0/3785928990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c109850 msgr2=0x7f505c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 --2- 172.21.15.119:0/3785928990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c109850 0x7f505c109c30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5044009900 tx=0x7f504402bc30 comp rx=0 tx=0).stop 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 -- 172.21.15.119:0/3785928990 shutdown_connections 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 --2- 172.21.15.119:0/3785928990 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c104200 0x7f505c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 --2- 172.21.15.119:0/3785928990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 0x7f505c103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 --2- 172.21.15.119:0/3785928990 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c109850 0x7f505c109c30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 -- 172.21.15.119:0/3785928990 >> 172.21.15.119:0/3785928990 conn(0x7f505c0ff2f0 msgr2=0x7f505c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 -- 172.21.15.119:0/3785928990 shutdown_connections 2024-03-28T18:51:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.708+0000 7f5062c00640 1 -- 172.21.15.119:0/3785928990 wait complete. 2024-03-28T18:51:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 Processor -- start 2024-03-28T18:51:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 -- start start 2024-03-28T18:51:53.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 0x7f505c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 0x7f505c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c113cb0 con 0x7f505c103840 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c113b30 con 0x7f505c104200 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f5062c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c113e30 con 0x7f505c109850 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f505b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.709+0000 7f505be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 0x7f505c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51434/0 (socket says 172.21.15.119:51434) 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 -- 172.21.15.119:0/492209409 learned_addr learned my addr 172.21.15.119:0/492209409 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5060c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 0x7f505c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 -- 172.21.15.119:0/492209409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 msgr2=0x7f505c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 0x7f505c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 -- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 msgr2=0x7f505c19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 0x7f505c19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 -- 172.21.15.119:0/492209409 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c19f120 con 0x7f505c104200 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5060c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 0x7f505c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f505b400640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f504c00db80 tx=0x7f504c00b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5058c00640 1 -- 172.21.15.119:0/492209409 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c017070 con 0x7f505c104200 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5058c00640 1 -- 172.21.15.119:0/492209409 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f504c003b70 con 0x7f505c104200 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5058c00640 1 -- 172.21.15.119:0/492209409 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c0106c0 con 0x7f505c104200 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.710+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c19f410 con 0x7f505c104200 2024-03-28T18:51:53.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.711+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1a6ca0 con 0x7f505c104200 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.713+0000 7f5058c00640 1 -- 172.21.15.119:0/492209409 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f504c004020 con 0x7f505c104200 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 Processor -- start 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 -- start start 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 0x55ba421d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421e3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba4213b2c0 con 0x55ba421e0c00 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba420e3c90 con 0x55ba421d76d0 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f50659ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba42137360 con 0x55ba421d8080 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f505be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f5060c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 0x55ba421d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f505be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49574/0 (socket says 172.21.15.119:49574) 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f505be00640 1 -- 172.21.15.119:0/45157857 learned_addr learned my addr 172.21.15.119:0/45157857 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.714+0000 7f505b400640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 -- 172.21.15.119:0/45157857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 msgr2=0x55ba421d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 0x55ba421d7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 -- 172.21.15.119:0/45157857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 msgr2=0x55ba421e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421e3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 -- 172.21.15.119:0/45157857 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba42108080 con 0x55ba421d8080 2024-03-28T18:51:53.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5060c00640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 0x55ba421d7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505b400640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f505be00640 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5044009230 tx=0x7f5044009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a9a018e3f5224b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5058c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 0x7f50400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5051200640 1 -- 172.21.15.119:0/45157857 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044042070 con 0x55ba421d8080 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5051200640 1 -- 172.21.15.119:0/45157857 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5044004240 con 0x55ba421d8080 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5051200640 1 -- 172.21.15.119:0/45157857 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5044004f80 con 0x55ba421d8080 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5058c00640 1 -- 172.21.15.119:0/492209409 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f504c0cec20 con 0x7f505c104200 2024-03-28T18:51:53.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.715+0000 7f5060c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 0x7f50400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 -- 172.21.15.119:0/45157857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 msgr2=0x55ba421e0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5044009230 tx=0x7f5044009560 comp rx=0 tx=0).stop 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 -- 172.21.15.119:0/45157857 shutdown_connections 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421e0630 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 --2- 172.21.15.119:0/45157857 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d76d0 0x55ba421d7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 -- 172.21.15.119:0/45157857 >> 172.21.15.119:0/45157857 conn(0x55ba421d1da0 msgr2=0x55ba421d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 -- 172.21.15.119:0/45157857 shutdown_connections 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f5060c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 0x7f50400a4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5054009fd0 tx=0x7f5054005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.716+0000 7f50659ab740 1 -- 172.21.15.119:0/45157857 wait complete. 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 -- start start 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 Processor -- start 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 0x55ba421dd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421ddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba421e3b10 con 0x55ba421e0c00 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba420f70d0 con 0x55ba421d8080 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba421e3c90 con 0x55ba421d76d0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f50659ab740 10 client.0 _ll_register_callbacks cb 0x55ba420fbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f505be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f505b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 0x55ba421dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.717+0000 7f505be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49580/0 (socket says 172.21.15.119:49580) 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 -- 172.21.15.119:0/3386391860 learned_addr learned my addr 172.21.15.119:0/3386391860 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f5060c00640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421ddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 -- 172.21.15.119:0/3386391860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 msgr2=0x55ba421dd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 0x55ba421dd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 -- 172.21.15.119:0/3386391860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 msgr2=0x55ba421ddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421ddd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 -- 172.21.15.119:0/3386391860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba41e4f3f0 con 0x55ba421d76d0 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505b400640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 0x55ba421dd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f505be00640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5044005ec0 tx=0x7f5044009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ecb810149d8d93d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f5049600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504404d070 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f504402ccc0 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.718+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5044004240 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5044042040 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 10 client.27724 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ba41f197d0 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5044004240 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5044058030 con 0x55ba421d76d0 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 10 client.27724 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 10 client.27724 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 20 client.27724 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba421f0820 0x55ba421f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 4 client.27724 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba42148800 con 0x55ba421f0820 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.719+0000 7f505b400640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba421f0820 0x55ba421f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.720+0000 7f505b400640 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba421f0820 0x55ba421f2c10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e7daf69e119bca2 server_cookie=5c2726152bcf67d7 in_seq=0 out_seq=0 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.720+0000 7f5049600640 10 client.27724 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f5049600640 1 -- 172.21.15.119:0/3386391860 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ba42148800 con 0x55ba421f0820 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f5049600640 10 client.27724 handle_command_reply: tid=0 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 client.27724 shutdown 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 msgr2=0x55ba421e04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5044005ec0 tx=0x7f5044009260 comp rx=0 tx=0).stop 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 shutdown_connections 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba421f0820 0x55ba421f2c10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421e0c00 0x55ba421ddd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f5049600640 0 client.27724 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421d8080 0x55ba421dd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 --2- 172.21.15.119:0/3386391860 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d76d0 0x55ba421e04a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 >> 172.21.15.119:0/3386391860 conn(0x55ba421d1da0 msgr2=0x55ba421d3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.721+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 shutdown_connections 2024-03-28T18:51:53.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.722+0000 7f50659ab740 1 -- 172.21.15.119:0/3386391860 wait complete. 2024-03-28T18:51:53.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.722+0000 7f50659ab740 20 client.27724 trim_cache size 0 max 16384 2024-03-28T18:51:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 Processor -- start 2024-03-28T18:51:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 -- start start 2024-03-28T18:51:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1af5e0 con 0x55ba421d76d0 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.735+0000 7f50659ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1b0f60 con 0x55ba421e0c00 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f50659ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1afc40 con 0x55ba421d8080 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f5060c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55024/0 (socket says 172.21.15.119:55024) 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505be00640 1 -- 172.21.15.119:0/2715334126 learned_addr learned my addr 172.21.15.119:0/2715334126 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49596/0 (socket says 172.21.15.119:49596) 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 -- 172.21.15.119:0/2715334126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 msgr2=0x55ba421e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 -- 172.21.15.119:0/2715334126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 msgr2=0x55ba421d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 -- 172.21.15.119:0/2715334126 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba421e3b70 con 0x55ba421d8080 2024-03-28T18:51:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f5060c00640 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.736+0000 7f505b400640 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f504c0cc280 tx=0x7f504c0bb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0cc26b972d79532 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f504be00640 1 -- 172.21.15.119:0/2715334126 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c0a3930 con 0x55ba421d8080 2024-03-28T18:51:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f504be00640 1 -- 172.21.15.119:0/2715334126 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f504c0bb0b0 con 0x55ba421d8080 2024-03-28T18:51:53.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f504be00640 1 -- 172.21.15.119:0/2715334126 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c097800 con 0x55ba421d8080 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 -- 172.21.15.119:0/2715334126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 msgr2=0x55ba421dff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f504c0cc280 tx=0x7f504c0bb040 comp rx=0 tx=0).stop 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 -- 172.21.15.119:0/2715334126 shutdown_connections 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421dff60 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 --2- 172.21.15.119:0/2715334126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 -- 172.21.15.119:0/2715334126 >> 172.21.15.119:0/2715334126 conn(0x55ba421d1da0 msgr2=0x55ba421d0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.737+0000 7f50659ab740 1 -- 172.21.15.119:0/2715334126 shutdown_connections 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.738+0000 7f50659ab740 1 -- 172.21.15.119:0/2715334126 wait complete. 2024-03-28T18:51:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.738+0000 7f50659ab740 1 -- start start 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.738+0000 7f50659ab740 1 Processor -- start 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.738+0000 7f50659ab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1b0900 con 0x55ba421d76d0 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1b02a0 con 0x55ba421e0c00 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c1ad7c0 con 0x55ba421d8080 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f50659ab740 10 client.0 _ll_register_callbacks cb 0x7f5054009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55040/0 (socket says 172.21.15.119:55040) 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505be00640 1 -- 172.21.15.119:0/3370474336 learned_addr learned my addr 172.21.15.119:0/3370474336 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f5060c00640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 -- 172.21.15.119:0/3370474336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 msgr2=0x55ba421f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 -- 172.21.15.119:0/3370474336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 msgr2=0x55ba421f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 -- 172.21.15.119:0/3370474336 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba4224afa0 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f505b400640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421f37f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55ba421d2c20 tx=0x7f504c00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=682c3017b963a064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.739+0000 7f5049600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c0a3930 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f504c097800 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f504c0bb0b0 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f504c0cc500 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 10 client.27730 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ba4224c2a0 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f504c0bb0b0 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f504c0b6020 con 0x55ba421d8080 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 10 client.27730 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 10 client.27730 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 20 client.27730 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba4224c7c0 0x55ba4224ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 4 client.27730 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.740+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ba421de100 con 0x55ba4224c7c0 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f505be00640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba4224c7c0 0x55ba4224ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f505be00640 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba4224c7c0 0x55ba4224ebe0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30e01fe1d11a92c3 server_cookie=81b43bc1b8af9b05 in_seq=0 out_seq=0 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f5049600640 10 client.27730 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f5049600640 1 -- 172.21.15.119:0/3370474336 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ba421de100 con 0x55ba4224c7c0 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f5049600640 10 client.27730 handle_command_reply: tid=0 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 client.27730 shutdown 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 msgr2=0x55ba421f37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421f37f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55ba421d2c20 tx=0x7f504c00be60 comp rx=0 tx=0).stop 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 shutdown_connections 2024-03-28T18:51:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ba4224c7c0 0x55ba4224ebe0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ba421e0c00 0x55ba421f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f5049600640 0 client.27730 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ba421d8080 0x55ba421f37f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 --2- 172.21.15.119:0/3370474336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ba421d76d0 0x55ba421f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.741+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 >> 172.21.15.119:0/3370474336 conn(0x55ba421d1da0 msgr2=0x55ba421d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 shutdown_connections 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f50659ab740 1 -- 172.21.15.119:0/3370474336 wait complete. 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f50659ab740 20 client.27730 trim_cache size 0 max 16384 2024-03-28T18:51:53.747 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 msgr2=0x7f50400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 0x7f50400a4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5054009fd0 tx=0x7f5054005f90 comp rx=0 tx=0).stop 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 msgr2=0x7f505c19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.742+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f504c00db80 tx=0x7f504c00b630 comp rx=0 tx=0).stop 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 shutdown_connections 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f50400a29f0 0x7f50400a4eb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f505c109850 0x7f505c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f505c104200 0x7f505c19a670 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 --2- 172.21.15.119:0/492209409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f505c103840 0x7f505c19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 >> 172.21.15.119:0/492209409 conn(0x7f505c0ff2f0 msgr2=0x7f505c0ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 shutdown_connections 2024-03-28T18:51:53.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.743+0000 7f5062c00640 1 -- 172.21.15.119:0/492209409 wait complete. 2024-03-28T18:51:53.761 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:51:53.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:51:53.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/870642307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/870642307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 -- 172.21.15.119:0/4235926427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf50075bd0 msgr2=0x7fcf50075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 --2- 172.21.15.119:0/4235926427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf50075bd0 0x7fcf50075fb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009930 tx=0x7fcf4402bc20 comp rx=0 tx=0).stop 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 -- 172.21.15.119:0/4235926427 shutdown_connections 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 --2- 172.21.15.119:0/4235926427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf501124e0 0x7fcf501148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 --2- 172.21.15.119:0/4235926427 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf500764f0 0x7fcf50111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 --2- 172.21.15.119:0/4235926427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf50075bd0 0x7fcf50075fb0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 -- 172.21.15.119:0/4235926427 >> 172.21.15.119:0/4235926427 conn(0x7fcf500ff2f0 msgr2=0x7fcf50101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 -- 172.21.15.119:0/4235926427 shutdown_connections 2024-03-28T18:51:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.970+0000 7fcf57400640 1 -- 172.21.15.119:0/4235926427 wait complete. 2024-03-28T18:51:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.971+0000 7fcf57400640 1 Processor -- start 2024-03-28T18:51:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.971+0000 7fcf57400640 1 -- start start 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf500764f0 0x7fcf5019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf501124e0 0x7fcf501a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50118090 con 0x7fcf500764f0 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50117f10 con 0x7fcf50075bd0 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf57400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf50118210 con 0x7fcf501124e0 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51482/0 (socket says 172.21.15.119:51482) 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 -- 172.21.15.119:0/2960421413 learned_addr learned my addr 172.21.15.119:0/2960421413 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf55400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf501124e0 0x7fcf501a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf500764f0 0x7fcf5019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 -- 172.21.15.119:0/2960421413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf501124e0 msgr2=0x7fcf501a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf501124e0 0x7fcf501a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 -- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf500764f0 msgr2=0x7fcf5019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf500764f0 0x7fcf5019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 -- 172.21.15.119:0/2960421413 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf501a3540 con 0x7fcf50075bd0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.972+0000 7fcf54a00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005ab0 tx=0x7fcf440043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.973+0000 7fcf4d600640 1 -- 172.21.15.119:0/2960421413 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44043020 con 0x7fcf50075bd0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.973+0000 7fcf4d600640 1 -- 172.21.15.119:0/2960421413 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf440054e0 con 0x7fcf50075bd0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.973+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf501a37d0 con 0x7fcf50075bd0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.973+0000 7fcf4d600640 1 -- 172.21.15.119:0/2960421413 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44003c50 con 0x7fcf50075bd0 2024-03-28T18:51:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.973+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501ab0c0 con 0x7fcf50075bd0 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf5a18f740 1 Processor -- start 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf5a18f740 1 -- start start 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf4d600640 1 -- 172.21.15.119:0/2960421413 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fcf44046020 con 0x7fcf50075bd0 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f8080 0x557781700630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.976+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781703050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55778165b2c0 con 0x557781700c00 2024-03-28T18:51:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557781603c90 con 0x5577816f8080 2024-03-28T18:51:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557781657360 con 0x5577816f76d0 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49626/0 (socket says 172.21.15.119:49626) 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 -- 172.21.15.119:0/4073244629 learned_addr learned my addr 172.21.15.119:0/4073244629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf54a00640 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f8080 0x557781700630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781703050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 -- 172.21.15.119:0/4073244629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f8080 msgr2=0x557781700630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f8080 0x557781700630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 -- 172.21.15.119:0/4073244629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 msgr2=0x557781703050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781703050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 -- 172.21.15.119:0/4073244629 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557781628080 con 0x5577816f76d0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.977+0000 7fcf55400640 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcf48007e60 tx=0x7fcf4800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef59c878a18d63fb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf41c00640 1 -- 172.21.15.119:0/4073244629 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48010840 con 0x5577816f76d0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf41c00640 1 -- 172.21.15.119:0/4073244629 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf480183f0 con 0x5577816f76d0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf41c00640 1 -- 172.21.15.119:0/4073244629 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48017600 con 0x5577816f76d0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf4d600640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 0x7fcf380a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf4d600640 1 -- 172.21.15.119:0/2960421413 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fcf440f0e20 con 0x7fcf50075bd0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf54a00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 0x7fcf380a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 -- 172.21.15.119:0/4073244629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 msgr2=0x5577816f7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcf48007e60 tx=0x7fcf4800f790 comp rx=0 tx=0).stop 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 -- 172.21.15.119:0/4073244629 shutdown_connections 2024-03-28T18:51:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781703050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f8080 0x557781700630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.978+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/4073244629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816f7ab0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf5a18f740 1 -- 172.21.15.119:0/4073244629 >> 172.21.15.119:0/4073244629 conn(0x5577816f1da0 msgr2=0x5577816f2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf5a18f740 1 -- 172.21.15.119:0/4073244629 shutdown_connections 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf5a18f740 1 -- 172.21.15.119:0/4073244629 wait complete. 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf54a00640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 0x7fcf380a4eb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5577816f71a0 tx=0x7fcf440dd040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf5a18f740 1 -- start start 2024-03-28T18:51:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.979+0000 7fcf5a18f740 1 Processor -- start 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557781703b70 con 0x5577816f76d0 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577816170d0 con 0x557781700c00 2024-03-28T18:51:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557781703cf0 con 0x5577816f8080 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf5a18f740 10 client.0 _ll_register_callbacks cb 0x55778161be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf55400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49642/0 (socket says 172.21.15.119:49642) 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 -- 172.21.15.119:0/2689320179 learned_addr learned my addr 172.21.15.119:0/2689320179 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf55400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55076/0 (socket says 172.21.15.119:55076) 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf55400640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 -- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 msgr2=0x5577816fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 -- 172.21.15.119:0/2689320179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 msgr2=0x5577817004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf4fe00640 1 -- 172.21.15.119:0/2689320179 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55778136f3f0 con 0x5577816f8080 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.980+0000 7fcf55400640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00bad0 tx=0x7fcf3c00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9041aedf50dcab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf36000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c003e00 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fcf3c005020 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf480183f0 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf3c005220 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf5a18f740 10 client.27742 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5577814397d0 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.981+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf480183f0 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcf3c01b630 con 0x5577816f8080 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 10 client.27742 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 10 client.27742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 20 client.27742 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557781710aa0 0x557781712e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 4 client.27742 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557781668800 con 0x557781710aa0 2024-03-28T18:51:53.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.982+0000 7fcf55400640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557781710aa0 0x557781712e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf55400640 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557781710aa0 0x557781712e90 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cbc66bf770a7374 server_cookie=62109269d3b7fbd4 in_seq=0 out_seq=0 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf36000640 10 client.27742 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf36000640 1 -- 172.21.15.119:0/2689320179 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557781668800 con 0x557781710aa0 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf36000640 10 client.27742 handle_command_reply: tid=0 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf5a18f740 1 client.27742 shutdown 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 msgr2=0x5577816fd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.983+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00bad0 tx=0x7fcf3c00bf90 comp rx=0 tx=0).stop 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 shutdown_connections 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557781710aa0 0x557781712e90 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x5577816fd870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf36000640 0 client.27742 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/2689320179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f76d0 0x5577817004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 >> 172.21.15.119:0/2689320179 conn(0x5577816f1da0 msgr2=0x5577816f3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:53.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 shutdown_connections 2024-03-28T18:51:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 1 -- 172.21.15.119:0/2689320179 wait complete. 2024-03-28T18:51:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.984+0000 7fcf5a18f740 20 client.27742 trim_cache size 0 max 16384 2024-03-28T18:51:53.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.997+0000 7fcf5a18f740 1 Processor -- start 2024-03-28T18:51:53.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.997+0000 7fcf5a18f740 1 -- start start 2024-03-28T18:51:53.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:53.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b3a10 con 0x5577816f8080 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b5390 con 0x557781700c00 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b4070 con 0x5577816f76d0 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf55400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55090/0 (socket says 172.21.15.119:55090) 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 -- 172.21.15.119:0/1729298793 learned_addr learned my addr 172.21.15.119:0/1729298793 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf55400640 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 -- 172.21.15.119:0/1729298793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 msgr2=0x5577816fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 -- 172.21.15.119:0/1729298793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 msgr2=0x5577816fab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf4fe00640 1 -- 172.21.15.119:0/1729298793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557781703a50 con 0x5577816f8080 2024-03-28T18:51:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.998+0000 7fcf55400640 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.999+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00baa0 tx=0x7fcf3c004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b963bea32165911e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.999+0000 7fcf41200640 1 -- 172.21.15.119:0/1729298793 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c02f070 con 0x5577816f8080 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.999+0000 7fcf41200640 1 -- 172.21.15.119:0/1729298793 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf3c01bcc0 con 0x5577816f8080 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:53.999+0000 7fcf41200640 1 -- 172.21.15.119:0/1729298793 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c011a00 con 0x5577816f8080 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- 172.21.15.119:0/1729298793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 msgr2=0x5577817004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00baa0 tx=0x7fcf3c004460 comp rx=0 tx=0).stop 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- 172.21.15.119:0/1729298793 shutdown_connections 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557781700c00 0x5577816fab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5577816f8080 0x5577817004e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/1729298793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f76d0 0x5577816fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- 172.21.15.119:0/1729298793 >> 172.21.15.119:0/1729298793 conn(0x5577816f1da0 msgr2=0x5577816f3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- 172.21.15.119:0/1729298793 shutdown_connections 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- 172.21.15.119:0/1729298793 wait complete. 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 -- start start 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.000+0000 7fcf5a18f740 1 Processor -- start 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 0x557781708ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x557781704e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781705390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b4d30 con 0x557781700c00 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b46d0 con 0x5577816f76d0 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf501b1bf0 con 0x5577816f8080 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf5a18f740 10 client.0 _ll_register_callbacks cb 0x7fcf3c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 0x557781708ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x557781704e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf55400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 0x557781708ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51530/0 (socket says 172.21.15.119:51530) 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf55400640 1 -- 172.21.15.119:0/3985298387 learned_addr learned my addr 172.21.15.119:0/3985298387 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf55400640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781705390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 -- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 msgr2=0x557781708ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 0x557781708ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 -- 172.21.15.119:0/3985298387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 msgr2=0x557781705390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781705390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.001+0000 7fcf4fe00640 1 -- 172.21.15.119:0/3985298387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557781705980 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf4fe00640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x557781704e50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5577816f4210 tx=0x7fcf3c0115e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78cc5519fa9eabab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c02f040 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fcf3c011780 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf3c01bcc0 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf3c0119c0 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf5a18f740 10 client.27748 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55778176d130 con 0x5577816f8080 2024-03-28T18:51:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf3c01bcc0 con 0x5577816f8080 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.002+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcf3c011c60 con 0x5577816f8080 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 10 client.27748 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 10 client.27748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 20 client.27748 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55778176d780 0x55778176fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 4 client.27748 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557781662b50 con 0x55778176d780 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf55400640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55778176d780 0x55778176fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf55400640 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55778176d780 0x55778176fba0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d86d5403819b8200 server_cookie=37b9d379b2ef4bba in_seq=0 out_seq=0 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf36000640 10 client.27748 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf36000640 1 -- 172.21.15.119:0/3985298387 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557781662b50 con 0x55778176d780 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf36000640 10 client.27748 handle_command_reply: tid=0 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.003+0000 7fcf5a18f740 1 client.27748 shutdown 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 msgr2=0x557781704e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x557781704e50 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5577816f4210 tx=0x7fcf3c0115e0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 shutdown_connections 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55778176d780 0x55778176fba0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557781700c00 0x557781705390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5577816f8080 0x557781704e50 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 --2- 172.21.15.119:0/3985298387 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5577816f76d0 0x557781708ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf36000640 0 client.27748 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 >> 172.21.15.119:0/3985298387 conn(0x5577816f1da0 msgr2=0x5577816f31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 shutdown_connections 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 1 -- 172.21.15.119:0/3985298387 wait complete. 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf5a18f740 20 client.27748 trim_cache size 0 max 16384 2024-03-28T18:51:54.009 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:51:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 msgr2=0x7fcf380a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 0x7fcf380a4eb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5577816f71a0 tx=0x7fcf440dd040 comp rx=0 tx=0).stop 2024-03-28T18:51:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 msgr2=0x7fcf5019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.004+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcf44005ab0 tx=0x7fcf440043b0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 shutdown_connections 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fcf380a29f0 0x7fcf380a4eb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf501124e0 0x7fcf501a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf500764f0 0x7fcf5019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 --2- 172.21.15.119:0/2960421413 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf50075bd0 0x7fcf5019e550 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 >> 172.21.15.119:0/2960421413 conn(0x7fcf500ff2f0 msgr2=0x7fcf501016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 shutdown_connections 2024-03-28T18:51:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:54.005+0000 7fcf57400640 1 -- 172.21.15.119:0/2960421413 wait complete. 2024-03-28T18:51:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:54 smithi119 ceph-mon[31813]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:54 smithi196 ceph-mon[36300]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:54 smithi131 ceph-mon[36065]: pgmap v240: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:55.023 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:51:55.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:51:55.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 -- 172.21.15.119:0/3690301203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea81124d0 msgr2=0x7efea81148c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 --2- 172.21.15.119:0/3690301203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea81124d0 0x7efea81148c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efea0009c10 tx=0x7efea002bef0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 -- 172.21.15.119:0/3690301203 shutdown_connections 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 --2- 172.21.15.119:0/3690301203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea81124d0 0x7efea81148c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 --2- 172.21.15.119:0/3690301203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea80764f0 0x7efea8111f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 --2- 172.21.15.119:0/3690301203 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea8075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 -- 172.21.15.119:0/3690301203 >> 172.21.15.119:0/3690301203 conn(0x7efea80ff320 msgr2=0x7efea8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 -- 172.21.15.119:0/3690301203 shutdown_connections 2024-03-28T18:51:55.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.252+0000 7efeafe00640 1 -- 172.21.15.119:0/3690301203 wait complete. 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 Processor -- start 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 -- start start 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 0x7efea819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea81124d0 0x7efea81a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8118080 con 0x7efea81124d0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8117f00 con 0x7efea8075bd0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.253+0000 7efeafe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea8118200 con 0x7efea80764f0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efeaca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 0x7efea819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51538/0 (socket says 172.21.15.119:51538) 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 -- 172.21.15.119:0/4152117755 learned_addr learned my addr 172.21.15.119:0/4152117755 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efeade00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea81124d0 0x7efea81a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 -- 172.21.15.119:0/4152117755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 msgr2=0x7efea819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 0x7efea819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 -- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea81124d0 msgr2=0x7efea81a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea81124d0 0x7efea81a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 -- 172.21.15.119:0/4152117755 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea81a35a0 con 0x7efea8075bd0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efeaca00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 0x7efea819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.254+0000 7efead400640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9c0 tx=0x7efe9c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.255+0000 7efea6000640 1 -- 172.21.15.119:0/4152117755 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00cde0 con 0x7efea8075bd0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.255+0000 7efea6000640 1 -- 172.21.15.119:0/4152117755 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe9c0035f0 con 0x7efea8075bd0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.255+0000 7efea6000640 1 -- 172.21.15.119:0/4152117755 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c010690 con 0x7efea8075bd0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.255+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea81a3890 con 0x7efea8075bd0 2024-03-28T18:51:55.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.255+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea81ab120 con 0x7efea8075bd0 2024-03-28T18:51:55.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.258+0000 7efea6000640 1 -- 172.21.15.119:0/4152117755 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7efe9c0037a0 con 0x7efea8075bd0 2024-03-28T18:51:55.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.258+0000 7efeb29df740 1 Processor -- start 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.258+0000 7efeb29df740 1 -- start start 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.258+0000 7efeb29df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeb29df740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeb1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeb29df740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeb29df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96ae0ceb0 con 0x55e96aea8080 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeb29df740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96adb5a60 con 0x55e96aeb1580 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeb29df740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96ae08f50 con 0x55e96aea89a0 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efead400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeb1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeade00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51542/0 (socket says 172.21.15.119:51542) 2024-03-28T18:51:55.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 -- 172.21.15.119:0/332532324 learned_addr learned my addr 172.21.15.119:0/332532324 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 -- 172.21.15.119:0/332532324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 msgr2=0x55e96aeb1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeb1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 -- 172.21.15.119:0/332532324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 msgr2=0x55e96aea8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efeaca00640 1 -- 172.21.15.119:0/332532324 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e96abe97e0 con 0x55e96aeb1580 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.259+0000 7efea6000640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 0x7efe900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efeaca00640 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efe94011470 tx=0x7efe94011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22270948d9096ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efea6000640 1 -- 172.21.15.119:0/4152117755 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7efe9c0cf210 con 0x7efea8075bd0 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efeade00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 0x7efe900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efe9a600640 1 -- 172.21.15.119:0/332532324 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9401e070 con 0x55e96aeb1580 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efe9a600640 1 -- 172.21.15.119:0/332532324 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe9400fae0 con 0x55e96aeb1580 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.260+0000 7efe9a600640 1 -- 172.21.15.119:0/332532324 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94019b60 con 0x55e96aeb1580 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- 172.21.15.119:0/332532324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 msgr2=0x55e96aeb3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeade00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 0x7efe900a4eb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efea002c430 tx=0x7efea0005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efe94011470 tx=0x7efe94011930 comp rx=0 tx=0).stop 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- 172.21.15.119:0/332532324 shutdown_connections 2024-03-28T18:51:55.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeb3970 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeb1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 --2- 172.21.15.119:0/332532324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- 172.21.15.119:0/332532324 >> 172.21.15.119:0/332532324 conn(0x55e96aea27b0 msgr2=0x55e96aea2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- 172.21.15.119:0/332532324 shutdown_connections 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- 172.21.15.119:0/332532324 wait complete. 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 -- start start 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.261+0000 7efeb29df740 1 Processor -- start 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeae250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeae790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96aeb4370 con 0x55e96aea8080 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96adc8eb0 con 0x55e96aeb1580 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e96aeb44f0 con 0x55e96aea89a0 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeb29df740 10 client.0 _ll_register_callbacks cb 0x55e96adbd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:55.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efead400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeade00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efeaca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efead400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55130/0 (socket says 172.21.15.119:55130) 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.262+0000 7efead400640 1 -- 172.21.15.119:0/2229639015 learned_addr learned my addr 172.21.15.119:0/2229639015 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 -- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 msgr2=0x55e96aeae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 -- 172.21.15.119:0/2229639015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 msgr2=0x55e96aeb0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 -- 172.21.15.119:0/2229639015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e96ac49f70 con 0x55e96aea89a0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efead400640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeade00640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeaca00640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeae250 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efe9401c040 tx=0x7efe94009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59b36a2ebd5fb97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efe8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9401e040 con 0x55e96aea89a0 2024-03-28T18:51:55.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe9400fae0 con 0x55e96aea89a0 2024-03-28T18:51:55.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7efe9400f700 con 0x55e96aea89a0 2024-03-28T18:51:55.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe94031050 con 0x55e96aea89a0 2024-03-28T18:51:55.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeb29df740 10 client.27760 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:55.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.263+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e96aae1170 con 0x55e96aea89a0 2024-03-28T18:51:55.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe9400fae0 con 0x55e96aea89a0 2024-03-28T18:51:55.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7efe9402c030 con 0x55e96aea89a0 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 10 client.27760 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 10 client.27760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 20 client.27760 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96aec1500 0x55e96aec38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 4 client.27760 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e96ae1a3f0 con 0x55e96aec1500 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.264+0000 7efead400640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96aec1500 0x55e96aec38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.265+0000 7efead400640 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96aec1500 0x55e96aec38f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cd524b4e767a7bc server_cookie=4da33cc9794c04fb in_seq=0 out_seq=0 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.265+0000 7efe8ea00640 10 client.27760 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efe8ea00640 1 -- 172.21.15.119:0/2229639015 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e96ae1a3f0 con 0x55e96aec1500 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efe8ea00640 10 client.27760 handle_command_reply: tid=0 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 client.27760 shutdown 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 msgr2=0x55e96aeae250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeae250 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7efe9401c040 tx=0x7efe94009c70 comp rx=0 tx=0).stop 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 shutdown_connections 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96aec1500 0x55e96aec38f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96aeae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeae250 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 --2- 172.21.15.119:0/2229639015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeb0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efe8ea00640 0 client.27760 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 >> 172.21.15.119:0/2229639015 conn(0x55e96aea27b0 msgr2=0x55e96aea4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 shutdown_connections 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 1 -- 172.21.15.119:0/2229639015 wait complete. 2024-03-28T18:51:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.266+0000 7efeb29df740 20 client.27760 trim_cache size 0 max 16384 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 Processor -- start 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 -- start start 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 0x55e96aeb0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b3ae0 con 0x55e96aea8080 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b5460 con 0x55e96aea89a0 2024-03-28T18:51:55.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeb29df740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b4140 con 0x55e96aeb1580 2024-03-28T18:51:55.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeaca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 0x55e96aeb0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeade00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efead400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeade00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49712/0 (socket says 172.21.15.119:49712) 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.284+0000 7efeade00640 1 -- 172.21.15.119:0/2138964544 learned_addr learned my addr 172.21.15.119:0/2138964544 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 -- 172.21.15.119:0/2138964544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 msgr2=0x55e96aeb0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 0x55e96aeb0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 -- 172.21.15.119:0/2138964544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 msgr2=0x55e96aea42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 -- 172.21.15.119:0/2138964544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e96ae9dc20 con 0x55e96aeb1580 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeaca00640 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 0x55e96aeb0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efead400640 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efeade00640 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efea0009c10 tx=0x7efea0042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e760a8a9d8433cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efe99c00640 1 -- 172.21.15.119:0/2138964544 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea002cde0 con 0x55e96aeb1580 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.285+0000 7efe99c00640 1 -- 172.21.15.119:0/2138964544 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efea0033830 con 0x55e96aeb1580 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efe99c00640 1 -- 172.21.15.119:0/2138964544 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea004e8d0 con 0x55e96aeb1580 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 -- 172.21.15.119:0/2138964544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 msgr2=0x55e96aeb0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7efea0009c10 tx=0x7efea0042000 comp rx=0 tx=0).stop 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 -- 172.21.15.119:0/2138964544 shutdown_connections 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aeb1580 0x55e96aeb0ec0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aea89a0 0x55e96aeb0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 --2- 172.21.15.119:0/2138964544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aea42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 -- 172.21.15.119:0/2138964544 >> 172.21.15.119:0/2138964544 conn(0x55e96aea27b0 msgr2=0x55e96aea3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 -- 172.21.15.119:0/2138964544 shutdown_connections 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.286+0000 7efeb29df740 1 -- 172.21.15.119:0/2138964544 wait complete. 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 -- start start 2024-03-28T18:51:55.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 Processor -- start 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96af1bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b4e00 con 0x55e96aea8080 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b47a0 con 0x55e96aeb1580 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea81b1cc0 con 0x55e96aea89a0 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeb29df740 10 client.0 _ll_register_callbacks cb 0x7efe9401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efead400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeaca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efead400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55152/0 (socket says 172.21.15.119:55152) 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 -- 172.21.15.119:0/260304619 learned_addr learned my addr 172.21.15.119:0/260304619 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.287+0000 7efeaca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49714/0 (socket says 172.21.15.119:49714) 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efeade00640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96af1bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 -- 172.21.15.119:0/260304619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 msgr2=0x55e96aeab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 -- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 msgr2=0x55e96af1bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96af1bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 -- 172.21.15.119:0/260304619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e96af1c110 con 0x55e96aea8080 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efeaca00640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efeade00640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96af1bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efead400640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe9c098100 tx=0x7efe9c005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dba12fe2fa986b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efe8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:51:55.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c0b4060 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7efe9c099a80 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.288+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efea0033830 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe9c09be80 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 10 client.17547 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e96af1d2b0 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efea0033830 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7efe9c097040 con 0x55e96aea8080 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 10 client.17547 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 10 client.17547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 20 client.17547 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96af1d910 0x55e96af1fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 4 client.17547 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.289+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e96ae14740 con 0x55e96af1d910 2024-03-28T18:51:55.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.290+0000 7efeaca00640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96af1d910 0x55e96af1fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.290+0000 7efeaca00640 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96af1d910 0x55e96af1fd30 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc8a47204260ff7b server_cookie=fc599848cad73516 in_seq=0 out_seq=0 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.290+0000 7efe8ea00640 10 client.17547 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efe8ea00640 1 -- 172.21.15.119:0/260304619 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e96ae14740 con 0x55e96af1d910 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efe8ea00640 10 client.17547 handle_command_reply: tid=0 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efeb29df740 1 client.17547 shutdown 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 msgr2=0x55e96aeaac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efe9c098100 tx=0x7efe9c005c80 comp rx=0 tx=0).stop 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.291+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 shutdown_connections 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e96af1d910 0x55e96af1fd30 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e96aeb1580 0x55e96af1bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e96aea89a0 0x55e96aeab150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 --2- 172.21.15.119:0/260304619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e96aea8080 0x55e96aeaac10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 >> 172.21.15.119:0/260304619 conn(0x55e96aea27b0 msgr2=0x55e96aea4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efe8ea00640 0 client.17547 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 shutdown_connections 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 1 -- 172.21.15.119:0/260304619 wait complete. 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeb29df740 20 client.17547 trim_cache size 0 max 16384 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "35fc76bd-5da1-4e6a-97b4-c2778f87551a", 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:51:55.297 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 msgr2=0x7efe900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 0x7efe900a4eb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efea002c430 tx=0x7efea0005fd0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 msgr2=0x7efea819e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.292+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9c0 tx=0x7efe9c00ee80 comp rx=0 tx=0).stop 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 shutdown_connections 2024-03-28T18:51:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efe900a29f0 0x7efe900a4eb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efea81124d0 0x7efea81a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea80764f0 0x7efea819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 --2- 172.21.15.119:0/4152117755 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efea8075bd0 0x7efea819e600 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 >> 172.21.15.119:0/4152117755 conn(0x7efea80ff320 msgr2=0x7efea8113bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 shutdown_connections 2024-03-28T18:51:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.293+0000 7efeafe00640 1 -- 172.21.15.119:0/4152117755 wait complete. 2024-03-28T18:51:55.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:51:55.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 -- 172.21.15.119:0/1986099315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280102750 msgr2=0x7fd280102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 --2- 172.21.15.119:0/1986099315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280102750 0x7fd280102b30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd270009ca0 tx=0x7fd27002c190 comp rx=0 tx=0).stop 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 -- 172.21.15.119:0/1986099315 shutdown_connections 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 --2- 172.21.15.119:0/1986099315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 0x7fd28010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 --2- 172.21.15.119:0/1986099315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280103070 0x7fd28010b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 --2- 172.21.15.119:0/1986099315 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280102750 0x7fd280102b30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 -- 172.21.15.119:0/1986099315 >> 172.21.15.119:0/1986099315 conn(0x7fd2800fbf50 msgr2=0x7fd2800fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 -- 172.21.15.119:0/1986099315 shutdown_connections 2024-03-28T18:51:55.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.523+0000 7fd286400640 1 -- 172.21.15.119:0/1986099315 wait complete. 2024-03-28T18:51:55.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.524+0000 7fd286400640 1 Processor -- start 2024-03-28T18:51:55.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.524+0000 7fd286400640 1 -- start start 2024-03-28T18:51:55.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280102750 0x7fd2801003a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 0x7fd280100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd280111930 con 0x7fd280102750 2024-03-28T18:51:55.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801117b0 con 0x7fd28010bb70 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd286400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd280111ab0 con 0x7fd280103070 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd285000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280102750 0x7fd2801003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49738/0 (socket says 172.21.15.119:49738) 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 -- 172.21.15.119:0/1095756211 learned_addr learned my addr 172.21.15.119:0/1095756211 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd285a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 0x7fd280100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 -- 172.21.15.119:0/1095756211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 msgr2=0x7fd280100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 0x7fd280100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 -- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280102750 msgr2=0x7fd2801003a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280102750 0x7fd2801003a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.525+0000 7fd27fe00640 1 -- 172.21.15.119:0/1095756211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2801a63b0 con 0x7fd280103070 2024-03-28T18:51:55.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd27fe00640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd26800e9f0 tx=0x7fd26800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd27d600640 1 -- 172.21.15.119:0/1095756211 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26800cd80 con 0x7fd280103070 2024-03-28T18:51:55.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd27d600640 1 -- 172.21.15.119:0/1095756211 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2680035f0 con 0x7fd280103070 2024-03-28T18:51:55.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2801a66a0 con 0x7fd280103070 2024-03-28T18:51:55.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd27d600640 1 -- 172.21.15.119:0/1095756211 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd268010670 con 0x7fd280103070 2024-03-28T18:51:55.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.526+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2801a6bf0 con 0x7fd280103070 2024-03-28T18:51:55.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.530+0000 7fd27d600640 1 -- 172.21.15.119:0/1095756211 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fd2680037a0 con 0x7fd280103070 2024-03-28T18:51:55.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.530+0000 7fd28a603740 1 Processor -- start 2024-03-28T18:51:55.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 -- start start 2024-03-28T18:51:55.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e3201d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32028850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e31f81250 con 0x556e32026460 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e31f29a30 con 0x556e3201d0f0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd28a603740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e31f7d2f0 con 0x556e3201ddd0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49746/0 (socket says 172.21.15.119:49746) 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285a00640 1 -- 172.21.15.119:0/3423815302 learned_addr learned my addr 172.21.15.119:0/3423815302 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285000640 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e3201d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd27fe00640 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32028850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285a00640 1 -- 172.21.15.119:0/3423815302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 msgr2=0x556e3201d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.531+0000 7fd285a00640 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e3201d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd285a00640 1 -- 172.21.15.119:0/3423815302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 msgr2=0x556e32028850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd285a00640 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32028850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd285a00640 1 -- 172.21.15.119:0/3423815302 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e31f14690 con 0x556e3201ddd0 2024-03-28T18:51:55.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd285a00640 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd278009c20 tx=0x7fd278011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c80439a7fad51b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd27d600640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 0x7fd2640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd276000640 1 -- 172.21.15.119:0/3423815302 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27801e070 con 0x556e3201ddd0 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd276000640 1 -- 172.21.15.119:0/3423815302 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd278019b10 con 0x556e3201ddd0 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd276000640 1 -- 172.21.15.119:0/3423815302 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27800eae0 con 0x556e3201ddd0 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd27d600640 1 -- 172.21.15.119:0/1095756211 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd2680cf220 con 0x7fd280103070 2024-03-28T18:51:55.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.532+0000 7fd285000640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 0x7fd2640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 -- 172.21.15.119:0/3423815302 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 msgr2=0x556e3201cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd278009c20 tx=0x7fd278011810 comp rx=0 tx=0).stop 2024-03-28T18:51:55.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 -- 172.21.15.119:0/3423815302 shutdown_connections 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32028850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e3201d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 --2- 172.21.15.119:0/3423815302 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e3201cb20 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 -- 172.21.15.119:0/3423815302 >> 172.21.15.119:0/3423815302 conn(0x556e31fe4200 msgr2=0x556e32017b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 -- 172.21.15.119:0/3423815302 shutdown_connections 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd285000640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 0x7fd2640a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd270000f80 tx=0x7fd270009360 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.533+0000 7fd28a603740 1 -- 172.21.15.119:0/3423815302 wait complete. 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 -- start start 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 Processor -- start 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 0x556e32025b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e320292d0 con 0x556e3201d0f0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e31f3ce80 con 0x556e32026460 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e32029450 con 0x556e3201ddd0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.534+0000 7fd28a603740 10 client.0 _ll_register_callbacks cb 0x556e31f4f130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 0x556e32025b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51600/0 (socket says 172.21.15.119:51600) 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49756/0 (socket says 172.21.15.119:49756) 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 -- 172.21.15.119:0/2966906941 learned_addr learned my addr 172.21.15.119:0/2966906941 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285000640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 0x556e32025b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55172/0 (socket says 172.21.15.119:55172) 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 -- 172.21.15.119:0/2966906941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 msgr2=0x556e32022f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 -- 172.21.15.119:0/2966906941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 msgr2=0x556e32025b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 0x556e32025b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 -- 172.21.15.119:0/2966906941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e31d5f7d0 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd27fe00640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd285a00640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd278011d10 tx=0x7fd27800e440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73862e310ea2bde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.535+0000 7fd26ea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278027070 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd278019840 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd278019b10 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd278032050 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 10 client.27803 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556e31c953f0 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd278019b10 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd27800e7e0 con 0x556e32026460 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 10 client.27803 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 10 client.27803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.536+0000 7fd28a603740 20 client.27803 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32036230 0x556e32038620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd28a603740 4 client.27803 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:51:55.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e31f8e790 con 0x556e32036230 2024-03-28T18:51:55.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd285000640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32036230 0x556e32038620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd285000640 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32036230 0x556e32038620 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f94c35280f3ea76a server_cookie=38d7925abe359ce9 in_seq=0 out_seq=0 2024-03-28T18:51:55.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.537+0000 7fd26ea00640 10 client.27803 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.538+0000 7fd26ea00640 1 -- 172.21.15.119:0/2966906941 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556e31f8e790 con 0x556e32036230 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.538+0000 7fd26ea00640 10 client.27803 handle_command_reply: tid=0 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 client.27803 shutdown 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 msgr2=0x556e32023450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd278011d10 tx=0x7fd27800e440 comp rx=0 tx=0).stop 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 shutdown_connections 2024-03-28T18:51:55.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32036230 0x556e32038620 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e32026460 0x556e32023450 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32022f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd26ea00640 0 client.27803 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 --2- 172.21.15.119:0/2966906941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e3201d0f0 0x556e32025b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 >> 172.21.15.119:0/2966906941 conn(0x556e31fe4200 msgr2=0x556e32019890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 shutdown_connections 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 1 -- 172.21.15.119:0/2966906941 wait complete. 2024-03-28T18:51:55.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.539+0000 7fd28a603740 20 client.27803 trim_cache size 0 max 16384 2024-03-28T18:51:55.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.556+0000 7fd28a603740 1 Processor -- start 2024-03-28T18:51:55.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 -- start start 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32031ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801af4a0 con 0x556e32026460 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801b0e20 con 0x556e3201ddd0 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd28a603740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801afb00 con 0x556e3201d0f0 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd285000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd285000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:49772/0 (socket says 172.21.15.119:49772) 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd285000640 1 -- 172.21.15.119:0/1691676342 learned_addr learned my addr 172.21.15.119:0/1691676342 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd27fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51608/0 (socket says 172.21.15.119:51608) 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.557+0000 7fd285a00640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32031ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 -- 172.21.15.119:0/1691676342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 msgr2=0x556e32025b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 -- 172.21.15.119:0/1691676342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 msgr2=0x556e32031ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32031ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 -- 172.21.15.119:0/1691676342 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e31c550f0 con 0x556e3201d0f0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd27fe00640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285a00640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32031ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd285000640 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd270009ca0 tx=0x7fd270041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98794fd726eec2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd275600640 1 -- 172.21.15.119:0/1691676342 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270039040 con 0x556e3201d0f0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd275600640 1 -- 172.21.15.119:0/1691676342 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2700552f0 con 0x556e3201d0f0 2024-03-28T18:51:55.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.558+0000 7fd275600640 1 -- 172.21.15.119:0/1691676342 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2700346a0 con 0x556e3201d0f0 2024-03-28T18:51:55.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- 172.21.15.119:0/1691676342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 msgr2=0x556e32025640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd270009ca0 tx=0x7fd270041000 comp rx=0 tx=0).stop 2024-03-28T18:51:55.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- 172.21.15.119:0/1691676342 shutdown_connections 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e32031ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201ddd0 0x556e32025b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 --2- 172.21.15.119:0/1691676342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201d0f0 0x556e32025640 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- 172.21.15.119:0/1691676342 >> 172.21.15.119:0/1691676342 conn(0x556e31fe4200 msgr2=0x556e32022ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- 172.21.15.119:0/1691676342 shutdown_connections 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- 172.21.15.119:0/1691676342 wait complete. 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.559+0000 7fd28a603740 1 -- start start 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 Processor -- start 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32020190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e320206d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801b07c0 con 0x556e32026460 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801b0160 con 0x556e3201d0f0 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2801ad680 con 0x556e3201ddd0 2024-03-28T18:51:55.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd28a603740 10 client.0 _ll_register_callbacks cb 0x7fd27801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd285000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd285000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:51616/0 (socket says 172.21.15.119:51616) 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd285000640 1 -- 172.21.15.119:0/386137443 learned_addr learned my addr 172.21.15.119:0/386137443 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.560+0000 7fd27fe00640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32020190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285a00640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e320206d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 -- 172.21.15.119:0/386137443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 msgr2=0x556e32020190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32020190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 -- 172.21.15.119:0/386137443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 msgr2=0x556e320206d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e320206d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 -- 172.21.15.119:0/386137443 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e32020cc0 con 0x556e3201d0f0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd285000640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd27002c7a0 tx=0x7fd27002c190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e14b94d5cd05ff72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd26ea00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270063060 con 0x556e3201d0f0 2024-03-28T18:51:55.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd2700554b0 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.561+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556e31f88ae0 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd27006a050 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 10 client.27809 fetch_fsmap learned FSMap version 28 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556e32027620 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556e31f88ae0 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd270034ba0 con 0x556e3201d0f0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 10 client.27809 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 10 client.27809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 20 client.27809 populate_metadata read hostname 'smithi119' 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32091d40 0x556e32094160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 4 client.27809 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556e31f063a0 con 0x556e32091d40 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.562+0000 7fd27fe00640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32091d40 0x556e32094160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd27fe00640 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32091d40 0x556e32094160 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b693c825e6261574 server_cookie=88fcb078675b0464 in_seq=0 out_seq=0 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd26ea00640 10 client.27809 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd26ea00640 1 -- 172.21.15.119:0/386137443 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556e31f063a0 con 0x556e32091d40 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd26ea00640 10 client.27809 handle_command_reply: tid=0 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 client.27809 shutdown 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 msgr2=0x556e32027d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd27002c7a0 tx=0x7fd27002c190 comp rx=0 tx=0).stop 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 shutdown_connections 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x556e32091d40 0x556e32094160 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e32026460 0x556e320206d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556e3201ddd0 0x556e32020190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 --2- 172.21.15.119:0/386137443 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x556e3201d0f0 0x556e32027d10 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 >> 172.21.15.119:0/386137443 conn(0x556e31fe4200 msgr2=0x556e32022eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd26ea00640 0 client.27809 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 shutdown_connections 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.563+0000 7fd28a603740 1 -- 172.21.15.119:0/386137443 wait complete. 2024-03-28T18:51:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd28a603740 20 client.27809 trim_cache size 0 max 16384 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "35fc76bd-5da1-4e6a-97b4-c2778f87551a": { 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "35fc76bd-5da1-4e6a-97b4-c2778f87551a", 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:51:55.569 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 msgr2=0x7fd2640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 0x7fd2640a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd270000f80 tx=0x7fd270009360 comp rx=0 tx=0).stop 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 msgr2=0x7fd2801008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd26800e9f0 tx=0x7fd26800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 shutdown_connections 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd2640a2970 0x7fd2640a4e30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd28010bb70 0x7fd280100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd280103070 0x7fd2801008e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 --2- 172.21.15.119:0/1095756211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd280102750 0x7fd2801003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:51:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 >> 172.21.15.119:0/1095756211 conn(0x7fd2800fbf50 msgr2=0x7fd2800fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:51:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 shutdown_connections 2024-03-28T18:51:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:51:55.564+0000 7fd286400640 1 -- 172.21.15.119:0/1095756211 wait complete. 2024-03-28T18:51:55.581 INFO:tasks.cephfs.filesystem:scrub status for tag:35fc76bd-5da1-4e6a-97b4-c2778f87551a - {'path': '/', 'tag': '35fc76bd-5da1-4e6a-97b4-c2778f87551a', 'options': 'recursive,force'} 2024-03-28T18:51:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:55 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:51:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:55 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:55 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:51:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:55 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:55.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:55 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:51:55.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:55 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:55.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:55 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:51:55.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:55 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:55 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:51:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:55 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:55 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:51:55.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:55 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:51:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:56 smithi119 ceph-mon[31813]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:56 smithi196 ceph-mon[36300]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:56 smithi131 ceph-mon[36065]: pgmap v241: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:51:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:58 smithi119 ceph-mon[31813]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:51:58 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:51:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:58 smithi196 ceph-mon[36300]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:51:58 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:51:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:58 smithi131 ceph-mon[36065]: pgmap v242: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:51:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:51:58 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:00 smithi119 ceph-mon[31813]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:00 smithi196 ceph-mon[36300]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:00 smithi131 ceph-mon[36065]: pgmap v243: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:02 smithi119 ceph-mon[31813]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:02 smithi196 ceph-mon[36300]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:02 smithi131 ceph-mon[36065]: pgmap v244: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:04 smithi119 ceph-mon[31813]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:04 smithi196 ceph-mon[36300]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:04 smithi131 ceph-mon[36065]: pgmap v245: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:06.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:06 smithi119 ceph-mon[31813]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:52:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:06 smithi196 ceph-mon[36300]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:52:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:06 smithi131 ceph-mon[36065]: pgmap v246: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:52:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:08 smithi119 ceph-mon[31813]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:52:08.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:08 smithi196 ceph-mon[36300]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:52:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:08 smithi131 ceph-mon[36065]: pgmap v247: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:52:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:09 smithi119 ceph-mon[31813]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:52:09.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:09 smithi196 ceph-mon[36300]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:52:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:09 smithi131 ceph-mon[36065]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T18:52:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:10 smithi119 ceph-mon[31813]: pgmap v249: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T18:52:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:10 smithi119 ceph-mon[31813]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:52:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:10 smithi196 ceph-mon[36300]: pgmap v249: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T18:52:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:10 smithi196 ceph-mon[36300]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:52:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:10 smithi131 ceph-mon[36065]: pgmap v249: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T18:52:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:10 smithi131 ceph-mon[36065]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T18:52:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:11 smithi119 ceph-mon[31813]: pgmap v251: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:52:11.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:11 smithi196 ceph-mon[36300]: pgmap v251: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:52:11.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:11 smithi131 ceph-mon[36065]: pgmap v251: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-28T18:52:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:14 smithi119 ceph-mon[31813]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:52:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:14 smithi196 ceph-mon[36300]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:52:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:14 smithi131 ceph-mon[36065]: pgmap v252: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T18:52:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:16 smithi119 ceph-mon[31813]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:52:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:16 smithi196 ceph-mon[36300]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:52:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:16 smithi131 ceph-mon[36065]: pgmap v253: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T18:52:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:18 smithi119 ceph-mon[31813]: pgmap v254: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:52:18.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:18 smithi196 ceph-mon[36300]: pgmap v254: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:52:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:18 smithi131 ceph-mon[36065]: pgmap v254: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2024-03-28T18:52:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:20 smithi119 ceph-mon[31813]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:52:20.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:20 smithi196 ceph-mon[36300]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:52:20.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:20 smithi131 ceph-mon[36065]: pgmap v255: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:52:22.745 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:22 smithi131 ceph-mon[36065]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:52:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:22 smithi119 ceph-mon[31813]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:52:22.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:22 smithi196 ceph-mon[36300]: pgmap v256: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:52:23.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:23 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:23.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:23 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:23.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:23 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:52:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:24 smithi119 ceph-mon[31813]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:52:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:24 smithi196 ceph-mon[36300]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:52:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:24 smithi131 ceph-mon[36065]: pgmap v257: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:52:25.583 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:52:25.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:52:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 -- 172.21.15.119:0/243547822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518073a40 msgr2=0x7fc518073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 --2- 172.21.15.119:0/243547822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518073a40 0x7fc518073e20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc508009900 tx=0x7fc50802bcd0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 -- 172.21.15.119:0/243547822 shutdown_connections 2024-03-28T18:52:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 --2- 172.21.15.119:0/243547822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 0x7fc518114a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 --2- 172.21.15.119:0/243547822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518074360 0x7fc518112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 --2- 172.21.15.119:0/243547822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518073a40 0x7fc518073e20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 -- 172.21.15.119:0/243547822 >> 172.21.15.119:0/243547822 conn(0x7fc518078aa0 msgr2=0x7fc518078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 -- 172.21.15.119:0/243547822 shutdown_connections 2024-03-28T18:52:25.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.809+0000 7fc51fe00640 1 -- 172.21.15.119:0/243547822 wait complete. 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.810+0000 7fc51fe00640 1 Processor -- start 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.810+0000 7fc51fe00640 1 -- start start 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518073a40 0x7fc51819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 0x7fc5181a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518118090 con 0x7fc518073a40 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518117f10 con 0x7fc518112640 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518118210 con 0x7fc518074360 2024-03-28T18:52:25.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518073a40 0x7fc51819e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53508/0 (socket says 172.21.15.119:53508) 2024-03-28T18:52:25.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51ca00640 1 -- 172.21.15.119:0/2101477400 learned_addr learned my addr 172.21.15.119:0/2101477400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:25.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.811+0000 7fc51de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 0x7fc5181a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 -- 172.21.15.119:0/2101477400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 msgr2=0x7fc5181a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 0x7fc5181a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 -- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518073a40 msgr2=0x7fc51819e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518073a40 0x7fc51819e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 -- 172.21.15.119:0/2101477400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5181a3500 con 0x7fc518074360 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51ca00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc50400e9f0 tx=0x7fc50400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc516000640 1 -- 172.21.15.119:0/2101477400 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50400cd80 con 0x7fc518074360 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc516000640 1 -- 172.21.15.119:0/2101477400 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc504010910 con 0x7fc518074360 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc516000640 1 -- 172.21.15.119:0/2101477400 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504018990 con 0x7fc518074360 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5181a37f0 con 0x7fc518074360 2024-03-28T18:52:25.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.812+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5181ab080 con 0x7fc518074360 2024-03-28T18:52:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.815+0000 7fc522c2a740 1 Processor -- start 2024-03-28T18:52:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 -- start start 2024-03-28T18:52:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc516000640 1 -- 172.21.15.119:0/2101477400 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc504010ac0 con 0x7fc518074360 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350af950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b9290 0x55f2350bb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235014250 con 0x55f2350b9290 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f234fbca30 con 0x55f2350b0c00 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc522c2a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2350102f0 con 0x55f2350aff20 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350af950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55750/0 (socket says 172.21.15.119:55750) 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53518/0 (socket says 172.21.15.119:53518) 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51de00640 1 -- 172.21.15.119:0/969649184 learned_addr learned my addr 172.21.15.119:0/969649184 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.816+0000 7fc51ca00640 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b9290 0x55f2350bb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 -- 172.21.15.119:0/969649184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 msgr2=0x55f2350af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 -- 172.21.15.119:0/969649184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b9290 msgr2=0x55f2350bb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b9290 0x55f2350bb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 -- 172.21.15.119:0/969649184 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f234fa7690 con 0x55f2350aff20 2024-03-28T18:52:25.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc51d400640 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc50802c210 tx=0x7fc5080095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24dbc65e9e0a5bec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc50e600640 1 -- 172.21.15.119:0/969649184 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508043070 con 0x55f2350aff20 2024-03-28T18:52:25.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc50e600640 1 -- 172.21.15.119:0/969649184 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50802c920 con 0x55f2350aff20 2024-03-28T18:52:25.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc50e600640 1 -- 172.21.15.119:0/969649184 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc508004ac0 con 0x55f2350aff20 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.817+0000 7fc516000640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 0x7fc4fc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc516000640 1 -- 172.21.15.119:0/2101477400 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc5040cf1f0 con 0x7fc518074360 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc51de00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 0x7fc4fc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 -- 172.21.15.119:0/969649184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 msgr2=0x55f2350b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc50802c210 tx=0x7fc5080095e0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 -- 172.21.15.119:0/969649184 shutdown_connections 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b9290 0x55f2350bb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b03a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 --2- 172.21.15.119:0/969649184 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 -- 172.21.15.119:0/969649184 >> 172.21.15.119:0/969649184 conn(0x55f235077200 msgr2=0x55f2350aa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 -- 172.21.15.119:0/969649184 shutdown_connections 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc51de00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 0x7fc4fc0a4eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc510013520 tx=0x7fc510009600 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.818+0000 7fc522c2a740 1 -- 172.21.15.119:0/969649184 wait complete. 2024-03-28T18:52:25.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 -- start start 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 Processor -- start 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2350bc0a0 con 0x55f2350aff20 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f234fcfe80 con 0x55f2350b0c00 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2350bc220 con 0x55f2350b9290 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.819+0000 7fc522c2a740 10 client.0 _ll_register_callbacks cb 0x55f234fe2130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53534/0 (socket says 172.21.15.119:53534) 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 -- 172.21.15.119:0/723413664 learned_addr learned my addr 172.21.15.119:0/723413664 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55752/0 (socket says 172.21.15.119:55752) 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 -- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 msgr2=0x55f2350b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 -- 172.21.15.119:0/723413664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 msgr2=0x55f2350b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 -- 172.21.15.119:0/723413664 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f234df27d0 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc51de00640 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc510011d40 tx=0x7fc510004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f2fbe087210475 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.820+0000 7fc502a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510020620 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc50802c920 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc5100207c0 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5100209c0 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 10 client.27796 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f234d283f0 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50802c920 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc51000f040 con 0x55f2350b9290 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 10 client.27796 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 10 client.27796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 20 client.27796 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:25.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f2350c9100 0x55f2350cb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 4 client.27796 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:25.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.821+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f235021790 con 0x55f2350c9100 2024-03-28T18:52:25.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.822+0000 7fc51d400640 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f2350c9100 0x55f2350cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.822+0000 7fc51d400640 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f2350c9100 0x55f2350cb510 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad7b4b95bd5418da server_cookie=74eaffb68cca0e6d in_seq=0 out_seq=0 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.822+0000 7fc502a00640 10 client.27796 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc502a00640 1 -- 172.21.15.119:0/723413664 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f235021790 con 0x55f2350c9100 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc502a00640 10 client.27796 handle_command_reply: tid=0 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 client.27796 shutdown 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 msgr2=0x55f2350b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc510011d40 tx=0x7fc510004a40 comp rx=0 tx=0).stop 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 shutdown_connections 2024-03-28T18:52:25.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f2350c9100 0x55f2350cb510 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350b6260 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc502a00640 0 client.27796 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.823+0000 7fc522c2a740 1 --2- 172.21.15.119:0/723413664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.824+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 >> 172.21.15.119:0/723413664 conn(0x55f235077200 msgr2=0x55f2350ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.824+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 shutdown_connections 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.824+0000 7fc522c2a740 1 -- 172.21.15.119:0/723413664 wait complete. 2024-03-28T18:52:25.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.824+0000 7fc522c2a740 20 client.27796 trim_cache size 0 max 16384 2024-03-28T18:52:25.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 Processor -- start 2024-03-28T18:52:25.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 -- start start 2024-03-28T18:52:25.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350ba680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b39d0 con 0x55f2350aff20 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b5350 con 0x55f2350b0c00 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.841+0000 7fc522c2a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b4030 con 0x55f2350b9290 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47762/0 (socket says 172.21.15.119:47762) 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51d400640 1 -- 172.21.15.119:0/169418841 learned_addr learned my addr 172.21.15.119:0/169418841 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51de00640 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 -- 172.21.15.119:0/169418841 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 msgr2=0x55f2350ba680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350ba680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 -- 172.21.15.119:0/169418841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 msgr2=0x55f2350b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 -- 172.21.15.119:0/169418841 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f234ce80f0 con 0x55f2350b0c00 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51d400640 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.842+0000 7fc51ca00640 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b8990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc5040cc850 tx=0x7fc504000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c448f79193ca5e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc50dc00640 1 -- 172.21.15.119:0/169418841 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50409b810 con 0x55f2350b0c00 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc50dc00640 1 -- 172.21.15.119:0/169418841 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50409f8b0 con 0x55f2350b0c00 2024-03-28T18:52:25.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc50dc00640 1 -- 172.21.15.119:0/169418841 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504018b30 con 0x55f2350b0c00 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 -- 172.21.15.119:0/169418841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 msgr2=0x55f2350b8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b8990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc5040cc850 tx=0x7fc504000bc0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 -- 172.21.15.119:0/169418841 shutdown_connections 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350b9290 0x55f2350ba680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b0c00 0x55f2350b8990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 --2- 172.21.15.119:0/169418841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350aff20 0x55f2350b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 -- 172.21.15.119:0/169418841 >> 172.21.15.119:0/169418841 conn(0x55f235077200 msgr2=0x55f2350aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 -- 172.21.15.119:0/169418841 shutdown_connections 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.843+0000 7fc522c2a740 1 -- 172.21.15.119:0/169418841 wait complete. 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 -- start start 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 Processor -- start 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b0c00 0x55f2350b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b4cf0 con 0x55f2350b0c00 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b4690 con 0x55f2350b9290 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5181b1bb0 con 0x55f2350aff20 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.844+0000 7fc522c2a740 10 client.0 _ll_register_callbacks cb 0x7fc510009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55768/0 (socket says 172.21.15.119:55768) 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 -- 172.21.15.119:0/2398406198 learned_addr learned my addr 172.21.15.119:0/2398406198 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51ca00640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b0c00 0x55f2350b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51d400640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 -- 172.21.15.119:0/2398406198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 msgr2=0x55f2350b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 -- 172.21.15.119:0/2398406198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b0c00 msgr2=0x55f2350b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b0c00 0x55f2350b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 -- 172.21.15.119:0/2398406198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2350b7d40 con 0x55f2350b9290 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc51de00640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc51001dca0 tx=0x7fc510016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a9608cd760168 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.845+0000 7fc502a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:52:25.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51003b030 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc510003b80 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc50409f8b0 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc510045050 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 10 client.27827 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f235125190 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc50409f8b0 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc51000f040 con 0x55f2350b9290 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 10 client.27827 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 10 client.27827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 20 client.27827 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f235125910 0x55f235127d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 4 client.27827 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.846+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f23501bae0 con 0x55f235125910 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc51d400640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f235125910 0x55f235127d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc51d400640 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f235125910 0x55f235127d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b2e6933db6842f server_cookie=c49a1571ad86a176 in_seq=0 out_seq=0 2024-03-28T18:52:25.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc502a00640 10 client.27827 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc502a00640 1 -- 172.21.15.119:0/2398406198 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f23501bae0 con 0x55f235125910 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc502a00640 10 client.27827 handle_command_reply: tid=0 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc522c2a740 1 client.27827 shutdown 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 msgr2=0x55f2350b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.847+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc51001dca0 tx=0x7fc510016000 comp rx=0 tx=0).stop 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 shutdown_connections 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55f235125910 0x55f235127d30 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f2350b9290 0x55f2350b7800 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f2350b0c00 0x55f2350b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 --2- 172.21.15.119:0/2398406198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f2350aff20 0x55f2350b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc502a00640 0 client.27827 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 >> 172.21.15.119:0/2398406198 conn(0x55f235077200 msgr2=0x55f2350abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 shutdown_connections 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 1 -- 172.21.15.119:0/2398406198 wait complete. 2024-03-28T18:52:25.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc522c2a740 20 client.27827 trim_cache size 0 max 16384 2024-03-28T18:52:25.852 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:52:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:52:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:52:25.852 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 msgr2=0x7fc4fc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 0x7fc4fc0a4eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc510013520 tx=0x7fc510009600 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 msgr2=0x7fc51819ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.848+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc50400e9f0 tx=0x7fc50400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 shutdown_connections 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc4fc0a29f0 0x7fc4fc0a4eb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc518112640 0x7fc5181a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc518074360 0x7fc51819ea50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 --2- 172.21.15.119:0/2101477400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc518073a40 0x7fc51819e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 >> 172.21.15.119:0/2101477400 conn(0x7fc518078aa0 msgr2=0x7fc51810e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 shutdown_connections 2024-03-28T18:52:25.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:25.849+0000 7fc51fe00640 1 -- 172.21.15.119:0/2101477400 wait complete. 2024-03-28T18:52:25.866 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:52:25.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:52:25.943 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:25 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:25.943 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:25 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:25.943 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:25 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:52:25.943 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:25 smithi119 ceph-mon[31813]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 -- 172.21.15.119:0/1441507233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 msgr2=0x7f3600109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 --2- 172.21.15.119:0/1441507233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f3600109c40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009930 tx=0x7f35f402bc20 comp rx=0 tx=0).stop 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 -- 172.21.15.119:0/1441507233 shutdown_connections 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 --2- 172.21.15.119:0/1441507233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f3600110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 --2- 172.21.15.119:0/1441507233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 0x7f3600103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 --2- 172.21.15.119:0/1441507233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f3600109c40 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 -- 172.21.15.119:0/1441507233 >> 172.21.15.119:0/1441507233 conn(0x7f36000ff2e0 msgr2=0x7f3600101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 -- 172.21.15.119:0/1441507233 shutdown_connections 2024-03-28T18:52:26.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.075+0000 7f3608400640 1 -- 172.21.15.119:0/1441507233 wait complete. 2024-03-28T18:52:26.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 Processor -- start 2024-03-28T18:52:26.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 -- start start 2024-03-28T18:52:26.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 0x7f360019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600113cc0 con 0x7f3600103850 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600113b40 con 0x7f3600104210 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3608400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600113e40 con 0x7f3600109860 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3605000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3606400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3606400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53574/0 (socket says 172.21.15.119:53574) 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3605000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55786/0 (socket says 172.21.15.119:55786) 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3606400640 1 -- 172.21.15.119:0/1511407377 learned_addr learned my addr 172.21.15.119:0/1511407377 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.076+0000 7f3605a00640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 0x7f360019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 -- 172.21.15.119:0/1511407377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 msgr2=0x7f360019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 -- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 msgr2=0x7f360019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 0x7f360019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 -- 172.21.15.119:0/1511407377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360019f190 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3605000640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3606400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f35f800eb10 tx=0x7f35f800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8004b20 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360019f480 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35f8004cc0 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.077+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8010760 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.078+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36001a6d10 con 0x7f3600109860 2024-03-28T18:52:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.079+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3600068490 con 0x7f3600109860 2024-03-28T18:52:26.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.081+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f35f8005000 con 0x7f3600109860 2024-03-28T18:52:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.082+0000 7f35fe000640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 0x7f35e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.082+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f35f80cf900 con 0x7f3600109860 2024-03-28T18:52:26.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.082+0000 7f3605a00640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 0x7f35e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.083+0000 7f3605a00640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 0x7f35e80a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009900 tx=0x7f35f4005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.089+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f35f809c570 con 0x7f3600109860 2024-03-28T18:52:26.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:25 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:26.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:25 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:26.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:25 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:52:26.185 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:25 smithi196 ceph-mon[36300]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:52:26.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:25 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:52:26.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:25 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:52:26.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:25 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:52:26.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:25 smithi131 ceph-mon[36065]: pgmap v258: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:52:26.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.328+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f360002d070 con 0x7f3600109860 2024-03-28T18:52:26.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.329+0000 7f35fe000640 1 -- 172.21.15.119:0/1511407377 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f35f809be70 con 0x7f3600109860 2024-03-28T18:52:26.329 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:52:26.329 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:52:26.330 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 msgr2=0x7f35e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 0x7f35e80a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009900 tx=0x7f35f4005a20 comp rx=0 tx=0).stop 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 msgr2=0x7f360019ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f35f800eb10 tx=0x7f35f800c2d0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 shutdown_connections 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35e80a2970 0x7f35e80a4e30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3600109860 0x7f360019ea60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f3600104210 0x7f360019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 --2- 172.21.15.119:0/1511407377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3600103850 0x7f360019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 >> 172.21.15.119:0/1511407377 conn(0x7f36000ff2e0 msgr2=0x7f36000ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 shutdown_connections 2024-03-28T18:52:26.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.332+0000 7f3608400640 1 -- 172.21.15.119:0/1511407377 wait complete. 2024-03-28T18:52:26.352 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:26.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 -- 172.21.15.119:0/83398229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c10de70 msgr2=0x7f840c1102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 --2- 172.21.15.119:0/83398229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c10de70 0x7f840c1102c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8404009ca0 tx=0x7f840402c180 comp rx=0 tx=0).stop 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 -- 172.21.15.119:0/83398229 shutdown_connections 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 --2- 172.21.15.119:0/83398229 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c10de70 0x7f840c1102c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 --2- 172.21.15.119:0/83398229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 0x7f840c10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 --2- 172.21.15.119:0/83398229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c073ad0 0x7f840c073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 -- 172.21.15.119:0/83398229 >> 172.21.15.119:0/83398229 conn(0x7f840c078aa0 msgr2=0x7f840c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 -- 172.21.15.119:0/83398229 shutdown_connections 2024-03-28T18:52:26.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.562+0000 7f8412400640 1 -- 172.21.15.119:0/83398229 wait complete. 2024-03-28T18:52:26.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.563+0000 7f8412400640 1 Processor -- start 2024-03-28T18:52:26.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 -- start start 2024-03-28T18:52:26.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 0x7f840c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 0x7f840c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c113c10 con 0x7f840c0743f0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c113a90 con 0x7f840c073ad0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8412400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c113d90 con 0x7f840c10de70 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f840b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f840b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55802/0 (socket says 172.21.15.119:55802) 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f840b400640 1 -- 172.21.15.119:0/2491957710 learned_addr learned my addr 172.21.15.119:0/2491957710 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f8402a00640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 0x7f840c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.564+0000 7f840be00640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 0x7f840c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 -- 172.21.15.119:0/2491957710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 msgr2=0x7f840c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 0x7f840c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 -- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 msgr2=0x7f840c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 0x7f840c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 -- 172.21.15.119:0/2491957710 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c19f0b0 con 0x7f840c073ad0 2024-03-28T18:52:26.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840be00640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 0x7f840c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f840b400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f8408c00640 1 -- 172.21.15.119:0/2491957710 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00cd80 con 0x7f840c073ad0 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f8408c00640 1 -- 172.21.15.119:0/2491957710 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83fc005500 con 0x7f840c073ad0 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c19f3a0 con 0x7f840c073ad0 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f8408c00640 1 -- 172.21.15.119:0/2491957710 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc010690 con 0x7f840c073ad0 2024-03-28T18:52:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.565+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c1a6c30 con 0x7f840c073ad0 2024-03-28T18:52:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.568+0000 7f8408c00640 1 -- 172.21.15.119:0/2491957710 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f83fc0046c0 con 0x7f840c073ad0 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 Processor -- start 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 -- start start 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac3bc00 0x56199ac3e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ab962c0 con 0x56199ac3bc00 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ab3ec90 con 0x56199ac33080 2024-03-28T18:52:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f841522c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ab92360 con 0x56199ac326d0 2024-03-28T18:52:26.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f840be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f840b400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f840be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53608/0 (socket says 172.21.15.119:53608) 2024-03-28T18:52:26.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f840be00640 1 -- 172.21.15.119:0/4197519178 learned_addr learned my addr 172.21.15.119:0/4197519178 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.569+0000 7f8402a00640 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac3bc00 0x56199ac3e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 -- 172.21.15.119:0/4197519178 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 msgr2=0x56199ac3b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 -- 172.21.15.119:0/4197519178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac3bc00 msgr2=0x56199ac3e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac3bc00 0x56199ac3e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 -- 172.21.15.119:0/4197519178 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56199ab63080 con 0x56199ac326d0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f8408c00640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 0x7f83f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f8408c00640 1 -- 172.21.15.119:0/2491957710 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f83fc0cf340 con 0x7f840c073ad0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840be00640 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f840402c790 tx=0x7f8404005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2111da49b217913 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f840b400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 0x7f83f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f8400c00640 1 -- 172.21.15.119:0/4197519178 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404043070 con 0x56199ac326d0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f8400c00640 1 -- 172.21.15.119:0/4197519178 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404003e50 con 0x56199ac326d0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.570+0000 7f8400c00640 1 -- 172.21.15.119:0/4197519178 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404039040 con 0x56199ac326d0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 -- 172.21.15.119:0/4197519178 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 msgr2=0x56199ac32ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f840b400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 0x7f83f00a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x56199ac321a0 tx=0x7f83fc0d2df0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f840402c790 tx=0x7f8404005f80 comp rx=0 tx=0).stop 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 -- 172.21.15.119:0/4197519178 shutdown_connections 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac3bc00 0x56199ac3e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 --2- 172.21.15.119:0/4197519178 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac32ab0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 -- 172.21.15.119:0/4197519178 >> 172.21.15.119:0/4197519178 conn(0x56199ac2cda0 msgr2=0x56199ac2d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 -- 172.21.15.119:0/4197519178 shutdown_connections 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.571+0000 7f841522c740 1 -- 172.21.15.119:0/4197519178 wait complete. 2024-03-28T18:52:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 -- start start 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 Processor -- start 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac3bc00 0x56199ac38d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ac3eb10 con 0x56199ac33080 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ab520d0 con 0x56199ac3bc00 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.572+0000 7f841522c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56199ac3ec90 con 0x56199ac326d0 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f841522c740 10 client.0 _ll_register_callbacks cb 0x56199ab56e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f840be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47822/0 (socket says 172.21.15.119:47822) 2024-03-28T18:52:26.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 -- 172.21.15.119:0/3534124713 learned_addr learned my addr 172.21.15.119:0/3534124713 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f840be00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac3bc00 0x56199ac38d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 -- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 msgr2=0x56199ac3b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 -- 172.21.15.119:0/3534124713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac3bc00 msgr2=0x56199ac38d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac3bc00 0x56199ac38d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 -- 172.21.15.119:0/3534124713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56199a8aa3f0 con 0x56199ac33080 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f840be00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f8402a00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f83f40126d0 tx=0x7f83f4012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937812879233147a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.573+0000 7f83f9600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4004050 con 0x56199ac33080 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f83f40041f0 con 0x56199ac33080 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8404003e50 con 0x56199ac33080 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83f40043f0 con 0x56199ac33080 2024-03-28T18:52:26.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f841522c740 10 client.17601 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56199a9747d0 con 0x56199ac33080 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404003e50 con 0x56199ac33080 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.574+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f83f4022030 con 0x56199ac33080 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 10 client.17601 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 10 client.17601 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 20 client.17601 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac4b9f0 0x56199ac4dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 4 client.17601 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56199aba3800 con 0x56199ac4b9f0 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f840be00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac4b9f0 0x56199ac4dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f840be00640 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac4b9f0 0x56199ac4dde0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=911502b93079d4d6 server_cookie=66cc6c6852923fc0 in_seq=0 out_seq=0 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.575+0000 7f83f9600640 10 client.17601 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.576+0000 7f83f9600640 1 -- 172.21.15.119:0/3534124713 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56199aba3800 con 0x56199ac4b9f0 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.576+0000 7f83f9600640 10 client.17601 handle_command_reply: tid=0 2024-03-28T18:52:26.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.576+0000 7f841522c740 1 client.17601 shutdown 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.576+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 msgr2=0x56199ac38820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.576+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f83f40126d0 tx=0x7f83f4012b90 comp rx=0 tx=0).stop 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 shutdown_connections 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac4b9f0 0x56199ac4dde0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac3bc00 0x56199ac38d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac33080 0x56199ac38820 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f83f9600640 0 client.17601 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 --2- 172.21.15.119:0/3534124713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 >> 172.21.15.119:0/3534124713 conn(0x56199ac2cda0 msgr2=0x56199ac2eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 shutdown_connections 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 1 -- 172.21.15.119:0/3534124713 wait complete. 2024-03-28T18:52:26.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.577+0000 7f841522c740 20 client.17601 trim_cache size 0 max 16384 2024-03-28T18:52:26.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.590+0000 7f841522c740 1 Processor -- start 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 -- start start 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 0x56199ac3af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac35a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1af5f0 con 0x56199ac2cda0 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1b0f70 con 0x56199ac33080 2024-03-28T18:52:26.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f841522c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1afc50 con 0x56199ac326d0 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f840be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 0x56199ac3af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f8402a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f8402a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53628/0 (socket says 172.21.15.119:53628) 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f840be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 0x56199ac3af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47836/0 (socket says 172.21.15.119:47836) 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f8402a00640 1 -- 172.21.15.119:0/331908800 learned_addr learned my addr 172.21.15.119:0/331908800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f840be00640 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac35a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f8402a00640 1 -- 172.21.15.119:0/331908800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 msgr2=0x56199ac35a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.591+0000 7f8402a00640 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac35a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f8402a00640 1 -- 172.21.15.119:0/331908800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 msgr2=0x56199ac3af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f8402a00640 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 0x56199ac3af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f8402a00640 1 -- 172.21.15.119:0/331908800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56199ac4e440 con 0x56199ac326d0 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f8402a00640 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83f4003cc0 tx=0x7f83f40040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63cd5accef119b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f83fbe00640 1 -- 172.21.15.119:0/331908800 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4010430 con 0x56199ac326d0 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f83fbe00640 1 -- 172.21.15.119:0/331908800 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83f400f920 con 0x56199ac326d0 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.592+0000 7f83fbe00640 1 -- 172.21.15.119:0/331908800 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f40052a0 con 0x56199ac326d0 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- 172.21.15.119:0/331908800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 msgr2=0x56199ac3b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83f4003cc0 tx=0x7f83f40040c0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- 172.21.15.119:0/331908800 shutdown_connections 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac35a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac326d0 0x56199ac3b490 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 --2- 172.21.15.119:0/331908800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac2cda0 0x56199ac3af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- 172.21.15.119:0/331908800 >> 172.21.15.119:0/331908800 conn(0x56199ac38ec0 msgr2=0x56199ac33cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- 172.21.15.119:0/331908800 shutdown_connections 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- 172.21.15.119:0/331908800 wait complete. 2024-03-28T18:52:26.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 -- start start 2024-03-28T18:52:26.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.593+0000 7f841522c740 1 Processor -- start 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac326d0 0x56199ac9fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1b0910 con 0x56199ac326d0 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1b02b0 con 0x56199ac33080 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840c1ad7d0 con 0x56199ac2cda0 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f841522c740 10 client.0 _ll_register_callbacks cb 0x7f83f40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53636/0 (socket says 172.21.15.119:53636) 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f8402a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac326d0 0x56199ac9fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 -- 172.21.15.119:0/16667751 learned_addr learned my addr 172.21.15.119:0/16667751 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 -- 172.21.15.119:0/16667751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 msgr2=0x56199ac3fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 -- 172.21.15.119:0/16667751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac326d0 msgr2=0x56199ac9fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac326d0 0x56199ac9fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.594+0000 7f840be00640 1 -- 172.21.15.119:0/16667751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56199ac34400 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f840be00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f8404005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c6db6effc5568d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404043040 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f83f400f920 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f84040347f0 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8404039040 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f841522c740 10 client.27826 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56199aca7c40 con 0x56199ac2cda0 2024-03-28T18:52:26.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83f400f920 con 0x56199ac2cda0 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.595+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8404055730 con 0x56199ac2cda0 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 10 client.27826 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 10 client.27826 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 20 client.27826 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac3bc00 0x56199aca9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 4 client.27826 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56199ab9db50 con 0x56199ac3bc00 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f8402a00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac3bc00 0x56199aca9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f83f9600640 10 client.27826 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.596+0000 7f8402a00640 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac3bc00 0x56199aca9f30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab9a72e8a374edeb server_cookie=f9edc8a36c6b7c43 in_seq=0 out_seq=0 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f83f9600640 1 -- 172.21.15.119:0/16667751 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56199ab9db50 con 0x56199ac3bc00 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f83f9600640 10 client.27826 handle_command_reply: tid=0 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 client.27826 shutdown 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 msgr2=0x56199ac9f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f8404005d80 comp rx=0 tx=0).stop 2024-03-28T18:52:26.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 shutdown_connections 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56199ac3bc00 0x56199aca9f30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56199ac33080 0x56199ac3fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f83f9600640 0 client.27826 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56199ac326d0 0x56199ac9fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 --2- 172.21.15.119:0/16667751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56199ac2cda0 0x56199ac9f530 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 >> 172.21.15.119:0/16667751 conn(0x56199ac38ec0 msgr2=0x56199ac2b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 shutdown_connections 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 1 -- 172.21.15.119:0/16667751 wait complete. 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.597+0000 7f841522c740 20 client.27826 trim_cache size 0 max 16384 2024-03-28T18:52:26.601 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:52:26.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 msgr2=0x7f83f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 0x7f83f00a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x56199ac321a0 tx=0x7f83fc0d2df0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 msgr2=0x7f840c19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00edf0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 shutdown_connections 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83f00a2970 0x7f83f00a4e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840c10de70 0x7f840c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f840c0743f0 0x7f840c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 --2- 172.21.15.119:0/2491957710 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f840c073ad0 0x7f840c19a110 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 >> 172.21.15.119:0/2491957710 conn(0x7f840c078aa0 msgr2=0x7f840c0fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 shutdown_connections 2024-03-28T18:52:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.598+0000 7f8412400640 1 -- 172.21.15.119:0/2491957710 wait complete. 2024-03-28T18:52:26.615 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:26.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 -- 172.21.15.119:0/3120987887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4108860 msgr2=0x7f97e410c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 --2- 172.21.15.119:0/3120987887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4108860 0x7f97e410c8a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009900 tx=0x7f97d002bc30 comp rx=0 tx=0).stop 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 -- 172.21.15.119:0/3120987887 shutdown_connections 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 --2- 172.21.15.119:0/3120987887 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e410cf80 0x7f97e4114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 --2- 172.21.15.119:0/3120987887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4108860 0x7f97e410c8a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 --2- 172.21.15.119:0/3120987887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e4107eb0 0x7f97e4108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 -- 172.21.15.119:0/3120987887 >> 172.21.15.119:0/3120987887 conn(0x7f97e4075be0 msgr2=0x7f97e4078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 -- 172.21.15.119:0/3120987887 shutdown_connections 2024-03-28T18:52:26.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.827+0000 7f97ec000640 1 -- 172.21.15.119:0/3120987887 wait complete. 2024-03-28T18:52:26.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 Processor -- start 2024-03-28T18:52:26.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 -- start start 2024-03-28T18:52:26.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 0x7f97e419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e41a2e80 0x7f97e41a3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4118040 con 0x7f97e410cf80 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e4117ec0 con 0x7f97e41a2e80 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.828+0000 7f97ec000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41181c0 con 0x7f97e4107eb0 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53658/0 (socket says 172.21.15.119:53658) 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 -- 172.21.15.119:0/3509404254 learned_addr learned my addr 172.21.15.119:0/3509404254 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97ea000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e41a2e80 0x7f97e41a3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e8c00640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 0x7f97e419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 -- 172.21.15.119:0/3509404254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e41a2e80 msgr2=0x7f97e41a3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e41a2e80 0x7f97e41a3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 -- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 msgr2=0x7f97e419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 0x7f97e419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 -- 172.21.15.119:0/3509404254 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e41a3970 con 0x7f97e4107eb0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e8c00640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 0x7f97e419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.829+0000 7f97e9600640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f97d800e9f0 tx=0x7f97d800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.830+0000 7f97e2000640 1 -- 172.21.15.119:0/3509404254 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d800ccc0 con 0x7f97e4107eb0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.830+0000 7f97e2000640 1 -- 172.21.15.119:0/3509404254 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d8010910 con 0x7f97e4107eb0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.830+0000 7f97e2000640 1 -- 172.21.15.119:0/3509404254 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80188a0 con 0x7f97e4107eb0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.830+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e41a3c60 con 0x7f97e4107eb0 2024-03-28T18:52:26.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.830+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e41ab470 con 0x7f97e4107eb0 2024-03-28T18:52:26.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.834+0000 7f97eec7a740 1 Processor -- start 2024-03-28T18:52:26.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.834+0000 7f97e2000640 1 -- 172.21.15.119:0/3509404254 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f97d8010ac0 con 0x7f97e4107eb0 2024-03-28T18:52:26.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.834+0000 7f97eec7a740 1 -- start start 2024-03-28T18:52:26.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.834+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec134630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 0x55a7ec137050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97eec7a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec08f2c0 con 0x55a7ec12c080 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97eec7a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec037c90 con 0x55a7ec134c00 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97eec7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec08b360 con 0x55a7ec12b6d0 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97e9600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec134630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 0x55a7ec137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53662/0 (socket says 172.21.15.119:53662) 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 -- 172.21.15.119:0/2195400304 learned_addr learned my addr 172.21.15.119:0/2195400304 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 -- 172.21.15.119:0/2195400304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 msgr2=0x55a7ec137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 0x55a7ec137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 -- 172.21.15.119:0/2195400304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 msgr2=0x55a7ec134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec134630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97ea000640 1 -- 172.21.15.119:0/2195400304 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7ec05c080 con 0x55a7ec12b6d0 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.835+0000 7f97e8c00640 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 0x55a7ec137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:26.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97ea000640 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009f60 tx=0x7f97dc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44e4c5317b0aa213 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97d6600640 1 -- 172.21.15.119:0/2195400304 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc01e070 con 0x55a7ec12b6d0 2024-03-28T18:52:26.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97d6600640 1 -- 172.21.15.119:0/2195400304 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97dc003c30 con 0x55a7ec12b6d0 2024-03-28T18:52:26.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97d6600640 1 -- 172.21.15.119:0/2195400304 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00fa20 con 0x55a7ec12b6d0 2024-03-28T18:52:26.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97e2000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 0x7f97cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97e2000640 1 -- 172.21.15.119:0/3509404254 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f97d80cf350 con 0x7f97e4107eb0 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.836+0000 7f97e9600640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 0x7f97cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- 172.21.15.119:0/2195400304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 msgr2=0x55a7ec12bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009f60 tx=0x7f97dc009f90 comp rx=0 tx=0).stop 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- 172.21.15.119:0/2195400304 shutdown_connections 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec134c00 0x55a7ec137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec134630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2195400304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec12bab0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- 172.21.15.119:0/2195400304 >> 172.21.15.119:0/2195400304 conn(0x55a7ec125da0 msgr2=0x55a7ec126180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- 172.21.15.119:0/2195400304 shutdown_connections 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- 172.21.15.119:0/2195400304 wait complete. 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97e9600640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 0x7f97cc0a4eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ccac0 tx=0x7f97d80d2ea0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.837+0000 7f97eec7a740 1 -- start start 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 Processor -- start 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec134500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec131e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec137b70 con 0x55a7ec134c00 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec04b0d0 con 0x55a7ec12b6d0 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7ec137cf0 con 0x55a7ec12c080 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97eec7a740 10 client.0 _ll_register_callbacks cb 0x55a7ec04fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53668/0 (socket says 172.21.15.119:53668) 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97e8c00640 1 -- 172.21.15.119:0/1635859934 learned_addr learned my addr 172.21.15.119:0/1635859934 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97ea000640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.838+0000 7f97ea000640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec131e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 -- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 msgr2=0x55a7ec134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec134500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 -- 172.21.15.119:0/1635859934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 msgr2=0x55a7ec131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 -- 172.21.15.119:0/1635859934 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7ebda33f0 con 0x55a7ec12c080 2024-03-28T18:52:26.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97e8c00640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009900 tx=0x7f97d00045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a18658106fe8577f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97caa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d002cc90 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f97d0005030 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.839+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97dc003c30 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97d0005230 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 10 client.27838 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a7ebe6d7d0 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97dc003c30 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f97d0003680 con 0x55a7ec12c080 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 10 client.27838 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 10 client.27838 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 20 client.27838 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec144a60 0x55a7ec146e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 4 client.27838 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a7ec09c800 con 0x55a7ec144a60 2024-03-28T18:52:26.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.840+0000 7f97ea000640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec144a60 0x55a7ec146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.841+0000 7f97ea000640 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec144a60 0x55a7ec146e50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1648b2a8246ff512 server_cookie=49b70b4279ad9aef in_seq=0 out_seq=0 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.841+0000 7f97caa00640 10 client.27838 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97caa00640 1 -- 172.21.15.119:0/1635859934 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a7ec09c800 con 0x55a7ec144a60 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97caa00640 10 client.27838 handle_command_reply: tid=0 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 client.27838 shutdown 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 msgr2=0x55a7ec131890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009900 tx=0x7f97d00045b0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 shutdown_connections 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec144a60 0x55a7ec146e50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec131e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12c080 0x55a7ec131890 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 --2- 172.21.15.119:0/1635859934 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec134500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97caa00640 0 client.27838 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 >> 172.21.15.119:0/1635859934 conn(0x55a7ec125da0 msgr2=0x55a7ec127af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 shutdown_connections 2024-03-28T18:52:26.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 1 -- 172.21.15.119:0/1635859934 wait complete. 2024-03-28T18:52:26.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.842+0000 7f97eec7a740 20 client.27838 trim_cache size 0 max 16384 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 Processor -- start 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 -- start start 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 0x55a7ec134500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec12eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b3b10 con 0x55a7ec134c00 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b5490 con 0x55a7ec12c080 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97eec7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b4170 con 0x55a7ec12b6d0 2024-03-28T18:52:26.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 0x55a7ec134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec133fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53674/0 (socket says 172.21.15.119:53674) 2024-03-28T18:52:26.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97ea000640 1 -- 172.21.15.119:0/3641204327 learned_addr learned my addr 172.21.15.119:0/3641204327 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.856+0000 7f97ea000640 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec12eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 -- 172.21.15.119:0/3641204327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 msgr2=0x55a7ec133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec133fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 -- 172.21.15.119:0/3641204327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 msgr2=0x55a7ec12eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec12eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 -- 172.21.15.119:0/3641204327 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a7ec1473e0 con 0x55a7ec12c080 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97e8c00640 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 0x55a7ec134500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f97d0003af0 tx=0x7f97d002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bef61fab2563246 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97d5c00640 1 -- 172.21.15.119:0/3641204327 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0000e60 con 0x55a7ec12c080 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97d5c00640 1 -- 172.21.15.119:0/3641204327 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d00035f0 con 0x55a7ec12c080 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.857+0000 7f97d5c00640 1 -- 172.21.15.119:0/3641204327 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0004920 con 0x55a7ec12c080 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- 172.21.15.119:0/3641204327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 msgr2=0x55a7ec134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 0x55a7ec134500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f97d0003af0 tx=0x7f97d002bc30 comp rx=0 tx=0).stop 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- 172.21.15.119:0/3641204327 shutdown_connections 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec134c00 0x55a7ec12eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12c080 0x55a7ec134500 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 --2- 172.21.15.119:0/3641204327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec12b6d0 0x55a7ec133fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- 172.21.15.119:0/3641204327 >> 172.21.15.119:0/3641204327 conn(0x55a7ec125da0 msgr2=0x55a7ec1279e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- 172.21.15.119:0/3641204327 shutdown_connections 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- 172.21.15.119:0/3641204327 wait complete. 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.858+0000 7f97eec7a740 1 -- start start 2024-03-28T18:52:26.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 Processor -- start 2024-03-28T18:52:26.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec13ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 0x55a7ec13b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b4e30 con 0x55a7ec12c080 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b47d0 con 0x55a7ec12b6d0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e41b1cf0 con 0x55a7ec134c00 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97eec7a740 10 client.0 _ll_register_callbacks cb 0x7f97dc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97ea000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec13ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97e8c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47896/0 (socket says 172.21.15.119:47896) 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97e8c00640 1 -- 172.21.15.119:0/2415341228 learned_addr learned my addr 172.21.15.119:0/2415341228 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.859+0000 7f97ea000640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 0x55a7ec13b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 -- 172.21.15.119:0/2415341228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 msgr2=0x55a7ec13b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 0x55a7ec13b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 -- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 msgr2=0x55a7ec13ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec13ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 -- 172.21.15.119:0/2415341228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a7ec13b930 con 0x55a7ec12c080 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97ea000640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 0x55a7ec13b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97e8c00640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f97d00039f0 tx=0x7f97d0004450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c2028b13653571 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:26.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97caa00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00055e0 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97d00035f0 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f97d00046c0 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.860+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97d00048c0 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 10 client.17625 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a7ec1a0f90 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d00035f0 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f97d0054780 con 0x55a7ec12c080 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 10 client.17625 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 10 client.17625 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 20 client.17625 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec1a15a0 0x55a7ec1a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 4 client.17625 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a7ec096b50 con 0x55a7ec1a15a0 2024-03-28T18:52:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.861+0000 7f97ea000640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec1a15a0 0x55a7ec1a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97ea000640 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec1a15a0 0x55a7ec1a3990 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3633ecfdbd493652 server_cookie=ad6522bbe004fd65 in_seq=0 out_seq=0 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97caa00640 10 client.17625 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97caa00640 1 -- 172.21.15.119:0/2415341228 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a7ec096b50 con 0x55a7ec1a15a0 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97caa00640 10 client.17625 handle_command_reply: tid=0 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 client.17625 shutdown 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 msgr2=0x55a7ec13ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f97d00039f0 tx=0x7f97d0004450 comp rx=0 tx=0).stop 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 shutdown_connections 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7ec134c00 0x55a7ec13b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55a7ec1a15a0 0x55a7ec1a3990 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a7ec12c080 0x55a7ec13ae80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 --2- 172.21.15.119:0/2415341228 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55a7ec12b6d0 0x55a7ec13ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 >> 172.21.15.119:0/2415341228 conn(0x55a7ec125da0 msgr2=0x55a7ec124620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97caa00640 0 client.17625 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:52:26.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 shutdown_connections 2024-03-28T18:52:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.862+0000 7f97eec7a740 1 -- 172.21.15.119:0/2415341228 wait complete. 2024-03-28T18:52:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97eec7a740 20 client.17625 trim_cache size 0 max 16384 2024-03-28T18:52:26.867 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:52:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 msgr2=0x7f97cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 0x7f97cc0a4eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ccac0 tx=0x7f97d80d2ea0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 msgr2=0x7f97e419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f97d800e9f0 tx=0x7f97d800edf0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 shutdown_connections 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97cc0a29f0 0x7f97cc0a4eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97e41a2e80 0x7f97e41a3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97e410cf80 0x7f97e419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 --2- 172.21.15.119:0/3509404254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97e4107eb0 0x7f97e419e5c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 >> 172.21.15.119:0/3509404254 conn(0x7f97e4075be0 msgr2=0x7f97e4077ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 shutdown_connections 2024-03-28T18:52:26.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:26.863+0000 7f97ec000640 1 -- 172.21.15.119:0/3509404254 wait complete. 2024-03-28T18:52:26.878 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:26 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1511407377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:52:26.880 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:26.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 -- 172.21.15.119:0/2884277938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 msgr2=0x7f8380114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 --2- 172.21.15.119:0/2884277938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f8380114b50 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8378009ca0 tx=0x7f837802c100 comp rx=0 tx=0).stop 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 -- 172.21.15.119:0/2884277938 shutdown_connections 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 --2- 172.21.15.119:0/2884277938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f8380114b50 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 --2- 172.21.15.119:0/2884277938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380108860 0x7f838010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 --2- 172.21.15.119:0/2884277938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8380107eb0 0x7f8380108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 -- 172.21.15.119:0/2884277938 >> 172.21.15.119:0/2884277938 conn(0x7f8380075be0 msgr2=0x7f8380078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 -- 172.21.15.119:0/2884277938 shutdown_connections 2024-03-28T18:52:27.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.091+0000 7f8384c00640 1 -- 172.21.15.119:0/2884277938 wait complete. 2024-03-28T18:52:27.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.092+0000 7f8384c00640 1 Processor -- start 2024-03-28T18:52:27.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 -- start start 2024-03-28T18:52:27.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f838019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83801a2da0 0x7f83801a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8380118030 con 0x7f83801a2da0 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8380117eb0 con 0x7f8380107eb0 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8384c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801181b0 con 0x7f838010cf80 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f837de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f837de00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55932/0 (socket says 172.21.15.119:55932) 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f837de00640 1 -- 172.21.15.119:0/662361786 learned_addr learned my addr 172.21.15.119:0/662361786 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f8375400640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f838019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.093+0000 7f837e800640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83801a2da0 0x7f83801a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 -- 172.21.15.119:0/662361786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 msgr2=0x7f838019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f838019ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 -- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83801a2da0 msgr2=0x7f83801a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83801a2da0 0x7f83801a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 -- 172.21.15.119:0/662361786 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83801a3890 con 0x7f8380107eb0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f8375400640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f838019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f837de00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f837000d950 tx=0x7f837000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f8377400640 1 -- 172.21.15.119:0/662361786 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370017070 con 0x7f8380107eb0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f8377400640 1 -- 172.21.15.119:0/662361786 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8370009c30 con 0x7f8380107eb0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83801a3b80 con 0x7f8380107eb0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.094+0000 7f8377400640 1 -- 172.21.15.119:0/662361786 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837000f6b0 con 0x7f8380107eb0 2024-03-28T18:52:27.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.095+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83801ab370 con 0x7f8380107eb0 2024-03-28T18:52:27.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f8377400640 1 -- 172.21.15.119:0/662361786 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f83700109d0 con 0x7f8380107eb0 2024-03-28T18:52:27.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 Processor -- start 2024-03-28T18:52:27.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 -- start start 2024-03-28T18:52:27.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38a9fe2c0 con 0x55c38aa9b080 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38a9a6c90 con 0x55c38aaa3c00 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f83879a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38a9fa360 con 0x55c38aa9a6d0 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f837e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f837e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53706/0 (socket says 172.21.15.119:53706) 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.098+0000 7f837e800640 1 -- 172.21.15.119:0/156156117 learned_addr learned my addr 172.21.15.119:0/156156117 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f8375400640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837de00640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 -- 172.21.15.119:0/156156117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 msgr2=0x55c38aaa6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 -- 172.21.15.119:0/156156117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 msgr2=0x55c38aaa3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 -- 172.21.15.119:0/156156117 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c38a9cb080 con 0x55c38aa9a6d0 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837de00640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f837e800640 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8378009ca0 tx=0x7f8378009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bb5dc68125b8dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f8377400640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 0x7f83640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.099+0000 7f836f000640 1 -- 172.21.15.119:0/156156117 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378043070 con 0x55c38aa9a6d0 2024-03-28T18:52:27.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f836f000640 1 -- 172.21.15.119:0/156156117 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8378003970 con 0x55c38aa9a6d0 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f836f000640 1 -- 172.21.15.119:0/156156117 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378039040 con 0x55c38aa9a6d0 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f8377400640 1 -- 172.21.15.119:0/662361786 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f83700ceb20 con 0x7f8380107eb0 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f837de00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 0x7f83640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 -- 172.21.15.119:0/156156117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 msgr2=0x55c38aa9aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f837de00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 0x7f83640a4e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55c38aa9a1a0 tx=0x7f83700bb040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8378009ca0 tx=0x7f8378009900 comp rx=0 tx=0).stop 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 -- 172.21.15.119:0/156156117 shutdown_connections 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 --2- 172.21.15.119:0/156156117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa9aab0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.100+0000 7f83879a6740 1 -- 172.21.15.119:0/156156117 >> 172.21.15.119:0/156156117 conn(0x55c38aa94da0 msgr2=0x55c38aa95180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.101+0000 7f83879a6740 1 -- 172.21.15.119:0/156156117 shutdown_connections 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.101+0000 7f83879a6740 1 -- 172.21.15.119:0/156156117 wait complete. 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.101+0000 7f83879a6740 1 -- start start 2024-03-28T18:52:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.101+0000 7f83879a6740 1 Processor -- start 2024-03-28T18:52:27.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 0x55c38aaa0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38aaa6b10 con 0x55c38aaa3c00 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38a9ba0d0 con 0x55c38aa9a6d0 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38aaa6c90 con 0x55c38aa9b080 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f83879a6740 10 client.0 _ll_register_callbacks cb 0x55c38a9bee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f837e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53722/0 (socket says 172.21.15.119:53722) 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f837e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55956/0 (socket says 172.21.15.119:55956) 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 -- 172.21.15.119:0/688233499 learned_addr learned my addr 172.21.15.119:0/688233499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f837e800640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 0x55c38aaa0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 -- 172.21.15.119:0/688233499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 msgr2=0x55c38aaa3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 -- 172.21.15.119:0/688233499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 msgr2=0x55c38aaa0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 0x55c38aaa0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f8375400640 1 -- 172.21.15.119:0/688233499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c38a7123f0 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.102+0000 7f837e800640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 0x55c38aaa0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8375400640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f83680129f0 tx=0x7f8368012eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a3d4266eed6790 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8363400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836800fa40 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8378003970 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f836800fbe0 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f836800fde0 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f83879a6740 10 client.27856 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c38a7dc7d0 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.103+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8378003970 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f836801c560 con 0x55c38aa9b080 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 10 client.27856 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 10 client.27856 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 20 client.27856 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38aab3990 0x55c38aab5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 4 client.27856 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c38aa0b800 con 0x55c38aab3990 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f837e800640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38aab3990 0x55c38aab5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f837e800640 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38aab3990 0x55c38aab5db0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25ed26c9d166fcf2 server_cookie=d120a209265cd89a in_seq=0 out_seq=0 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.104+0000 7f8363400640 10 client.27856 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.105+0000 7f8363400640 1 -- 172.21.15.119:0/688233499 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c38aa0b800 con 0x55c38aab3990 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.105+0000 7f8363400640 10 client.27856 handle_command_reply: tid=0 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.105+0000 7f83879a6740 1 client.27856 shutdown 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 msgr2=0x55c38aaa07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f83680129f0 tx=0x7f8368012eb0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 shutdown_connections 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38aab3990 0x55c38aab5db0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aaa3c00 0x55c38aaa0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9b080 0x55c38aaa07e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 --2- 172.21.15.119:0/688233499 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f8363400640 0 client.27856 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 >> 172.21.15.119:0/688233499 conn(0x55c38aa94da0 msgr2=0x55c38aa96af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 shutdown_connections 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 1 -- 172.21.15.119:0/688233499 wait complete. 2024-03-28T18:52:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.106+0000 7f83879a6740 20 client.27856 trim_cache size 0 max 16384 2024-03-28T18:52:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 Processor -- start 2024-03-28T18:52:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 -- start start 2024-03-28T18:52:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa96900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b3a00 con 0x55c38aa9b080 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b5380 con 0x55c38aaa3c00 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f83879a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b4060 con 0x55c38aa9a6d0 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f8375400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.120+0000 7f8375400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47938/0 (socket says 172.21.15.119:47938) 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 -- 172.21.15.119:0/480345168 learned_addr learned my addr 172.21.15.119:0/480345168 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f837e800640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa96900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f837e800640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 -- 172.21.15.119:0/480345168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 msgr2=0x55c38aa96900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa96900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 -- 172.21.15.119:0/480345168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 msgr2=0x55c38aaa3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 -- 172.21.15.119:0/480345168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c38aab6410 con 0x55c38aa9b080 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f837e800640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:52:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.121+0000 7f8375400640 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f836801c980 tx=0x7f83680059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8927d065080153 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f836e600640 1 -- 172.21.15.119:0/480345168 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836801cb70 con 0x55c38aa9b080 2024-03-28T18:52:27.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f836e600640 1 -- 172.21.15.119:0/480345168 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f836801cd10 con 0x55c38aa9b080 2024-03-28T18:52:27.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f836e600640 1 -- 172.21.15.119:0/480345168 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8368023b80 con 0x55c38aa9b080 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 -- 172.21.15.119:0/480345168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 msgr2=0x55c38aaa2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f836801c980 tx=0x7f83680059b0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 -- 172.21.15.119:0/480345168 shutdown_connections 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aaa3c00 0x55c38aaa3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa2f10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 --2- 172.21.15.119:0/480345168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aa9a6d0 0x55c38aa96900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 -- 172.21.15.119:0/480345168 >> 172.21.15.119:0/480345168 conn(0x55c38aa94da0 msgr2=0x55c38aa961f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 -- 172.21.15.119:0/480345168 shutdown_connections 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.122+0000 7f83879a6740 1 -- 172.21.15.119:0/480345168 wait complete. 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.123+0000 7f83879a6740 1 -- start start 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.123+0000 7f83879a6740 1 Processor -- start 2024-03-28T18:52:27.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.123+0000 7f83879a6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.123+0000 7f83879a6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f83879a6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aaa3c00 0x55c38ab0e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f83879a6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b4d20 con 0x55c38aa9b080 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f83879a6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b46c0 con 0x55c38aa9a6d0 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f83879a6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83801b1be0 con 0x55c38aaa3c00 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f83879a6740 10 client.0 _ll_register_callbacks cb 0x7f8368009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55968/0 (socket says 172.21.15.119:55968) 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 -- 172.21.15.119:0/1355103996 learned_addr learned my addr 172.21.15.119:0/1355103996 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f8375400640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aaa3c00 0x55c38ab0e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 -- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aaa3c00 msgr2=0x55c38ab0e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aaa3c00 0x55c38ab0e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 -- 172.21.15.119:0/1355103996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 msgr2=0x55c38aaa7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 -- 172.21.15.119:0/1355103996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c38ab0e790 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f837e800640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f837802c640 tx=0x7f837804c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2b38c74e8ced13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.124+0000 7f8363400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378039040 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c38aaa10b0 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f8378032590 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8378043040 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 10 client.27884 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c38ab0fa60 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c38aaa10b0 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f83780042c0 con 0x55c38aa9a6d0 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 10 client.27884 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 10 client.27884 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 20 client.27884 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38ab0fda0 0x55c38ab12190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 4 client.27884 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.125+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c38aa05b50 con 0x55c38ab0fda0 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f8375400640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38ab0fda0 0x55c38ab12190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f8363400640 10 client.27884 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f8375400640 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38ab0fda0 0x55c38ab12190 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a7e2361ac915ec8e server_cookie=d5891c0962fe62c5 in_seq=0 out_seq=0 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f8363400640 1 -- 172.21.15.119:0/1355103996 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c38aa05b50 con 0x55c38ab0fda0 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f8363400640 10 client.27884 handle_command_reply: tid=0 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f83879a6740 1 client.27884 shutdown 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 msgr2=0x55c38aaa7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.126+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f837802c640 tx=0x7f837804c040 comp rx=0 tx=0).stop 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 shutdown_connections 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55c38ab0fda0 0x55c38ab12190 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c38aaa3c00 0x55c38ab0e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c38aa9b080 0x55c38aaa7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 --2- 172.21.15.119:0/1355103996 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55c38aa9a6d0 0x55c38aaa7a80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f8363400640 0 client.27884 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 >> 172.21.15.119:0/1355103996 conn(0x55c38aa94da0 msgr2=0x55c38aa9ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 shutdown_connections 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 1 -- 172.21.15.119:0/1355103996 wait complete. 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f83879a6740 20 client.27884 trim_cache size 0 max 16384 2024-03-28T18:52:27.131 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 msgr2=0x7f83640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 0x7f83640a4e30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55c38aa9a1a0 tx=0x7f83700bb040 comp rx=0 tx=0).stop 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 msgr2=0x7f838019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.127+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f837000d950 tx=0x7f837000de10 comp rx=0 tx=0).stop 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 shutdown_connections 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f83640a2970 0x7f83640a4e30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83801a2da0 0x7f83801a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f838010cf80 0x7f838019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 --2- 172.21.15.119:0/662361786 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8380107eb0 0x7f838019e530 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 >> 172.21.15.119:0/662361786 conn(0x7f8380075be0 msgr2=0x7f8380077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 shutdown_connections 2024-03-28T18:52:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.128+0000 7f8384c00640 1 -- 172.21.15.119:0/662361786 wait complete. 2024-03-28T18:52:27.145 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:27.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:52:27.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:26 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1511407377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:52:27.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:26 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1511407377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 -- 172.21.15.119:0/744358039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 msgr2=0x7f2aa4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/744358039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa4109c30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009930 tx=0x7f2a9402bc30 comp rx=0 tx=0).stop 2024-03-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 -- 172.21.15.119:0/744358039 shutdown_connections 2024-03-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/744358039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 0x7f2aa4110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/744358039 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa4103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/744358039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa4109c30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 -- 172.21.15.119:0/744358039 >> 172.21.15.119:0/744358039 conn(0x7f2aa40ff2f0 msgr2=0x7f2aa4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 -- 172.21.15.119:0/744358039 shutdown_connections 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.357+0000 7f2aa9a00640 1 -- 172.21.15.119:0/744358039 wait complete. 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.358+0000 7f2aa9a00640 1 Processor -- start 2024-03-28T18:52:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 -- start start 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 0x7f2aa419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4113c70 con 0x7f2aa4104200 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4113af0 con 0x7f2aa4103840 2024-03-28T18:52:27.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa9a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4113df0 con 0x7f2aa4109850 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55984/0 (socket says 172.21.15.119:55984) 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa2a00640 1 -- 172.21.15.119:0/758976938 learned_addr learned my addr 172.21.15.119:0/758976938 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2aa3400640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.359+0000 7f2a9be00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 0x7f2aa419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 -- 172.21.15.119:0/758976938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 msgr2=0x7f2aa419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 -- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 msgr2=0x7f2aa419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 0x7f2aa419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 -- 172.21.15.119:0/758976938 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa419f1b0 con 0x7f2aa4103840 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009900 tx=0x7f2a94004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2a9b400640 1 -- 172.21.15.119:0/758976938 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94043020 con 0x7f2aa4103840 2024-03-28T18:52:27.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2a9b400640 1 -- 172.21.15.119:0/758976938 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a940054e0 con 0x7f2aa4103840 2024-03-28T18:52:27.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa419f440 con 0x7f2aa4103840 2024-03-28T18:52:27.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2a9b400640 1 -- 172.21.15.119:0/758976938 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94003bb0 con 0x7f2aa4103840 2024-03-28T18:52:27.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.360+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa41a6d30 con 0x7f2aa4103840 2024-03-28T18:52:27.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.364+0000 7f2aac821740 1 Processor -- start 2024-03-28T18:52:27.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 -- start start 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd841630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd844050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd79c2c0 con 0x55bbfd841c00 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd744c90 con 0x55bbfd8386d0 2024-03-28T18:52:27.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aac821740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd798360 con 0x55bbfd839080 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2a9b400640 1 -- 172.21.15.119:0/758976938 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f2a94046020 con 0x7f2aa4103840 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55992/0 (socket says 172.21.15.119:55992) 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2aa3400640 1 -- 172.21.15.119:0/769424100 learned_addr learned my addr 172.21.15.119:0/769424100 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.365+0000 7f2a9be00640 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd844050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd841630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 -- 172.21.15.119:0/769424100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 msgr2=0x55bbfd841630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd841630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 -- 172.21.15.119:0/769424100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 msgr2=0x55bbfd844050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd844050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 -- 172.21.15.119:0/769424100 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbfd769080 con 0x55bbfd8386d0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2aa3400640 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009f50 tx=0x7f2a9c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fce71e84df70b6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2a93e00640 1 -- 172.21.15.119:0/769424100 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c004d20 con 0x55bbfd8386d0 2024-03-28T18:52:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2a93e00640 1 -- 172.21.15.119:0/769424100 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a9c004050 con 0x55bbfd8386d0 2024-03-28T18:52:27.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.366+0000 7f2a93e00640 1 -- 172.21.15.119:0/769424100 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c005220 con 0x55bbfd8386d0 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2a9b400640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 0x7f2a880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2aac821740 1 -- 172.21.15.119:0/769424100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 msgr2=0x55bbfd838ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2aac821740 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009f50 tx=0x7f2a9c0118d0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2a9b400640 1 -- 172.21.15.119:0/758976938 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2a940f0d30 con 0x7f2aa4103840 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 0x7f2a880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.367+0000 7f2aac821740 1 -- 172.21.15.119:0/769424100 shutdown_connections 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd844050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd841630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 --2- 172.21.15.119:0/769424100 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd838ab0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 -- 172.21.15.119:0/769424100 >> 172.21.15.119:0/769424100 conn(0x55bbfd832da0 msgr2=0x55bbfd833180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 -- 172.21.15.119:0/769424100 shutdown_connections 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 -- 172.21.15.119:0/769424100 wait complete. 2024-03-28T18:52:27.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aa2a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 0x7f2a880a4eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55bbfd8381a0 tx=0x7f2a940b9040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 -- start start 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.368+0000 7f2aac821740 1 Processor -- start 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd839080 0x55bbfd83e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd83ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd844b70 con 0x55bbfd839080 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd7580d0 con 0x55bbfd841c00 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bbfd844cf0 con 0x55bbfd8386d0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aac821740 10 client.0 _ll_register_callbacks cb 0x55bbfd75ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53772/0 (socket says 172.21.15.119:53772) 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aa3400640 1 -- 172.21.15.119:0/3641887914 learned_addr learned my addr 172.21.15.119:0/3641887914 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2a9be00640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd839080 0x55bbfd83e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.369+0000 7f2aa3400640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd83ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 -- 172.21.15.119:0/3641887914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 msgr2=0x55bbfd83ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd83ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 -- 172.21.15.119:0/3641887914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd839080 msgr2=0x55bbfd83e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd839080 0x55bbfd83e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 -- 172.21.15.119:0/3641887914 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbfd4b03f0 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aa3400640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c004990 tx=0x7f2a9c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5bb0685bc7cfb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2a90c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c02b070 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2a9c027440 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a9c004050 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a9c037050 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aac821740 10 client.27874 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.370+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bbfd57a7d0 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a9c004050 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2a9c00f6e0 con 0x55bbfd8386d0 2024-03-28T18:52:27.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 10 client.27874 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 10 client.27874 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 20 client.27874 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8518d0 0x55bbfd853cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 4 client.27874 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bbfd7a9800 con 0x55bbfd8518d0 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.371+0000 7f2a9be00640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8518d0 0x55bbfd853cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.372+0000 7f2a9be00640 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8518d0 0x55bbfd853cc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2c82c1b7c71869d server_cookie=5c351a49d3d0e9d in_seq=0 out_seq=0 2024-03-28T18:52:27.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.372+0000 7f2a90c00640 10 client.27874 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.372+0000 7f2a90c00640 1 -- 172.21.15.119:0/3641887914 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bbfd7a9800 con 0x55bbfd8518d0 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.372+0000 7f2a90c00640 10 client.27874 handle_command_reply: tid=0 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 client.27874 shutdown 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 msgr2=0x55bbfd841460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c004990 tx=0x7f2a9c0047b0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 shutdown_connections 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8518d0 0x55bbfd853cc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd83ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd839080 0x55bbfd83e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 --2- 172.21.15.119:0/3641887914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd8386d0 0x55bbfd841460 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 >> 172.21.15.119:0/3641887914 conn(0x55bbfd832da0 msgr2=0x55bbfd834af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2a90c00640 0 client.27874 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 shutdown_connections 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 1 -- 172.21.15.119:0/3641887914 wait complete. 2024-03-28T18:52:27.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.373+0000 7f2aac821740 20 client.27874 trim_cache size 0 max 16384 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.386+0000 7f2aac821740 1 Processor -- start 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 -- start start 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd840f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd841460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41af650 con 0x55bbfd841c00 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41b0fd0 con 0x55bbfd8386d0 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aac821740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41afcb0 con 0x55bbfd839080 2024-03-28T18:52:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2a9be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd840f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:55998/0 (socket says 172.21.15.119:55998) 2024-03-28T18:52:27.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aa3400640 1 -- 172.21.15.119:0/739827270 learned_addr learned my addr 172.21.15.119:0/739827270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.387+0000 7f2aa3400640 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd841460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 -- 172.21.15.119:0/739827270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 msgr2=0x55bbfd839460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 -- 172.21.15.119:0/739827270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 msgr2=0x55bbfd841460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd841460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 -- 172.21.15.119:0/739827270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bbfd838e20 con 0x55bbfd839080 2024-03-28T18:52:27.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2aa3400640 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a9be00640 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd840f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55bbfd854340 tx=0x7f2a8c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c056ccda03ee1acc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a93400640 1 -- 172.21.15.119:0/739827270 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c023070 con 0x55bbfd839080 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a93400640 1 -- 172.21.15.119:0/739827270 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c01fbe0 con 0x55bbfd839080 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.388+0000 7f2a93400640 1 -- 172.21.15.119:0/739827270 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c01ec20 con 0x55bbfd839080 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- 172.21.15.119:0/739827270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 msgr2=0x55bbfd840f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd840f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55bbfd854340 tx=0x7f2a8c005f90 comp rx=0 tx=0).stop 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- 172.21.15.119:0/739827270 shutdown_connections 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd841c00 0x55bbfd841460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd840f20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 --2- 172.21.15.119:0/739827270 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd8386d0 0x55bbfd839460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- 172.21.15.119:0/739827270 >> 172.21.15.119:0/739827270 conn(0x55bbfd832da0 msgr2=0x55bbfd831eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- 172.21.15.119:0/739827270 shutdown_connections 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- 172.21.15.119:0/739827270 wait complete. 2024-03-28T18:52:27.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 -- start start 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.389+0000 7f2aac821740 1 Processor -- start 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 0x55bbfd845600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd846080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41b0970 con 0x55bbfd8386d0 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41b0310 con 0x55bbfd841c00 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41ad830 con 0x55bbfd839080 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aac821740 10 client.0 _ll_register_callbacks cb 0x7f2a8c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2a9be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 0x55bbfd845600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2a9be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53784/0 (socket says 172.21.15.119:53784) 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2a9be00640 1 -- 172.21.15.119:0/2440080092 learned_addr learned my addr 172.21.15.119:0/2440080092 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd846080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 -- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 msgr2=0x55bbfd845b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 -- 172.21.15.119:0/2440080092 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 msgr2=0x55bbfd846080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.390+0000 7f2aa3400640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd846080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2aa3400640 1 -- 172.21.15.119:0/2440080092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bbfd8ac210 con 0x55bbfd8386d0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2a9be00640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2aa3400640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 0x55bbfd845600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00fd50 tx=0x7f2a9c004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188934b74da37e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2a90c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c02b040 con 0x55bbfd8386d0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f2a9c003c90 con 0x55bbfd8386d0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a8c01fbe0 con 0x55bbfd8386d0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a9c003eb0 con 0x55bbfd8386d0 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2aac821740 10 client.17661 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.391+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bbfd8ad4f0 con 0x55bbfd8386d0 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c01fbe0 con 0x55bbfd8386d0 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2a9c027790 con 0x55bbfd8386d0 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 10 client.17661 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 10 client.17661 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 20 client.17661 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8adb90 0x55bbfd8affb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 4 client.17661 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bbfd7a3b50 con 0x55bbfd8adb90 2024-03-28T18:52:27.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.392+0000 7f2a9be00640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8adb90 0x55bbfd8affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2a9be00640 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8adb90 0x55bbfd8affb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=499d3dd8be1afc30 server_cookie=2a94d119f6056e7e in_seq=0 out_seq=0 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2a90c00640 10 client.17661 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2a90c00640 1 -- 172.21.15.119:0/2440080092 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bbfd7a3b50 con 0x55bbfd8adb90 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2a90c00640 10 client.17661 handle_command_reply: tid=0 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 client.17661 shutdown 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 msgr2=0x55bbfd845600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 0x55bbfd845600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00fd50 tx=0x7f2a9c004700 comp rx=0 tx=0).stop 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 shutdown_connections 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55bbfd8adb90 0x55bbfd8affb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55bbfd841c00 0x55bbfd846080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bbfd839080 0x55bbfd845b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2a90c00640 0 client.17661 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 --2- 172.21.15.119:0/2440080092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bbfd8386d0 0x55bbfd845600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 >> 172.21.15.119:0/2440080092 conn(0x55bbfd832da0 msgr2=0x55bbfd83e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 shutdown_connections 2024-03-28T18:52:27.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 1 -- 172.21.15.119:0/2440080092 wait complete. 2024-03-28T18:52:27.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.393+0000 7f2aac821740 20 client.17661 trim_cache size 0 max 16384 2024-03-28T18:52:27.399 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 msgr2=0x7f2a880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 0x7f2a880a4eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55bbfd8381a0 tx=0x7f2a940b9040 comp rx=0 tx=0).stop 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 msgr2=0x7f2aa419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2a94009900 tx=0x7f2a94004360 comp rx=0 tx=0).stop 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 shutdown_connections 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f2a880a29f0 0x7f2a880a4eb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2aa4109850 0x7f2aa419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2aa4104200 0x7f2aa419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 --2- 172.21.15.119:0/758976938 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2aa4103840 0x7f2aa419a210 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 >> 172.21.15.119:0/758976938 conn(0x7f2aa40ff2f0 msgr2=0x7f2aa40ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 shutdown_connections 2024-03-28T18:52:27.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.394+0000 7f2aa9a00640 1 -- 172.21.15.119:0/758976938 wait complete. 2024-03-28T18:52:27.413 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:27.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:52:27.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 -- 172.21.15.119:0/171313386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18109850 msgr2=0x7ffa18109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/171313386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18109850 0x7ffa18109c30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009930 tx=0x7ffa0802bc20 comp rx=0 tx=0).stop 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 -- 172.21.15.119:0/171313386 shutdown_connections 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/171313386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa18110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/171313386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18103840 0x7ffa18103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/171313386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18109850 0x7ffa18109c30 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 -- 172.21.15.119:0/171313386 >> 172.21.15.119:0/171313386 conn(0x7ffa180ff2f0 msgr2=0x7ffa18101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 -- 172.21.15.119:0/171313386 shutdown_connections 2024-03-28T18:52:27.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.620+0000 7ffa1ca00640 1 -- 172.21.15.119:0/171313386 wait complete. 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.621+0000 7ffa1ca00640 1 Processor -- start 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.621+0000 7ffa1ca00640 1 -- start start 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 0x7ffa1819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 0x7ffa1819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18113c70 con 0x7ffa18109850 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18113af0 con 0x7ffa18104200 2024-03-28T18:52:27.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa1ca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa18113df0 con 0x7ffa18103840 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 0x7ffa1819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa16800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 0x7ffa1819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56038/0 (socket says 172.21.15.119:56038) 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1279984986 learned_addr learned my addr 172.21.15.119:0/1279984986 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1279984986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 msgr2=0x7ffa1819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 0x7ffa1819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 msgr2=0x7ffa1819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.622+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 0x7ffa1819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1279984986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1819f180 con 0x7ffa18104200 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 0x7ffa1819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa16800640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 0x7ffa1819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000e9f0 tx=0x7ffa0000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa0ea00640 1 -- 172.21.15.119:0/1279984986 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000cd80 con 0x7ffa18104200 2024-03-28T18:52:27.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa0ea00640 1 -- 172.21.15.119:0/1279984986 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa00010910 con 0x7ffa18104200 2024-03-28T18:52:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa0ea00640 1 -- 172.21.15.119:0/1279984986 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00018990 con 0x7ffa18104200 2024-03-28T18:52:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1819f470 con 0x7ffa18104200 2024-03-28T18:52:27.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.623+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa181a6d00 con 0x7ffa18104200 2024-03-28T18:52:27.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.626+0000 7ffa0ea00640 1 -- 172.21.15.119:0/1279984986 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7ffa00010ac0 con 0x7ffa18104200 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.626+0000 7ffa1f7cb740 1 Processor -- start 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 -- start start 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644055630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x561644058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561643fb02c0 con 0x56164404d080 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561643f58c90 con 0x56164404c6d0 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561643fac360 con 0x561644055c00 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa16800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644055630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa16800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56044/0 (socket says 172.21.15.119:56044) 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa16800640 1 -- 172.21.15.119:0/1914778917 learned_addr learned my addr 172.21.15.119:0/1914778917 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.627+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x561644058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 -- 172.21.15.119:0/1914778917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 msgr2=0x561644058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x561644058050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 -- 172.21.15.119:0/1914778917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 msgr2=0x561644055630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644055630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 -- 172.21.15.119:0/1914778917 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561643f7d080 con 0x56164404c6d0 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa16800640 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ffa10011790 tx=0x7ffa10011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=531bdd48484b6c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa0ea00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 0x7ff9fc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa07000640 1 -- 172.21.15.119:0/1914778917 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10003c30 con 0x56164404c6d0 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa0ea00640 1 -- 172.21.15.119:0/1279984986 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7ffa000cf1f0 con 0x7ffa18104200 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa07000640 1 -- 172.21.15.119:0/1914778917 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1000fe80 con 0x56164404c6d0 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa07000640 1 -- 172.21.15.119:0/1914778917 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa100193e0 con 0x56164404c6d0 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.628+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 0x7ff9fc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1914778917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 msgr2=0x56164404cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ffa10011790 tx=0x7ffa10011c50 comp rx=0 tx=0).stop 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 0x7ff9fc0a4eb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009900 tx=0x7ffa08005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1914778917 shutdown_connections 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x561644058050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644055630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1914778917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164404cab0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1914778917 >> 172.21.15.119:0/1914778917 conn(0x561644046da0 msgr2=0x561644047180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1914778917 shutdown_connections 2024-03-28T18:52:27.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.629+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1914778917 wait complete. 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 -- start start 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 Processor -- start 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644052890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561644055c00 0x561644052e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561644058b70 con 0x56164404d080 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x561643f6c0d0 con 0x561644055c00 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x561644058cf0 con 0x56164404c6d0 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa1f7cb740 10 client.0 _ll_register_callbacks cb 0x561643f70e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644052890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa16800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561644055c00 0x561644052e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.630+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:53836/0 (socket says 172.21.15.119:53836) 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 -- 172.21.15.119:0/430319740 learned_addr learned my addr 172.21.15.119:0/430319740 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 -- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561644055c00 msgr2=0x561644052e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561644055c00 0x561644052e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 -- 172.21.15.119:0/430319740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 msgr2=0x561644052890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644052890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 -- 172.21.15.119:0/430319740 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561643cc43f0 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa15e00640 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffa08000c00 tx=0x7ffa08009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e864cdaac3fdb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ff9fb400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa08046070 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7ffa08004ec0 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.631+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561643fbd800 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa080050c0 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 10 client.27892 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561643d8e7d0 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561643fbd800 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffa08034440 con 0x56164404c6d0 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 10 client.27892 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 10 client.27892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:27.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 20 client.27892 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561644065ab0 0x561644067ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 4 client.27892 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.632+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561643fb7b50 con 0x561644065ab0 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.633+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561644065ab0 0x561644067ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.633+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561644065ab0 0x561644067ea0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37159ac0fa703a97 server_cookie=c94afd4d0f4efe39 in_seq=0 out_seq=0 2024-03-28T18:52:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.633+0000 7ff9fb400640 10 client.27892 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ff9fb400640 1 -- 172.21.15.119:0/430319740 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561643fb7b50 con 0x561644065ab0 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ff9fb400640 10 client.27892 handle_command_reply: tid=0 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 client.27892 shutdown 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 msgr2=0x561644055500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ffa08000c00 tx=0x7ffa08009260 comp rx=0 tx=0).stop 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 shutdown_connections 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x561644065ab0 0x561644067ea0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x561644055c00 0x561644052e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x561644052890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/430319740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404c6d0 0x561644055500 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 >> 172.21.15.119:0/430319740 conn(0x561644046da0 msgr2=0x561644048af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ff9fb400640 0 client.27892 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:27.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 shutdown_connections 2024-03-28T18:52:27.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/430319740 wait complete. 2024-03-28T18:52:27.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.634+0000 7ffa1f7cb740 20 client.27892 trim_cache size 0 max 16384 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 Processor -- start 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 -- start start 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x561644045eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 0x561644054fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181af630 con 0x561644055c00 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181b0fb0 con 0x56164404c6d0 2024-03-28T18:52:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.648+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181afc90 con 0x56164404d080 2024-03-28T18:52:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48088/0 (socket says 172.21.15.119:48088) 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 -- 172.21.15.119:0/549650581 learned_addr learned my addr 172.21.15.119:0/549650581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 0x561644054fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa15e00640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x561644045eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 -- 172.21.15.119:0/549650581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 msgr2=0x561644054fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 0x561644054fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 -- 172.21.15.119:0/549650581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 msgr2=0x561644045eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x561644045eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 -- 172.21.15.119:0/549650581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561644068500 con 0x561644055c00 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 0x561644054fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa15e00640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x561644045eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.649+0000 7ffa16800640 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa10011730 tx=0x7ffa100045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab5f6cc35e614cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa06600640 1 -- 172.21.15.119:0/549650581 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1001f040 con 0x561644055c00 2024-03-28T18:52:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa06600640 1 -- 172.21.15.119:0/549650581 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1000fc20 con 0x561644055c00 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa06600640 1 -- 172.21.15.119:0/549650581 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10005050 con 0x561644055c00 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/549650581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 msgr2=0x561644055500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa10011730 tx=0x7ffa100045a0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/549650581 shutdown_connections 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561644055c00 0x561644055500 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56164404d080 0x561644054fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/549650581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x561644045eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/549650581 >> 172.21.15.119:0/549650581 conn(0x561644046da0 msgr2=0x5616440480a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/549650581 shutdown_connections 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.650+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/549650581 wait complete. 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.651+0000 7ffa1f7cb740 1 -- start start 2024-03-28T18:52:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.651+0000 7ffa1f7cb740 1 Processor -- start 2024-03-28T18:52:27.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.651+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.651+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x56164405dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.651+0000 7ffa1f7cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x56164405bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181b0950 con 0x56164404d080 2024-03-28T18:52:27.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181b02f0 con 0x56164404c6d0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa1f7cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa181ad810 con 0x561644055c00 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa1f7cb740 10 client.0 _ll_register_callbacks cb 0x7ffa08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x56164405dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa15e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:56072/0 (socket says 172.21.15.119:56072) 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa15e00640 1 -- 172.21.15.119:0/1755697328 learned_addr learned my addr 172.21.15.119:0/1755697328 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa16800640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x56164405bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1755697328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 msgr2=0x56164405bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x56164405bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 msgr2=0x56164405d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 -- 172.21.15.119:0/1755697328 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56164404e470 con 0x56164404d080 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ffa0fe00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x56164405dbd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa000cc850 tx=0x7ffa000d2ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d345c88dc7d00c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ff9fb400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:52:27.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.652+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0009b170 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7ffa000860b0 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa1000fc20 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa0009b810 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 10 client.17679 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616440c1be0 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1000fc20 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffa000ccad0 con 0x56164404d080 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 10 client.17679 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 10 client.17679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 20 client.17679 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616440c2310 0x5616440c4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 4 client.17679 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.653+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616440465b0 con 0x5616440c2310 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616440c2310 0x5616440c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:27.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa15e00640 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616440c2310 0x5616440c4730 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed03b51368eaafa server_cookie=431a4780d830b812 in_seq=0 out_seq=0 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ff9fb400640 10 client.17679 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ff9fb400640 1 -- 172.21.15.119:0/1755697328 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616440465b0 con 0x5616440c2310 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ff9fb400640 10 client.17679 handle_command_reply: tid=0 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 client.17679 shutdown 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 msgr2=0x56164405dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x56164405dbd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa000cc850 tx=0x7ffa000d2ca0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 shutdown_connections 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5616440c2310 0x5616440c4730 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561644055c00 0x56164405bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56164404d080 0x56164405dbd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 --2- 172.21.15.119:0/1755697328 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56164404c6d0 0x56164405d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 >> 172.21.15.119:0/1755697328 conn(0x561644046da0 msgr2=0x56164404dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.654+0000 7ff9fb400640 0 client.17679 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 shutdown_connections 2024-03-28T18:52:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1f7cb740 1 -- 172.21.15.119:0/1755697328 wait complete. 2024-03-28T18:52:27.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1f7cb740 20 client.17679 trim_cache size 0 max 16384 2024-03-28T18:52:27.660 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:52:27.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 msgr2=0x7ff9fc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 0x7ff9fc0a4eb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009900 tx=0x7ffa08005a20 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 msgr2=0x7ffa1819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000e9f0 tx=0x7ffa0000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 shutdown_connections 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7ff9fc0a29f0 0x7ff9fc0a4eb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa18109850 0x7ffa1819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ffa18104200 0x7ffa1819a6d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 --2- 172.21.15.119:0/1279984986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa18103840 0x7ffa1819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.655+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 >> 172.21.15.119:0/1279984986 conn(0x7ffa180ff2f0 msgr2=0x7ffa180ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.656+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 shutdown_connections 2024-03-28T18:52:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:27.656+0000 7ffa1ca00640 1 -- 172.21.15.119:0/1279984986 wait complete. 2024-03-28T18:52:28.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:27 smithi119 ceph-mon[31813]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:28.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:27 smithi196 ceph-mon[36300]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:28.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:27 smithi131 ceph-mon[36065]: pgmap v259: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:28.673 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:52:28.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:52:28.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 -- 172.21.15.119:0/1192650084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04109850 msgr2=0x7f1c04109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 --2- 172.21.15.119:0/1192650084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04109850 0x7f1c04109c30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009900 tx=0x7f1bf802bc50 comp rx=0 tx=0).stop 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 -- 172.21.15.119:0/1192650084 shutdown_connections 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 --2- 172.21.15.119:0/1192650084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04104200 0x7f1c04110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 --2- 172.21.15.119:0/1192650084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04103840 0x7f1c04103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 --2- 172.21.15.119:0/1192650084 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04109850 0x7f1c04109c30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 -- 172.21.15.119:0/1192650084 >> 172.21.15.119:0/1192650084 conn(0x7f1c040ff2f0 msgr2=0x7f1c04101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 -- 172.21.15.119:0/1192650084 shutdown_connections 2024-03-28T18:52:28.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.903+0000 7f1c0c000640 1 -- 172.21.15.119:0/1192650084 wait complete. 2024-03-28T18:52:28.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.904+0000 7f1c0c000640 1 Processor -- start 2024-03-28T18:52:28.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 -- start start 2024-03-28T18:52:28.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04103840 0x7f1c0419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04104200 0x7f1c0419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04113c70 con 0x7f1c04103840 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04113af0 con 0x7f1c04104200 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0c000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04113df0 con 0x7f1c04109850 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c09600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04103840 0x7f1c0419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35772/0 (socket says 172.21.15.119:35772) 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c0a000640 1 -- 172.21.15.119:0/3024255979 learned_addr learned my addr 172.21.15.119:0/3024255979 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.905+0000 7f1c08c00640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04104200 0x7f1c0419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 -- 172.21.15.119:0/3024255979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04104200 msgr2=0x7f1c0419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04104200 0x7f1c0419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 -- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04103840 msgr2=0x7f1c0419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04103840 0x7f1c0419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 -- 172.21.15.119:0/3024255979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0419f170 con 0x7f1c04109850 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00eb10 tx=0x7f1bfc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c02000640 1 -- 172.21.15.119:0/3024255979 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc004b20 con 0x7f1c04109850 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0419f460 con 0x7f1c04109850 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c02000640 1 -- 172.21.15.119:0/3024255979 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc004cc0 con 0x7f1c04109850 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.906+0000 7f1c02000640 1 -- 172.21.15.119:0/3024255979 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc010760 con 0x7f1c04109850 2024-03-28T18:52:28.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.907+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c041a6cf0 con 0x7f1c04109850 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 Processor -- start 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c02000640 1 -- 172.21.15.119:0/3024255979 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1bfc005020 con 0x7f1c04109850 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 -- start start 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a40080 0x55e824a40460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a49580 0x55e824a4b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8249a4eb0 con 0x55e824a49580 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82494da60 con 0x55e824a409a0 2024-03-28T18:52:28.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.910+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8249a0f50 con 0x55e824a40080 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a40080 0x55e824a40460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44858/0 (socket says 172.21.15.119:44858) 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 -- 172.21.15.119:0/3053875377 learned_addr learned my addr 172.21.15.119:0/3053875377 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c08c00640 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a49580 0x55e824a4b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 -- 172.21.15.119:0/3053875377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a40080 msgr2=0x55e824a40460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a40080 0x55e824a40460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 -- 172.21.15.119:0/3053875377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a49580 msgr2=0x55e824a4b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a49580 0x55e824a4b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 -- 172.21.15.119:0/3053875377 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8247817e0 con 0x55e824a409a0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c09600640 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1bf802cfd0 tx=0x7f1bf8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc89f504b8458318 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.911+0000 7f1c02000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 0x7f1bec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1c02000640 1 -- 172.21.15.119:0/3024255979 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f1bfc0cf400 con 0x7f1c04109850 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1bf2600640 1 -- 172.21.15.119:0/3053875377 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8042070 con 0x55e824a409a0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1bf2600640 1 -- 172.21.15.119:0/3053875377 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf802c8d0 con 0x55e824a409a0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1bf2600640 1 -- 172.21.15.119:0/3053875377 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8004ee0 con 0x55e824a409a0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1c08c00640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 0x7f1bec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3053875377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 msgr2=0x55e824a49040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1bf802cfd0 tx=0x7f1bf8009560 comp rx=0 tx=0).stop 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.912+0000 7f1c08c00640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 0x7f1bec0a4eb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55e824a3b940 tx=0x7f1bf4005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3053875377 shutdown_connections 2024-03-28T18:52:28.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a49580 0x55e824a4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a409a0 0x55e824a49040 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3053875377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a40080 0x55e824a40460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3053875377 >> 172.21.15.119:0/3053875377 conn(0x55e824a3a7b0 msgr2=0x55e824a3ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3053875377 shutdown_connections 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3053875377 wait complete. 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 -- start start 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.913+0000 7f1c0edc2740 1 Processor -- start 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a46230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a46770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e824a4c310 con 0x55e824a40080 2024-03-28T18:52:28.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e824960eb0 con 0x55e824a49580 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e824a4c490 con 0x55e824a409a0 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0edc2740 10 client.0 _ll_register_callbacks cb 0x55e824955150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a46230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c09600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a46230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35798/0 (socket says 172.21.15.119:35798) 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a46770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c08c00640 1 -- 172.21.15.119:0/3587055590 learned_addr learned my addr 172.21.15.119:0/3587055590 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 -- 172.21.15.119:0/3587055590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 msgr2=0x55e824a46230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a46230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 -- 172.21.15.119:0/3587055590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 msgr2=0x55e824a48ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a48ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c0a000640 1 -- 172.21.15.119:0/3587055590 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8247e1f70 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.914+0000 7f1c09600640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a48ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0a000640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a46770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0cca60 tx=0x7f1bfc09b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6280920a4410cee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0862e0 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f1bfc0a3c80 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bf802c8d0 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bfc09fac0 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 10 client.27929 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e824679170 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf802c8d0 con 0x55e824a49580 2024-03-28T18:52:28.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1bfc005310 con 0x55e824a49580 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 10 client.27929 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 10 client.27929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.915+0000 7f1c0edc2740 20 client.27929 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824a59550 0x55e824a5b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1c0edc2740 4 client.27929 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e8249b23f0 con 0x55e824a59550 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1c09600640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824a59550 0x55e824a5b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1c09600640 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824a59550 0x55e824a5b940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b73a814de731bf2 server_cookie=4a4fcb6dd02b28f9 in_seq=0 out_seq=0 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.916+0000 7f1be6a00640 10 client.27929 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1be6a00640 1 -- 172.21.15.119:0/3587055590 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e8249b23f0 con 0x55e824a59550 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1be6a00640 10 client.27929 handle_command_reply: tid=0 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 client.27929 shutdown 2024-03-28T18:52:28.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 msgr2=0x55e824a46770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a46770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0cca60 tx=0x7f1bfc09b390 comp rx=0 tx=0).stop 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 shutdown_connections 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824a59550 0x55e824a5b940 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a46770 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a46230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1be6a00640 0 client.27929 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/3587055590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a48ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 >> 172.21.15.119:0/3587055590 conn(0x55e824a3a7b0 msgr2=0x55e824a3c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 shutdown_connections 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.917+0000 7f1c0edc2740 1 -- 172.21.15.119:0/3587055590 wait complete. 2024-03-28T18:52:28.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.918+0000 7f1c0edc2740 20 client.27929 trim_cache size 0 max 16384 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.935+0000 7f1c0edc2740 1 Processor -- start 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.935+0000 7f1c0edc2740 1 -- start start 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 0x55e824a48960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 0x55e824a48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041af5b0 con 0x55e824a409a0 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041b0f30 con 0x55e824a40080 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041afc10 con 0x55e824a49580 2024-03-28T18:52:28.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c09600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 0x55e824a48960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 0x55e824a48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 0x55e824a48ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45774/0 (socket says 172.21.15.119:45774) 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35808/0 (socket says 172.21.15.119:35808) 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c09600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 0x55e824a48960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44882/0 (socket says 172.21.15.119:44882) 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c08c00640 1 -- 172.21.15.119:0/2700228309 learned_addr learned my addr 172.21.15.119:0/2700228309 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 -- 172.21.15.119:0/2700228309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 msgr2=0x55e824a48960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 0x55e824a48960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 -- 172.21.15.119:0/2700228309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 msgr2=0x55e824a48ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 0x55e824a48ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.936+0000 7f1c0a000640 1 -- 172.21.15.119:0/2700228309 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e824a4bce0 con 0x55e824a49580 2024-03-28T18:52:28.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.937+0000 7f1c0a000640 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a3be0 tx=0x7f1bfc005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3cdbcb7610d8b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.937+0000 7f1bf1c00640 1 -- 172.21.15.119:0/2700228309 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc09b4b0 con 0x55e824a49580 2024-03-28T18:52:28.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.937+0000 7f1bf1c00640 1 -- 172.21.15.119:0/2700228309 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc0a4110 con 0x55e824a49580 2024-03-28T18:52:28.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.937+0000 7f1bf1c00640 1 -- 172.21.15.119:0/2700228309 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0a3ee0 con 0x55e824a49580 2024-03-28T18:52:28.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- 172.21.15.119:0/2700228309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 msgr2=0x55e824a54bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0a3be0 tx=0x7f1bfc005840 comp rx=0 tx=0).stop 2024-03-28T18:52:28.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- 172.21.15.119:0/2700228309 shutdown_connections 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a49580 0x55e824a54bd0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a409a0 0x55e824a48ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/2700228309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a40080 0x55e824a48960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- 172.21.15.119:0/2700228309 >> 172.21.15.119:0/2700228309 conn(0x55e824a3a7b0 msgr2=0x55e824a41750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- 172.21.15.119:0/2700228309 shutdown_connections 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- 172.21.15.119:0/2700228309 wait complete. 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 -- start start 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.938+0000 7f1c0edc2740 1 Processor -- start 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a43e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a4ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041b08d0 con 0x55e824a40080 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041b0270 con 0x55e824a49580 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041ad790 con 0x55e824a409a0 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0edc2740 10 client.0 _ll_register_callbacks cb 0x7f1bfc09b7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0a000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a4ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35820/0 (socket says 172.21.15.119:35820) 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 -- 172.21.15.119:0/237159933 learned_addr learned my addr 172.21.15.119:0/237159933 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c09600640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a43e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 -- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 msgr2=0x55e824a4ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a4ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 -- 172.21.15.119:0/237159933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 msgr2=0x55e824a43e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a43e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c08c00640 1 -- 172.21.15.119:0/237159933 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e824a4f300 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c0a000640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a4ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.939+0000 7f1c09600640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a43e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1c08c00640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009fd0 tx=0x7f1bf4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43656f7a635ef2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1be6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4032030 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f1bf401a050 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bfc0a4110 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bf403c050 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1c0edc2740 10 client.27913 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.940+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e824ab5d10 con 0x55e824a409a0 2024-03-28T18:52:28.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc0a4110 con 0x55e824a409a0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1bf4004c80 con 0x55e824a409a0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 10 client.27913 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 10 client.27913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 20 client.27913 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824ab64f0 0x55e824ab8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 4 client.27913 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e8249ac740 con 0x55e824ab64f0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.941+0000 7f1c09600640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824ab64f0 0x55e824ab8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.942+0000 7f1c09600640 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824ab64f0 0x55e824ab8910 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57309ef71567b86e server_cookie=8f2d5681166162b2 in_seq=0 out_seq=0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.942+0000 7f1be6a00640 10 client.27913 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1be6a00640 1 -- 172.21.15.119:0/237159933 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e8249ac740 con 0x55e824ab64f0 2024-03-28T18:52:28.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1be6a00640 10 client.27913 handle_command_reply: tid=0 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 client.27913 shutdown 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 msgr2=0x55e824a4e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009fd0 tx=0x7f1bf4004730 comp rx=0 tx=0).stop 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 shutdown_connections 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55e824ab64f0 0x55e824ab8910 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55e824a49580 0x55e824a4ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e824a409a0 0x55e824a4e6e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 --2- 172.21.15.119:0/237159933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e824a40080 0x55e824a43e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1be6a00640 0 client.27913 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 >> 172.21.15.119:0/237159933 conn(0x55e824a3a7b0 msgr2=0x55e824a45a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 shutdown_connections 2024-03-28T18:52:28.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 1 -- 172.21.15.119:0/237159933 wait complete. 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.943+0000 7f1c0edc2740 20 client.27913 trim_cache size 0 max 16384 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "f2605f87-22e0-469b-87ec-6d62c1a6d358", 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:52:28.950 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 msgr2=0x7f1bec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 0x7f1bec0a4eb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55e824a3b940 tx=0x7f1bf4005f90 comp rx=0 tx=0).stop 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 msgr2=0x7f1c0419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00eb10 tx=0x7f1bfc00c2d0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 shutdown_connections 2024-03-28T18:52:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f1bec0a29f0 0x7f1bec0a4eb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c04109850 0x7f1c0419ea40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c04104200 0x7f1c0419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 --2- 172.21.15.119:0/3024255979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c04103840 0x7f1c0419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 >> 172.21.15.119:0/3024255979 conn(0x7f1c040ff2f0 msgr2=0x7f1c040ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 shutdown_connections 2024-03-28T18:52:28.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:28.944+0000 7f1c0c000640 1 -- 172.21.15.119:0/3024255979 wait complete. 2024-03-28T18:52:28.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:52:28.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 -- 172.21.15.119:0/662866777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef01124e0 msgr2=0x7fbef01148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 --2- 172.21.15.119:0/662866777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef01124e0 0x7fbef01148d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009cd0 tx=0x7fbee802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 -- 172.21.15.119:0/662866777 shutdown_connections 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 --2- 172.21.15.119:0/662866777 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef01124e0 0x7fbef01148d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 --2- 172.21.15.119:0/662866777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef00764f0 0x7fbef0111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 --2- 172.21.15.119:0/662866777 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef0075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 -- 172.21.15.119:0/662866777 >> 172.21.15.119:0/662866777 conn(0x7fbef00ff2f0 msgr2=0x7fbef0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 -- 172.21.15.119:0/662866777 shutdown_connections 2024-03-28T18:52:29.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.172+0000 7fbef5800640 1 -- 172.21.15.119:0/662866777 wait complete. 2024-03-28T18:52:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 Processor -- start 2024-03-28T18:52:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 -- start start 2024-03-28T18:52:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef00764f0 0x7fbef019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef01124e0 0x7fbef01a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0118040 con 0x7fbef01124e0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0117ec0 con 0x7fbef00764f0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbef5800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01181c0 con 0x7fbef0075bd0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35828/0 (socket says 172.21.15.119:35828) 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbeeea00640 1 -- 172.21.15.119:0/2830879491 learned_addr learned my addr 172.21.15.119:0/2830879491 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbeef400640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef01124e0 0x7fbef01a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.174+0000 7fbee7e00640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef00764f0 0x7fbef019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 -- 172.21.15.119:0/2830879491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef00764f0 msgr2=0x7fbef019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef00764f0 0x7fbef019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 -- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef01124e0 msgr2=0x7fbef01a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef01124e0 0x7fbef01a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 -- 172.21.15.119:0/2830879491 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef01a3500 con 0x7fbef0075bd0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbeeea00640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e9f0 tx=0x7fbee000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbee7400640 1 -- 172.21.15.119:0/2830879491 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee000cd80 con 0x7fbef0075bd0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef01a37f0 con 0x7fbef0075bd0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbee7400640 1 -- 172.21.15.119:0/2830879491 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbee0010910 con 0x7fbef0075bd0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.175+0000 7fbee7400640 1 -- 172.21.15.119:0/2830879491 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00188f0 con 0x7fbef0075bd0 2024-03-28T18:52:29.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.176+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef01ab080 con 0x7fbef0075bd0 2024-03-28T18:52:29.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 Processor -- start 2024-03-28T18:52:29.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbee7400640 1 -- 172.21.15.119:0/2830879491 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbee0010ac0 con 0x7fbef0075bd0 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 -- start start 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 0x55eeda272950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda272f20 0x55eeda2733a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda1d7250 con 0x55eeda273c00 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda17fa30 con 0x55eeda27c290 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbef8573740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda1d32f0 con 0x55eeda272f20 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda272f20 0x55eeda2733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44918/0 (socket says 172.21.15.119:44918) 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbeef400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 0x55eeda272950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.179+0000 7fbee7e00640 1 -- 172.21.15.119:0/1447316155 learned_addr learned my addr 172.21.15.119:0/1447316155 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 -- 172.21.15.119:0/1447316155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda272f20 msgr2=0x55eeda2733a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda272f20 0x55eeda2733a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 -- 172.21.15.119:0/1447316155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 msgr2=0x55eeda272950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 0x55eeda272950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 -- 172.21.15.119:0/1447316155 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeda16a690 con 0x55eeda27c290 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbeef400640 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 0x55eeda272950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009c10 tx=0x7fbed8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec891a4e881b8af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbee7400640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 0x7fbed40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbedfe00640 1 -- 172.21.15.119:0/1447316155 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8004230 con 0x55eeda27c290 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbedfe00640 1 -- 172.21.15.119:0/1447316155 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbed80043d0 con 0x55eeda27c290 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.180+0000 7fbedfe00640 1 -- 172.21.15.119:0/1447316155 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80197a0 con 0x55eeda27c290 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbee7400640 1 -- 172.21.15.119:0/2830879491 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbee00cf160 con 0x7fbef0075bd0 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbeef400640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 0x7fbed40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 -- 172.21.15.119:0/1447316155 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 msgr2=0x55eeda27e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009c10 tx=0x7fbed8011810 comp rx=0 tx=0).stop 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 -- 172.21.15.119:0/1447316155 shutdown_connections 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda27c290 0x55eeda27e680 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda272f20 0x55eeda2733a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 --2- 172.21.15.119:0/1447316155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda273c00 0x55eeda272950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 -- 172.21.15.119:0/1447316155 >> 172.21.15.119:0/1447316155 conn(0x55eeda23a200 msgr2=0x55eeda26d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbeef400640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 0x7fbed40a4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009bf0 tx=0x7fbee802e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.181+0000 7fbef8573740 1 -- 172.21.15.119:0/1447316155 shutdown_connections 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.182+0000 7fbef8573740 1 -- 172.21.15.119:0/1447316155 wait complete. 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.182+0000 7fbef8573740 1 -- start start 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.182+0000 7fbef8573740 1 Processor -- start 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 0x55eeda278c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 0x55eeda2791a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda27f100 con 0x55eeda27c290 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda192e80 con 0x55eeda272f20 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eeda27f280 con 0x55eeda273c00 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbef8573740 10 client.0 _ll_register_callbacks cb 0x55eeda1a5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 0x55eeda278c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44928/0 (socket says 172.21.15.119:44928) 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 0x55eeda278c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35840/0 (socket says 172.21.15.119:35840) 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 -- 172.21.15.119:0/130563641 learned_addr learned my addr 172.21.15.119:0/130563641 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeef400640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 0x55eeda2791a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 -- 172.21.15.119:0/130563641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 msgr2=0x55eeda278c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 0x55eeda278c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 -- 172.21.15.119:0/130563641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 msgr2=0x55eeda2791a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 0x55eeda2791a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 -- 172.21.15.119:0/130563641 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eed9fb57d0 con 0x55eeda272f20 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeef400640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 0x55eeda2791a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.183+0000 7fbeeea00640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e9c0 tx=0x7fbee0097b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=630ab4079b398e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee009b1e0 con 0x55eeda272f20 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbee009acd0 con 0x55eeda272f20 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbed80043d0 con 0x55eeda272f20 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbee009b880 con 0x55eeda272f20 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 10 client.27947 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:29.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eed9eeb3f0 con 0x55eeda272f20 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbed80043d0 con 0x55eeda272f20 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbee00afcd0 con 0x55eeda272f20 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 10 client.27947 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 10 client.27947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 20 client.27947 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda28c090 0x55eeda28e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 4 client.27947 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.184+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eeda1e4790 con 0x55eeda28c090 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.185+0000 7fbee7e00640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda28c090 0x55eeda28e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.185+0000 7fbee7e00640 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda28c090 0x55eeda28e480 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7ac70c2b58fdf7f server_cookie=6b82d95c090e9f1e in_seq=0 out_seq=0 2024-03-28T18:52:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.185+0000 7fbedcc00640 10 client.27947 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbedcc00640 1 -- 172.21.15.119:0/130563641 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eeda1e4790 con 0x55eeda28c090 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbedcc00640 10 client.27947 handle_command_reply: tid=0 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 client.27947 shutdown 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 msgr2=0x55eeda27b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e9c0 tx=0x7fbee0097b20 comp rx=0 tx=0).stop 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 shutdown_connections 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda28c090 0x55eeda28e480 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda27c290 0x55eeda2791a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda273c00 0x55eeda278c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbedcc00640 0 client.27947 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 --2- 172.21.15.119:0/130563641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda272f20 0x55eeda27b8d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 >> 172.21.15.119:0/130563641 conn(0x55eeda23a200 msgr2=0x55eeda26f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 shutdown_connections 2024-03-28T18:52:29.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 1 -- 172.21.15.119:0/130563641 wait complete. 2024-03-28T18:52:29.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.186+0000 7fbef8573740 20 client.27947 trim_cache size 0 max 16384 2024-03-28T18:52:29.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 Processor -- start 2024-03-28T18:52:29.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 -- start start 2024-03-28T18:52:29.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda27b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda27d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b3a00 con 0x55eeda272f20 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b5380 con 0x55eeda273c00 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbef8573740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b4060 con 0x55eeda27c290 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbeeea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda27b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbeef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda27d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44940/0 (socket says 172.21.15.119:44940) 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbeef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda27d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35854/0 (socket says 172.21.15.119:35854) 2024-03-28T18:52:29.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.204+0000 7fbee7e00640 1 -- 172.21.15.119:0/1006543687 learned_addr learned my addr 172.21.15.119:0/1006543687 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 -- 172.21.15.119:0/1006543687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 msgr2=0x55eeda27d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda27d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 -- 172.21.15.119:0/1006543687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 msgr2=0x55eeda27b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda27b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 -- 172.21.15.119:0/1006543687 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eeda28e7f0 con 0x55eeda273c00 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbee7e00640 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbed8003f10 tx=0x7fbed80193e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e2c8cf8650646b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbedf400640 1 -- 172.21.15.119:0/1006543687 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8031070 con 0x55eeda273c00 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbedf400640 1 -- 172.21.15.119:0/1006543687 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbed8029070 con 0x55eeda273c00 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.205+0000 7fbedf400640 1 -- 172.21.15.119:0/1006543687 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed801f040 con 0x55eeda273c00 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- 172.21.15.119:0/1006543687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 msgr2=0x55eeda27b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbed8003f10 tx=0x7fbed80193e0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- 172.21.15.119:0/1006543687 shutdown_connections 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda27d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda27b8d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 --2- 172.21.15.119:0/1006543687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda27b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- 172.21.15.119:0/1006543687 >> 172.21.15.119:0/1006543687 conn(0x55eeda23a200 msgr2=0x55eeda26d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- 172.21.15.119:0/1006543687 shutdown_connections 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- 172.21.15.119:0/1006543687 wait complete. 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.206+0000 7fbef8573740 1 -- start start 2024-03-28T18:52:29.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 Processor -- start 2024-03-28T18:52:29.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda283ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda2762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b4d20 con 0x55eeda272f20 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b46c0 con 0x55eeda273c00 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef01b1be0 con 0x55eeda27c290 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbef8573740 10 client.0 _ll_register_callbacks cb 0x7fbee00d2a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbee7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:44946/0 (socket says 172.21.15.119:44946) 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbee7e00640 1 -- 172.21.15.119:0/3023229707 learned_addr learned my addr 172.21.15.119:0/3023229707 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbeef400640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda2762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.207+0000 7fbeeea00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda283ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 -- 172.21.15.119:0/3023229707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 msgr2=0x55eeda2762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda2762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 -- 172.21.15.119:0/3023229707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 msgr2=0x55eeda283ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda283ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 -- 172.21.15.119:0/3023229707 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eed9eab0f0 con 0x55eeda273c00 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbeeea00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda283ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbeef400640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda2762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbee7e00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005700 tx=0x7fbed8011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe618ce37359c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbedcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8031070 con 0x55eeda273c00 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbed8029070 con 0x55eeda273c00 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbed80038d0 con 0x55eeda273c00 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbed8005bf0 con 0x55eeda273c00 2024-03-28T18:52:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbef8573740 10 client.27953 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.208+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eeda2e8470 con 0x55eeda273c00 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbed8029070 con 0x55eeda273c00 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbed8018070 con 0x55eeda273c00 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 10 client.27953 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 10 client.27953 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 20 client.27953 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda2e88c0 0x55eeda2eace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 4 client.27953 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eeda1deae0 con 0x55eeda2e88c0 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbeeea00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda2e88c0 0x55eeda2eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbedcc00640 10 client.27953 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:29.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.209+0000 7fbeeea00640 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda2e88c0 0x55eeda2eace0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2398d0fa7d1d3fb9 server_cookie=2fbfa861609144f6 in_seq=0 out_seq=0 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbedcc00640 1 -- 172.21.15.119:0/3023229707 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55eeda1deae0 con 0x55eeda2e88c0 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbedcc00640 10 client.27953 handle_command_reply: tid=0 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 client.27953 shutdown 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 msgr2=0x55eeda275d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005700 tx=0x7fbed8011dc0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 shutdown_connections 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55eeda2e88c0 0x55eeda2eace0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55eeda27c290 0x55eeda2762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55eeda273c00 0x55eeda275d80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 --2- 172.21.15.119:0/3023229707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eeda272f20 0x55eeda283ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbedcc00640 0 client.27953 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:29.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 >> 172.21.15.119:0/3023229707 conn(0x55eeda23a200 msgr2=0x55eeda26c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 shutdown_connections 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 1 -- 172.21.15.119:0/3023229707 wait complete. 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.210+0000 7fbef8573740 20 client.27953 trim_cache size 0 max 16384 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout: "f2605f87-22e0-469b-87ec-6d62c1a6d358": { 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:52:29.218 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "f2605f87-22e0-469b-87ec-6d62c1a6d358", 2024-03-28T18:52:29.219 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:52:29.219 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:52:29.219 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:52:29.219 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 msgr2=0x7fbed40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 0x7fbed40a4eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009bf0 tx=0x7fbee802e040 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 msgr2=0x7fbef019e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e9f0 tx=0x7fbee000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 shutdown_connections 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbed40a29f0 0x7fbed40a4eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbef01124e0 0x7fbef01a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbef00764f0 0x7fbef019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 --2- 172.21.15.119:0/2830879491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbef0075bd0 0x7fbef019e560 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:29.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 >> 172.21.15.119:0/2830879491 conn(0x7fbef00ff2f0 msgr2=0x7fbef01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:29.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 shutdown_connections 2024-03-28T18:52:29.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:29.211+0000 7fbef5800640 1 -- 172.21.15.119:0/2830879491 wait complete. 2024-03-28T18:52:29.230 INFO:tasks.cephfs.filesystem:scrub status for tag:f2605f87-22e0-469b-87ec-6d62c1a6d358 - {'path': '/', 'tag': 'f2605f87-22e0-469b-87ec-6d62c1a6d358', 'options': 'recursive,force'} 2024-03-28T18:52:29.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:29 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:52:29.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:29 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:29.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:29 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:52:29.686 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:29 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:29.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:29 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:52:29.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:29 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:29.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:29 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:52:29.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:29 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:29.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:29 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:52:29.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:29 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:29.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:29 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:52:29.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:29 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:52:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:30 smithi119 ceph-mon[31813]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:30.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:30 smithi196 ceph-mon[36300]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:30.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:30 smithi131 ceph-mon[36065]: pgmap v260: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:32 smithi119 ceph-mon[31813]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:32.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:32 smithi196 ceph-mon[36300]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:32.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:32 smithi131 ceph-mon[36065]: pgmap v261: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:33.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:33 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:52:33.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:33 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:52:33.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:33 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:52:34.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:34 smithi119 ceph-mon[31813]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:34.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:34 smithi196 ceph-mon[36300]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:34 smithi131 ceph-mon[36065]: pgmap v262: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:36 smithi119 ceph-mon[31813]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:36.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:36 smithi196 ceph-mon[36300]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:36 smithi131 ceph-mon[36065]: pgmap v263: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:38.736 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:38 smithi196 ceph-mon[36300]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:38 smithi119 ceph-mon[31813]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:38 smithi131 ceph-mon[36065]: pgmap v264: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:52:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:40 smithi119 ceph-mon[31813]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:40 smithi196 ceph-mon[36300]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:40 smithi131 ceph-mon[36065]: pgmap v265: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:52:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:42 smithi119 ceph-mon[31813]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:42 smithi196 ceph-mon[36300]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:42.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:42 smithi131 ceph-mon[36065]: pgmap v266: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:44 smithi119 ceph-mon[31813]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:44 smithi196 ceph-mon[36300]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:44 smithi131 ceph-mon[36065]: pgmap v267: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:46 smithi119 ceph-mon[31813]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:46 smithi196 ceph-mon[36300]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:46 smithi131 ceph-mon[36065]: pgmap v268: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:48 smithi119 ceph-mon[31813]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:48 smithi196 ceph-mon[36300]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:48 smithi131 ceph-mon[36065]: pgmap v269: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:50 smithi119 ceph-mon[31813]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:50.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:50 smithi196 ceph-mon[36300]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:50 smithi131 ceph-mon[36065]: pgmap v270: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:52 smithi119 ceph-mon[31813]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:52 smithi196 ceph-mon[36300]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:52.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:52 smithi131 ceph-mon[36065]: pgmap v271: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:54 smithi119 ceph-mon[31813]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:54 smithi196 ceph-mon[36300]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:54 smithi131 ceph-mon[36065]: pgmap v272: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:56 smithi119 ceph-mon[31813]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:56 smithi196 ceph-mon[36300]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:56 smithi131 ceph-mon[36065]: pgmap v273: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:52:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:52:58 smithi119 ceph-mon[31813]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:52:58 smithi196 ceph-mon[36300]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:52:58 smithi131 ceph-mon[36065]: pgmap v274: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:52:59.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:52:59.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:52:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 -- 172.21.15.119:0/357008433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 msgr2=0x7f32cc103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 --2- 172.21.15.119:0/357008433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc103cc0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009930 tx=0x7f32b402bc20 comp rx=0 tx=0).stop 2024-03-28T18:52:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 -- 172.21.15.119:0/357008433 shutdown_connections 2024-03-28T18:52:59.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 --2- 172.21.15.119:0/357008433 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc104200 0x7f32cc110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 --2- 172.21.15.119:0/357008433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc103cc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 --2- 172.21.15.119:0/357008433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc109850 0x7f32cc109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 -- 172.21.15.119:0/357008433 >> 172.21.15.119:0/357008433 conn(0x7f32cc0ff2f0 msgr2=0x7f32cc101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 -- 172.21.15.119:0/357008433 shutdown_connections 2024-03-28T18:52:59.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.456+0000 7f32d2400640 1 -- 172.21.15.119:0/357008433 wait complete. 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.457+0000 7f32d2400640 1 Processor -- start 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.457+0000 7f32d2400640 1 -- start start 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc104200 0x7f32cc0760f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 0x7f32cc076630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113c20 con 0x7f32cc104200 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113aa0 con 0x7f32cc109850 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32d2400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113da0 con 0x7f32cc103840 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32caa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc104200 0x7f32cc0760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35520/0 (socket says 172.21.15.119:35520) 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32cb400640 1 -- 172.21.15.119:0/4281607676 learned_addr learned my addr 172.21.15.119:0/4281607676 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.458+0000 7f32cbe00640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 0x7f32cc076630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 -- 172.21.15.119:0/4281607676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 msgr2=0x7f32cc076630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 0x7f32cc076630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 -- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc104200 msgr2=0x7f32cc0760f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc104200 0x7f32cc0760f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 -- 172.21.15.119:0/4281607676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc1aa830 con 0x7f32cc103840 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cbe00640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 0x7f32cc076630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32cb400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00e9f0 tx=0x7f32bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32c3e00640 1 -- 172.21.15.119:0/4281607676 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00cd80 con 0x7f32cc103840 2024-03-28T18:52:59.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32c3e00640 1 -- 172.21.15.119:0/4281607676 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32bc010910 con 0x7f32cc103840 2024-03-28T18:52:59.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc1aab20 con 0x7f32cc103840 2024-03-28T18:52:59.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32c3e00640 1 -- 172.21.15.119:0/4281607676 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc018a00 con 0x7f32cc103840 2024-03-28T18:52:59.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.459+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc1ab040 con 0x7f32cc103840 2024-03-28T18:52:59.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.462+0000 7f32c3e00640 1 -- 172.21.15.119:0/4281607676 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f32bc010ac0 con 0x7f32cc103840 2024-03-28T18:52:59.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 Processor -- start 2024-03-28T18:52:59.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 -- start start 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 0x5589675293a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x558967534680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55896748d250 con 0x558967532290 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558967435a30 con 0x558967528f20 2024-03-28T18:52:59.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.463+0000 7f32d5167740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589674892f0 con 0x558967529c00 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 0x5589675293a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35528/0 (socket says 172.21.15.119:35528) 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 -- 172.21.15.119:0/966993516 learned_addr learned my addr 172.21.15.119:0/966993516 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32caa00640 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x558967534680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32c3e00640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 0x7f32b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 -- 172.21.15.119:0/966993516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 msgr2=0x5589675293a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 0x5589675293a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 -- 172.21.15.119:0/966993516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 msgr2=0x558967534680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x558967534680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 -- 172.21.15.119:0/966993516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558967420690 con 0x558967529c00 2024-03-28T18:52:59.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32c3e00640 1 -- 172.21.15.119:0/4281607676 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f32bc0cf350 con 0x7f32cc103840 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cb400640 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 0x5589675293a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.464+0000 7f32cbe00640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 0x7f32b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32cbe00640 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f32c4010620 tx=0x7f32c4010ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbcb2bff4041442d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32c0c00640 1 -- 172.21.15.119:0/966993516 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4019070 con 0x558967529c00 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32c0c00640 1 -- 172.21.15.119:0/966993516 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32c4022070 con 0x558967529c00 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32c0c00640 1 -- 172.21.15.119:0/966993516 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c4026d40 con 0x558967529c00 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32d5167740 1 -- 172.21.15.119:0/966993516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 msgr2=0x558967528950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32cbe00640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 0x7f32b00a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f32c4021000 tx=0x7f32c401f000 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.465+0000 7f32d5167740 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f32c4010620 tx=0x7f32c4010ae0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 -- 172.21.15.119:0/966993516 shutdown_connections 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x558967534680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967528f20 0x5589675293a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 --2- 172.21.15.119:0/966993516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967529c00 0x558967528950 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 -- 172.21.15.119:0/966993516 >> 172.21.15.119:0/966993516 conn(0x5589674f0200 msgr2=0x558967523970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 -- 172.21.15.119:0/966993516 shutdown_connections 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 -- 172.21.15.119:0/966993516 wait complete. 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 -- start start 2024-03-28T18:52:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.466+0000 7f32d5167740 1 Processor -- start 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967529c00 0x55896752ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x55896752f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558967535100 con 0x558967532290 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x558967448e80 con 0x558967529c00 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x558967535280 con 0x558967528f20 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32d5167740 10 client.0 _ll_register_callbacks cb 0x55896745b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35536/0 (socket says 172.21.15.119:35536) 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 -- 172.21.15.119:0/3664632269 learned_addr learned my addr 172.21.15.119:0/3664632269 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cbe00640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x55896752f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32caa00640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967529c00 0x55896752ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 -- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967529c00 msgr2=0x55896752ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967529c00 0x55896752ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 -- 172.21.15.119:0/3664632269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 msgr2=0x55896752f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x55896752f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.467+0000 7f32cb400640 1 -- 172.21.15.119:0/3664632269 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55896726b7d0 con 0x558967528f20 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32cb400640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0cca10 tx=0x7f32bc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a28b705256e7da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:59.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc09fa10 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f32bc0ccc30 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32c4022070 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32bc0cce30 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32d5167740 10 client.27943 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5589671a13f0 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32c4022070 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.468+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f32bc018cc0 con 0x558967528f20 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 10 client.27943 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 10 client.27943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 20 client.27943 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558967542170 0x558967544560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 4 client.27943 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55896749a790 con 0x558967542170 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32caa00640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558967542170 0x558967544560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32caa00640 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558967542170 0x558967544560 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f959ceaf08c6173 server_cookie=7115ac42476b8ef7 in_seq=0 out_seq=0 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.469+0000 7f32b9600640 10 client.27943 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.470+0000 7f32b9600640 1 -- 172.21.15.119:0/3664632269 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55896749a790 con 0x558967542170 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.470+0000 7f32b9600640 10 client.27943 handle_command_reply: tid=0 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.470+0000 7f32d5167740 1 client.27943 shutdown 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 msgr2=0x5589675318c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0cca10 tx=0x7f32bc000bc0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 shutdown_connections 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x558967542170 0x558967544560 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967532290 0x55896752f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967529c00 0x55896752ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 --2- 172.21.15.119:0/3664632269 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x5589675318c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32b9600640 0 client.27943 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 >> 172.21.15.119:0/3664632269 conn(0x5589674f0200 msgr2=0x5589675256c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 shutdown_connections 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 1 -- 172.21.15.119:0/3664632269 wait complete. 2024-03-28T18:52:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.471+0000 7f32d5167740 20 client.27943 trim_cache size 0 max 16384 2024-03-28T18:52:59.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.488+0000 7f32d5167740 1 Processor -- start 2024-03-28T18:52:59.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.488+0000 7f32d5167740 1 -- start start 2024-03-28T18:52:59.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.488+0000 7f32d5167740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.488+0000 7f32d5167740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967533680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32d5167740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967532290 0x558967533bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32d5167740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b3af0 con 0x558967529c00 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32d5167740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b5470 con 0x558967532290 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32d5167740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b4150 con 0x558967528f20 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967532290 0x558967533bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35544/0 (socket says 172.21.15.119:35544) 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32caa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967533680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 -- 172.21.15.119:0/2168917358 learned_addr learned my addr 172.21.15.119:0/2168917358 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 -- 172.21.15.119:0/2168917358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967532290 msgr2=0x558967533bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967532290 0x558967533bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 -- 172.21.15.119:0/2168917358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 msgr2=0x558967533680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967533680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 -- 172.21.15.119:0/2168917358 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589675448d0 con 0x558967528f20 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.489+0000 7f32cb400640 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f32bc098290 tx=0x7f32bc0cf240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=609b40fbdf232689 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32bbe00640 1 -- 172.21.15.119:0/2168917358 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc0a4060 con 0x558967528f20 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32bbe00640 1 -- 172.21.15.119:0/2168917358 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32bc0b4020 con 0x558967528f20 2024-03-28T18:52:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32bbe00640 1 -- 172.21.15.119:0/2168917358 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc0862a0 con 0x558967528f20 2024-03-28T18:52:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 -- 172.21.15.119:0/2168917358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 msgr2=0x55896752e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f32bc098290 tx=0x7f32bc0cf240 comp rx=0 tx=0).stop 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 -- 172.21.15.119:0/2168917358 shutdown_connections 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x558967532290 0x558967533bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967533680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 --2- 172.21.15.119:0/2168917358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967528f20 0x55896752e530 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 -- 172.21.15.119:0/2168917358 >> 172.21.15.119:0/2168917358 conn(0x5589674f0200 msgr2=0x558967522a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.490+0000 7f32d5167740 1 -- 172.21.15.119:0/2168917358 shutdown_connections 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.491+0000 7f32d5167740 1 -- 172.21.15.119:0/2168917358 wait complete. 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.491+0000 7f32d5167740 1 -- start start 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.491+0000 7f32d5167740 1 Processor -- start 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.491+0000 7f32d5167740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967539ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 0x55896752c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b4e10 con 0x558967529c00 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b47b0 con 0x55896752c150 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1b1cd0 con 0x558967532290 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32d5167740 10 client.0 _ll_register_callbacks cb 0x7f32bc00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32cbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 0x55896752c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32cb400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967539ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35546/0 (socket says 172.21.15.119:35546) 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 -- 172.21.15.119:0/1132831382 learned_addr learned my addr 172.21.15.119:0/1132831382 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 -- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 msgr2=0x55896752c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 0x55896752c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 -- 172.21.15.119:0/1132831382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 msgr2=0x558967539ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:52:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967539ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 -- 172.21.15.119:0/1132831382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589671610f0 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32cbe00640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 0x55896752c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32caa00640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f32b4036040 tx=0x7f32b4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f96c8ae6a4782dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.492+0000 7f32b9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b404d070 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f32b4004040 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32bc0b4020 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32b4004260 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32d5167740 10 client.27949 fetch_fsmap learned FSMap version 28 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55896759e940 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32bc0b4020 con 0x558967532290 2024-03-28T18:52:59.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f32b4005230 con 0x558967532290 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.493+0000 7f32d5167740 10 client.27949 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 10 client.27949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 20 client.27949 populate_metadata read hostname 'smithi119' 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55896759f050 0x5589675a1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 4 client.27949 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558967494ae0 con 0x55896759f050 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32cb400640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55896759f050 0x5589675a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32cb400640 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55896759f050 0x5589675a1470 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91be350976219249 server_cookie=139d29b397c53e28 in_seq=0 out_seq=0 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32b9600640 10 client.27949 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32b9600640 1 -- 172.21.15.119:0/1132831382 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558967494ae0 con 0x55896759f050 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32b9600640 10 client.27949 handle_command_reply: tid=0 2024-03-28T18:52:59.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.494+0000 7f32d5167740 1 client.27949 shutdown 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 msgr2=0x55896752bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f32b4036040 tx=0x7f32b4000bc0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 shutdown_connections 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55896759f050 0x5589675a1470 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55896752c150 0x55896752c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32b9600640 0 client.27949 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558967532290 0x55896752bc10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 --2- 172.21.15.119:0/1132831382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558967529c00 0x558967539ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 >> 172.21.15.119:0/1132831382 conn(0x5589674f0200 msgr2=0x558967524a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 shutdown_connections 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 1 -- 172.21.15.119:0/1132831382 wait complete. 2024-03-28T18:52:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.495+0000 7f32d5167740 20 client.27949 trim_cache size 0 max 16384 2024-03-28T18:52:59.501 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:52:59.501 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:52:59.501 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:52:59.501 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 msgr2=0x7f32b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 0x7f32b00a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f32c4021000 tx=0x7f32c401f000 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 msgr2=0x7f32cc075bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00e9f0 tx=0x7f32bc00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 shutdown_connections 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f32b00a2970 0x7f32b00a4e30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f32cc109850 0x7f32cc076630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc104200 0x7f32cc0760f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 --2- 172.21.15.119:0/4281607676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32cc103840 0x7f32cc075bb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 >> 172.21.15.119:0/4281607676 conn(0x7f32cc0ff2f0 msgr2=0x7f32cc0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 shutdown_connections 2024-03-28T18:52:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.496+0000 7f32d2400640 1 -- 172.21.15.119:0/4281607676 wait complete. 2024-03-28T18:52:59.514 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:52:59.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:52:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 -- 172.21.15.119:0/3081811602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c103200 msgr2=0x7f736c10b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 --2- 172.21.15.119:0/3081811602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c103200 0x7f736c10b6e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802bcc0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 -- 172.21.15.119:0/3081811602 shutdown_connections 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 --2- 172.21.15.119:0/3081811602 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c10bcb0 0x7f736c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 --2- 172.21.15.119:0/3081811602 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c103200 0x7f736c10b6e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 --2- 172.21.15.119:0/3081811602 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 -- 172.21.15.119:0/3081811602 >> 172.21.15.119:0/3081811602 conn(0x7f736c076920 msgr2=0x7f736c076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 -- 172.21.15.119:0/3081811602 shutdown_connections 2024-03-28T18:52:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.722+0000 7f7373c00640 1 -- 172.21.15.119:0/3081811602 wait complete. 2024-03-28T18:52:59.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.723+0000 7f7373c00640 1 Processor -- start 2024-03-28T18:52:59.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.723+0000 7f7373c00640 1 -- start start 2024-03-28T18:52:59.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c103200 0x7f736c1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c111a90 con 0x7f736c103200 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c111910 con 0x7f736c10bcb0 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7373c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c111c10 con 0x7f736c1028e0 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35564/0 (socket says 172.21.15.119:35564) 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57726/0 (socket says 172.21.15.119:57726) 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 -- 172.21.15.119:0/3072626049 learned_addr learned my addr 172.21.15.119:0/3072626049 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f736be00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c103200 0x7f736c1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 -- 172.21.15.119:0/3072626049 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 msgr2=0x7f736c19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 -- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c103200 msgr2=0x7f736c1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c103200 0x7f736c1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371200640 1 -- 172.21.15.119:0/3072626049 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c19cf60 con 0x7f736c1028e0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.724+0000 7f7371c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7371200640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9f0 tx=0x7f736000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736000cd80 con 0x7f736c1028e0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7360005500 con 0x7f736c1028e0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c19d250 con 0x7f736c1028e0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360010690 con 0x7f736c1028e0 2024-03-28T18:52:59.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.725+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c1a4ae0 con 0x7f736c1028e0 2024-03-28T18:52:59.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.726+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c06a530 con 0x7f736c1028e0 2024-03-28T18:52:59.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.728+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f73600046c0 con 0x7f736c1028e0 2024-03-28T18:52:59.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.729+0000 7f7369600640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 0x7f73540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:52:59.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.729+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f73600cf340 con 0x7f736c1028e0 2024-03-28T18:52:59.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.729+0000 7f736be00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 0x7f73540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:52:59.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.730+0000 7f736be00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 0x7f73540a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7358005ab0 tx=0x7f7358005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:52:59.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.733+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f736009bfb0 con 0x7f736c1028e0 2024-03-28T18:52:59.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.969+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f736c1992c0 con 0x7f736c1028e0 2024-03-28T18:52:59.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.970+0000 7f7369600640 1 -- 172.21.15.119:0/3072626049 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f736009b8b0 con 0x7f736c1028e0 2024-03-28T18:52:59.970 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:52:59.971 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:52:59.971 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 msgr2=0x7f73540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 0x7f73540a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7358005ab0 tx=0x7f7358005a20 comp rx=0 tx=0).stop 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 msgr2=0x7f736c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9f0 tx=0x7f736000edf0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 shutdown_connections 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f73540a2970 0x7f73540a4e30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f736c10bcb0 0x7f736c19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f736c103200 0x7f736c1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 --2- 172.21.15.119:0/3072626049 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f736c1028e0 0x7f736c197f70 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:52:59.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 >> 172.21.15.119:0/3072626049 conn(0x7f736c076920 msgr2=0x7f736c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:52:59.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 shutdown_connections 2024-03-28T18:52:59.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:52:59.973+0000 7f7373c00640 1 -- 172.21.15.119:0/3072626049 wait complete. 2024-03-28T18:52:59.991 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:52:59.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:53:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 -- 172.21.15.119:0/660183671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484073a70 msgr2=0x7f4484073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 --2- 172.21.15.119:0/660183671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484073a70 0x7f4484073e50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4478009900 tx=0x7f447802bcd0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 -- 172.21.15.119:0/660183671 shutdown_connections 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 --2- 172.21.15.119:0/660183671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f448410ddb0 0x7f44841101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 --2- 172.21.15.119:0/660183671 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4484074390 0x7f448410d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 --2- 172.21.15.119:0/660183671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484073a70 0x7f4484073e50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 -- 172.21.15.119:0/660183671 >> 172.21.15.119:0/660183671 conn(0x7f44840fd040 msgr2=0x7f44840ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 -- 172.21.15.119:0/660183671 shutdown_connections 2024-03-28T18:53:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.226+0000 7f448ae00640 1 -- 172.21.15.119:0/660183671 wait complete. 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.227+0000 7f448ae00640 1 Processor -- start 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 -- start start 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4484073a70 0x7f448419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f448410ddb0 0x7f448419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4484113b10 con 0x7f4484073a70 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4484113990 con 0x7f448410ddb0 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4484113c90 con 0x7f4484074390 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f4489000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f4489000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35588/0 (socket says 172.21.15.119:35588) 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f4489000640 1 -- 172.21.15.119:0/1933213917 learned_addr learned my addr 172.21.15.119:0/1933213917 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f448a400640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f448410ddb0 0x7f448419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.228+0000 7f4489a00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4484073a70 0x7f448419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 -- 172.21.15.119:0/1933213917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f448410ddb0 msgr2=0x7f448419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f448410ddb0 0x7f448419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 -- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4484073a70 msgr2=0x7f448419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4484073a70 0x7f448419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 -- 172.21.15.119:0/1933213917 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448419f010 con 0x7f4484074390 2024-03-28T18:53:00.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4489000640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f447000e9f0 tx=0x7f447000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4482000640 1 -- 172.21.15.119:0/1933213917 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000cd80 con 0x7f4484074390 2024-03-28T18:53:00.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4482000640 1 -- 172.21.15.119:0/1933213917 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4470010910 con 0x7f4484074390 2024-03-28T18:53:00.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f4482000640 1 -- 172.21.15.119:0/1933213917 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470018990 con 0x7f4484074390 2024-03-28T18:53:00.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.229+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448419f300 con 0x7f4484074390 2024-03-28T18:53:00.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.230+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44841a6b90 con 0x7f4484074390 2024-03-28T18:53:00.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 Processor -- start 2024-03-28T18:53:00.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 -- start start 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 0x56552b296230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b1ee2c0 con 0x56552b28a8b0 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b196c90 con 0x56552b28b260 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.234+0000 7f448f078740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b1ea360 con 0x56552b293de0 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57762/0 (socket says 172.21.15.119:57762) 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48280/0 (socket says 172.21.15.119:48280) 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489a00640 1 -- 172.21.15.119:0/3235688735 learned_addr learned my addr 172.21.15.119:0/3235688735 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4482000640 1 -- 172.21.15.119:0/1933213917 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4470010ac0 con 0x7f4484074390 2024-03-28T18:53:00.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489000640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 0x56552b296230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 -- 172.21.15.119:0/3235688735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 msgr2=0x56552b296230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 0x56552b296230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 -- 172.21.15.119:0/3235688735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 msgr2=0x56552b293810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 -- 172.21.15.119:0/3235688735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56552b1bb080 con 0x56552b28a8b0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489000640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 0x56552b296230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f4489a00640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.235+0000 7f448a400640 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44840749c0 tx=0x7f447c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fedd45dae92fb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4476a00640 1 -- 172.21.15.119:0/3235688735 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c004d20 con 0x56552b28a8b0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4476a00640 1 -- 172.21.15.119:0/3235688735 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f447c004050 con 0x56552b28a8b0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4476a00640 1 -- 172.21.15.119:0/3235688735 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c005280 con 0x56552b28a8b0 2024-03-28T18:53:00.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4482000640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 0x7f446c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4482000640 1 -- 172.21.15.119:0/1933213917 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f44700cf1f0 con 0x7f4484074390 2024-03-28T18:53:00.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.236+0000 7f4489a00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 0x7f446c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- 172.21.15.119:0/3235688735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 msgr2=0x56552b28ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44840749c0 tx=0x7f447c011810 comp rx=0 tx=0).stop 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- 172.21.15.119:0/3235688735 shutdown_connections 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b293de0 0x56552b296230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28b260 0x56552b293810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 --2- 172.21.15.119:0/3235688735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28a8b0 0x56552b28ac90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- 172.21.15.119:0/3235688735 >> 172.21.15.119:0/3235688735 conn(0x56552b284f80 msgr2=0x56552b285360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- 172.21.15.119:0/3235688735 shutdown_connections 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- 172.21.15.119:0/3235688735 wait complete. 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f4489a00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 0x7f446c0a4eb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56552b28b660 tx=0x7f447802cf90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.237+0000 7f448f078740 1 -- start start 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 Processor -- start 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28b260 0x56552b290a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b293de0 0x56552b291030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b296d50 con 0x56552b28b260 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b1aa0d0 con 0x56552b293de0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552b296ed0 con 0x56552b28a8b0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f448f078740 10 client.0 _ll_register_callbacks cb 0x56552b1aee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f4489a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f4489a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35600/0 (socket says 172.21.15.119:35600) 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.238+0000 7f4489a00640 1 -- 172.21.15.119:0/2287608297 learned_addr learned my addr 172.21.15.119:0/2287608297 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f448a400640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b293de0 0x56552b291030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489000640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28b260 0x56552b290a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 -- 172.21.15.119:0/2287608297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b293de0 msgr2=0x56552b291030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b293de0 0x56552b291030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 -- 172.21.15.119:0/2287608297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28b260 msgr2=0x56552b290a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28b260 0x56552b290a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 -- 172.21.15.119:0/2287608297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56552af023f0 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f4489a00640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4478009900 tx=0x7f4478009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d653e40ae1f4f884 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.239+0000 7f446b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4478004ad0 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4478004c70 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f447c004050 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4478004f10 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f448f078740 10 client.27964 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56552afcc7d0 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f447c004050 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.240+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4478046030 con 0x56552b28a8b0 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 10 client.27964 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 10 client.27964 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 20 client.27964 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b2a3bd0 0x56552b2a5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 4 client.27964 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:00.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56552b1fb800 con 0x56552b2a3bd0 2024-03-28T18:53:00.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f4489000640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b2a3bd0 0x56552b2a5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f4489000640 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b2a3bd0 0x56552b2a5fc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e2d3e0adfd3bb71a server_cookie=6b089d31d8efbc7d in_seq=0 out_seq=0 2024-03-28T18:53:00.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.241+0000 7f446b400640 10 client.27964 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.242+0000 7f446b400640 1 -- 172.21.15.119:0/2287608297 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56552b1fb800 con 0x56552b2a3bd0 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.242+0000 7f446b400640 10 client.27964 handle_command_reply: tid=0 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.242+0000 7f448f078740 1 client.27964 shutdown 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.242+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 msgr2=0x56552b293700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.242+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4478009900 tx=0x7f4478009260 comp rx=0 tx=0).stop 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 shutdown_connections 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b2a3bd0 0x56552b2a5fc0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b293de0 0x56552b291030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b28b260 0x56552b290a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f446b400640 0 client.27964 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 --2- 172.21.15.119:0/2287608297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28a8b0 0x56552b293700 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 >> 172.21.15.119:0/2287608297 conn(0x56552b284f80 msgr2=0x56552b286cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 shutdown_connections 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 1 -- 172.21.15.119:0/2287608297 wait complete. 2024-03-28T18:53:00.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.243+0000 7f448f078740 20 client.27964 trim_cache size 0 max 16384 2024-03-28T18:53:00.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 Processor -- start 2024-03-28T18:53:00.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 -- start start 2024-03-28T18:53:00.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b2931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b28dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841af3a0 con 0x56552b293de0 2024-03-28T18:53:00.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841b0d20 con 0x56552b28a8b0 2024-03-28T18:53:00.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.258+0000 7f448f078740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841afa00 con 0x56552b28b260 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35606/0 (socket says 172.21.15.119:35606) 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 -- 172.21.15.119:0/2145460115 learned_addr learned my addr 172.21.15.119:0/2145460115 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489a00640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b2931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f448a400640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b28dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 -- 172.21.15.119:0/2145460115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 msgr2=0x56552b2931c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b2931c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 -- 172.21.15.119:0/2145460115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 msgr2=0x56552b28dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b28dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f4489000640 1 -- 172.21.15.119:0/2145460115 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56552b2a65f0 con 0x56552b28b260 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.259+0000 7f448a400640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b28dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.260+0000 7f4489000640 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f447009b7a0 tx=0x7f4470000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28acfd56c7a52b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.260+0000 7f4476000640 1 -- 172.21.15.119:0/2145460115 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44700860c0 con 0x56552b28b260 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.260+0000 7f4476000640 1 -- 172.21.15.119:0/2145460115 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4470097e80 con 0x56552b28b260 2024-03-28T18:53:00.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.260+0000 7f4476000640 1 -- 172.21.15.119:0/2145460115 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470018d90 con 0x56552b28b260 2024-03-28T18:53:00.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- 172.21.15.119:0/2145460115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 msgr2=0x56552b293700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f447009b7a0 tx=0x7f4470000bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- 172.21.15.119:0/2145460115 shutdown_connections 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b28dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b293700 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 --2- 172.21.15.119:0/2145460115 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b2931c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- 172.21.15.119:0/2145460115 >> 172.21.15.119:0/2145460115 conn(0x56552b284f80 msgr2=0x56552b284160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- 172.21.15.119:0/2145460115 shutdown_connections 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- 172.21.15.119:0/2145460115 wait complete. 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 -- start start 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.261+0000 7f448f078740 1 Processor -- start 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b29aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b29af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841b06c0 con 0x56552b293de0 2024-03-28T18:53:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841b0060 con 0x56552b28a8b0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841ad580 con 0x56552b28b260 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f448f078740 10 client.0 _ll_register_callbacks cb 0x7f4478009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f4489a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f4489a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57792/0 (socket says 172.21.15.119:57792) 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f4489a00640 1 -- 172.21.15.119:0/4166026790 learned_addr learned my addr 172.21.15.119:0/4166026790 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.262+0000 7f4489000640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b29aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f448a400640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b29af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 -- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 msgr2=0x56552b29aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b29aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 -- 172.21.15.119:0/4166026790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 msgr2=0x56552b29af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b29af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 -- 172.21.15.119:0/4166026790 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56552b307c10 con 0x56552b28a8b0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f4489a00640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4478005ab0 tx=0x7f4478004b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b9b2fce845433d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f446b400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4478003a00 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f44780050b0 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4470097e80 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4478069050 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f448f078740 10 client.27995 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.263+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56552b308ee0 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4470097e80 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4478061020 con 0x56552b28a8b0 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 10 client.27995 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 10 client.27995 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 20 client.27995 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b309600 0x56552b30ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 4 client.27995 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56552b1f5b50 con 0x56552b309600 2024-03-28T18:53:00.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.264+0000 7f4489000640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b309600 0x56552b30ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f4489000640 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b309600 0x56552b30ba20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a1d7746408d9841b server_cookie=c4d302665f3c3ddd in_seq=0 out_seq=0 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f446b400640 10 client.27995 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f446b400640 1 -- 172.21.15.119:0/4166026790 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56552b1f5b50 con 0x56552b309600 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f446b400640 10 client.27995 handle_command_reply: tid=0 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 client.27995 shutdown 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 msgr2=0x56552b29a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4478005ab0 tx=0x7f4478004b00 comp rx=0 tx=0).stop 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 shutdown_connections 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x56552b309600 0x56552b30ba20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56552b293de0 0x56552b29af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56552b28b260 0x56552b29aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 --2- 172.21.15.119:0/4166026790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56552b28a8b0 0x56552b29a4c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f446b400640 0 client.27995 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:00.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 >> 172.21.15.119:0/4166026790 conn(0x56552b284f80 msgr2=0x56552b2838d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 shutdown_connections 2024-03-28T18:53:00.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 1 -- 172.21.15.119:0/4166026790 wait complete. 2024-03-28T18:53:00.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.265+0000 7f448f078740 20 client.27995 trim_cache size 0 max 16384 2024-03-28T18:53:00.271 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 msgr2=0x7f446c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 0x7f446c0a4eb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56552b28b660 tx=0x7f447802cf90 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 msgr2=0x7f448419a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f447000e9f0 tx=0x7f447000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 shutdown_connections 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f446c0a29f0 0x7f446c0a4eb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f448410ddb0 0x7f448419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4484074390 0x7f448419a5b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 --2- 172.21.15.119:0/1933213917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4484073a70 0x7f448419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 >> 172.21.15.119:0/1933213917 conn(0x7f44840fd040 msgr2=0x7f44840fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 shutdown_connections 2024-03-28T18:53:00.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.266+0000 7f448ae00640 1 -- 172.21.15.119:0/1933213917 wait complete. 2024-03-28T18:53:00.282 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:00.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:53:00.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 -- 172.21.15.119:0/3258312949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c1075e0 msgr2=0x7f4a6c1079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 --2- 172.21.15.119:0/3258312949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c1079c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009930 tx=0x7f4a5c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:53:00.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 -- 172.21.15.119:0/3258312949 shutdown_connections 2024-03-28T18:53:00.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 --2- 172.21.15.119:0/3258312949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c101ee0 0x7f4a6c10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 --2- 172.21.15.119:0/3258312949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101520 0x7f4a6c1019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 --2- 172.21.15.119:0/3258312949 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c1079c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 -- 172.21.15.119:0/3258312949 >> 172.21.15.119:0/3258312949 conn(0x7f4a6c0fd010 msgr2=0x7f4a6c0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 -- 172.21.15.119:0/3258312949 shutdown_connections 2024-03-28T18:53:00.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.471+0000 7f4a70a00640 1 -- 172.21.15.119:0/3258312949 wait complete. 2024-03-28T18:53:00.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 Processor -- start 2024-03-28T18:53:00.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 -- start start 2024-03-28T18:53:00.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101ee0 0x7f4a6c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c111910 con 0x7f4a6c101ee0 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c111790 con 0x7f4a6c101520 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.472+0000 7f4a70a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c111a90 con 0x7f4a6c1075e0 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57806/0 (socket says 172.21.15.119:57806) 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 -- 172.21.15.119:0/431408691 learned_addr learned my addr 172.21.15.119:0/431408691 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a69e00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101ee0 0x7f4a6c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 -- 172.21.15.119:0/431408691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 msgr2=0x7f4a6c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 -- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101ee0 msgr2=0x7f4a6c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101ee0 0x7f4a6c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 -- 172.21.15.119:0/431408691 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c19cd60 con 0x7f4a6c101520 2024-03-28T18:53:00.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6b200640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.473+0000 7f4a6a800640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c005ab0 tx=0x7f4a5c0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.474+0000 7f4a63400640 1 -- 172.21.15.119:0/431408691 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c043020 con 0x7f4a6c101520 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.474+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c19cff0 con 0x7f4a6c101520 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.474+0000 7f4a63400640 1 -- 172.21.15.119:0/431408691 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c0044d0 con 0x7f4a6c101520 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.474+0000 7f4a63400640 1 -- 172.21.15.119:0/431408691 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c003bb0 con 0x7f4a6c101520 2024-03-28T18:53:00.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.474+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c1a48e0 con 0x7f4a6c101520 2024-03-28T18:53:00.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 Processor -- start 2024-03-28T18:53:00.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a63400640 1 -- 172.21.15.119:0/431408691 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4a5c046020 con 0x7f4a6c101520 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 -- start start 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b641c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b5982c0 con 0x55910b634630 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b540c90 con 0x55910b634fe0 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a7418b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b594360 con 0x55910b6421f0 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.477+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b641c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48312/0 (socket says 172.21.15.119:48312) 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35640/0 (socket says 172.21.15.119:35640) 2024-03-28T18:53:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a6b200640 1 -- 172.21.15.119:0/4019489286 learned_addr learned my addr 172.21.15.119:0/4019489286 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 -- 172.21.15.119:0/4019489286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 msgr2=0x55910b641c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b641c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 -- 172.21.15.119:0/4019489286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 msgr2=0x55910b634a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 -- 172.21.15.119:0/4019489286 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55910b565080 con 0x55910b6421f0 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a6b200640 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a69e00640 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4a54005d00 tx=0x7f4a540107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c16ae62f9df4fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a63400640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 0x7f4a500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a5b400640 1 -- 172.21.15.119:0/4019489286 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5401c680 con 0x55910b6421f0 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a5b400640 1 -- 172.21.15.119:0/4019489286 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5400be20 con 0x55910b6421f0 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.478+0000 7f4a5b400640 1 -- 172.21.15.119:0/4019489286 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54011c40 con 0x55910b6421f0 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a63400640 1 -- 172.21.15.119:0/431408691 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4a5c0f0d90 con 0x7f4a6c101520 2024-03-28T18:53:00.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a6b200640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 0x7f4a500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 -- 172.21.15.119:0/4019489286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 msgr2=0x55910b644600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4a54005d00 tx=0x7f4a540107a0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 -- 172.21.15.119:0/4019489286 shutdown_connections 2024-03-28T18:53:00.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b6421f0 0x55910b644600 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b641c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 --2- 172.21.15.119:0/4019489286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634630 0x55910b634a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 -- 172.21.15.119:0/4019489286 >> 172.21.15.119:0/4019489286 conn(0x55910b62ef80 msgr2=0x55910b62f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 -- 172.21.15.119:0/4019489286 shutdown_connections 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a6b200640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 0x7f4a500a4e30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009fd0 tx=0x7f4a64009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.479+0000 7f4a7418b740 1 -- 172.21.15.119:0/4019489286 wait complete. 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 -- start start 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 Processor -- start 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b641b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b63f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b645120 con 0x55910b6421f0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b5540d0 con 0x55910b634fe0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55910b6452a0 con 0x55910b634630 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.480+0000 7f4a7418b740 10 client.0 _ll_register_callbacks cb 0x55910b558e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b63f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b641b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57824/0 (socket says 172.21.15.119:57824) 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a69e00640 1 -- 172.21.15.119:0/3390798510 learned_addr learned my addr 172.21.15.119:0/3390798510 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 -- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 msgr2=0x55910b63ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 -- 172.21.15.119:0/3390798510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 msgr2=0x55910b63f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b63f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 -- 172.21.15.119:0/3390798510 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55910b2ac3f0 con 0x55910b634630 2024-03-28T18:53:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a69e00640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a6a800640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b641b00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0ee500 tx=0x7f4a5c005cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e65fdeeda6d0482 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.481+0000 7f4a4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0bd1e0 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4a5c0c1250 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a5400be20 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a5c0bcd70 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 10 client.27982 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55910b3767d0 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5400be20 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4a5c0ee670 con 0x55910b634630 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 10 client.27982 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 10 client.27982 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 20 client.27982 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6520b0 0x55910b6544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 4 client.27982 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.482+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55910b5a5800 con 0x55910b6520b0 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.483+0000 7f4a69e00640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6520b0 0x55910b6544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.483+0000 7f4a69e00640 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6520b0 0x55910b6544e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57ebe0d99e3fac27 server_cookie=8581f8e019f1342 in_seq=0 out_seq=0 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.483+0000 7f4a4fe00640 10 client.27982 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a4fe00640 1 -- 172.21.15.119:0/3390798510 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55910b5a5800 con 0x55910b6520b0 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a4fe00640 10 client.27982 handle_command_reply: tid=0 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 client.27982 shutdown 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 msgr2=0x55910b641b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b641b00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0ee500 tx=0x7f4a5c005cb0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 shutdown_connections 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6520b0 0x55910b6544e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b63f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b63ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 --2- 172.21.15.119:0/3390798510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b641b00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a4fe00640 0 client.27982 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.484+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 >> 172.21.15.119:0/3390798510 conn(0x55910b62ef80 msgr2=0x55910b630cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.485+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 shutdown_connections 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.485+0000 7f4a7418b740 1 -- 172.21.15.119:0/3390798510 wait complete. 2024-03-28T18:53:00.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.485+0000 7f4a7418b740 20 client.27982 trim_cache size 0 max 16384 2024-03-28T18:53:00.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 Processor -- start 2024-03-28T18:53:00.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 -- start start 2024-03-28T18:53:00.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b630ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 0x55910b639610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1aaf80 con 0x55910b634fe0 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1ac900 con 0x55910b6421f0 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a7418b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1ab5e0 con 0x55910b634630 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.498+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 0x55910b639610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57840/0 (socket says 172.21.15.119:57840) 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 -- 172.21.15.119:0/2963898159 learned_addr learned my addr 172.21.15.119:0/2963898159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6a800640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b630ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 -- 172.21.15.119:0/2963898159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 msgr2=0x55910b630ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b630ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 -- 172.21.15.119:0/2963898159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 msgr2=0x55910b639610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 0x55910b639610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 -- 172.21.15.119:0/2963898159 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55910b63f7a0 con 0x55910b6421f0 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6a800640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b630ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a69e00640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 0x55910b639610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a6b200640 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4a640099d0 tx=0x7f4a6401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=812626acdad3c361 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a5aa00640 1 -- 172.21.15.119:0/2963898159 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a640109b0 con 0x55910b6421f0 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.499+0000 7f4a5aa00640 1 -- 172.21.15.119:0/2963898159 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a64010b50 con 0x55910b6421f0 2024-03-28T18:53:00.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a5aa00640 1 -- 172.21.15.119:0/2963898159 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6401f660 con 0x55910b6421f0 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 -- 172.21.15.119:0/2963898159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 msgr2=0x55910b6415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4a640099d0 tx=0x7f4a6401b000 comp rx=0 tx=0).stop 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 -- 172.21.15.119:0/2963898159 shutdown_connections 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b6421f0 0x55910b6415c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b634fe0 0x55910b639610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 --2- 172.21.15.119:0/2963898159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b630ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 -- 172.21.15.119:0/2963898159 >> 172.21.15.119:0/2963898159 conn(0x55910b62ef80 msgr2=0x55910b6303d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 -- 172.21.15.119:0/2963898159 shutdown_connections 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.500+0000 7f4a7418b740 1 -- 172.21.15.119:0/2963898159 wait complete. 2024-03-28T18:53:00.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 -- start start 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 Processor -- start 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b64a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b6386f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1ac2a0 con 0x55910b6421f0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1abc40 con 0x55910b634fe0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c1a9160 con 0x55910b634630 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a7418b740 10 client.0 _ll_register_callbacks cb 0x7f4a5c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a6a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35670/0 (socket says 172.21.15.119:35670) 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a69e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b64a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a6a800640 1 -- 172.21.15.119:0/884921355 learned_addr learned my addr 172.21.15.119:0/884921355 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.501+0000 7f4a6b200640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b6386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 -- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 msgr2=0x55910b64a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b64a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 -- 172.21.15.119:0/884921355 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 msgr2=0x55910b6386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b6386f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 -- 172.21.15.119:0/884921355 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55910b638ef0 con 0x55910b634630 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a69e00640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b64a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a6a800640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0bd110 tx=0x7f4a5c003f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=698473984771e66a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0bcf50 con 0x55910b634630 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4a5c02cb80 con 0x55910b634630 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a64010b50 con 0x55910b634630 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.502+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a5c0c5720 con 0x55910b634630 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 10 client.27988 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55910b6b2b80 con 0x55910b634630 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a64010b50 con 0x55910b634630 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4a5c0c11e0 con 0x55910b634630 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 10 client.27988 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 10 client.27988 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 20 client.27988 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6b3070 0x55910b6b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 4 client.27988 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55910b59fb50 con 0x55910b6b3070 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.503+0000 7f4a69e00640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6b3070 0x55910b6b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a69e00640 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6b3070 0x55910b6b5490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ba182b4429235b61 server_cookie=8d120eb284bff358 in_seq=0 out_seq=0 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a4fe00640 10 client.27988 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a4fe00640 1 -- 172.21.15.119:0/884921355 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55910b59fb50 con 0x55910b6b3070 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a4fe00640 10 client.27988 handle_command_reply: tid=0 2024-03-28T18:53:00.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 client.27988 shutdown 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 msgr2=0x55910b649cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0bd110 tx=0x7f4a5c003f60 comp rx=0 tx=0).stop 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 shutdown_connections 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55910b6b3070 0x55910b6b5490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55910b6421f0 0x55910b6386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a4fe00640 0 client.27988 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55910b634fe0 0x55910b64a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 --2- 172.21.15.119:0/884921355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55910b634630 0x55910b649cd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 >> 172.21.15.119:0/884921355 conn(0x55910b62ef80 msgr2=0x55910b63a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 shutdown_connections 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 1 -- 172.21.15.119:0/884921355 wait complete. 2024-03-28T18:53:00.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.504+0000 7f4a7418b740 20 client.27988 trim_cache size 0 max 16384 2024-03-28T18:53:00.511 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 msgr2=0x7f4a500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 0x7f4a500a4e30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009fd0 tx=0x7f4a64009500 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 msgr2=0x7f4a6c197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c005ab0 tx=0x7f4a5c0042e0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 shutdown_connections 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4a500a2970 0x7f4a500a4e30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a6c1075e0 0x7f4a6c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c101ee0 0x7f4a6c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 --2- 172.21.15.119:0/431408691 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4a6c101520 0x7f4a6c197d70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 >> 172.21.15.119:0/431408691 conn(0x7f4a6c0fd010 msgr2=0x7f4a6c0fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 shutdown_connections 2024-03-28T18:53:00.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.505+0000 7f4a70a00640 1 -- 172.21.15.119:0/431408691 wait complete. 2024-03-28T18:53:00.521 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:00.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.732+0000 7f96df200640 1 -- 172.21.15.119:0/3956068400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e0108800 msgr2=0x7f96e010c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.732+0000 7f96df200640 1 --2- 172.21.15.119:0/3956068400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e0108800 0x7f96e010c7c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f96c8009900 tx=0x7f96c802bc40 comp rx=0 tx=0).stop 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 -- 172.21.15.119:0/3956068400 shutdown_connections 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 --2- 172.21.15.119:0/3956068400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e010cea0 0x7f96e0114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 --2- 172.21.15.119:0/3956068400 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e0108800 0x7f96e010c7c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 --2- 172.21.15.119:0/3956068400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e0107e50 0x7f96e0108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 -- 172.21.15.119:0/3956068400 >> 172.21.15.119:0/3956068400 conn(0x7f96e00774c0 msgr2=0x7f96e01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 -- 172.21.15.119:0/3956068400 shutdown_connections 2024-03-28T18:53:00.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.733+0000 7f96df200640 1 -- 172.21.15.119:0/3956068400 wait complete. 2024-03-28T18:53:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.734+0000 7f96df200640 1 Processor -- start 2024-03-28T18:53:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.734+0000 7f96df200640 1 -- start start 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e010cea0 0x7f96e019e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e01a2c70 0x7f96e01a3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e0117fa0 con 0x7f96e010cea0 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e0117e20 con 0x7f96e01a2c70 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96df200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e0118120 con 0x7f96e0107e50 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35690/0 (socket says 172.21.15.119:35690) 2024-03-28T18:53:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 -- 172.21.15.119:0/1542649895 learned_addr learned my addr 172.21.15.119:0/1542649895 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96de800640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e01a2c70 0x7f96e01a3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96d5400640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e010cea0 0x7f96e019e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 -- 172.21.15.119:0/1542649895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e01a2c70 msgr2=0x7f96e01a3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e01a2c70 0x7f96e01a3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 -- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e010cea0 msgr2=0x7f96e019e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e010cea0 0x7f96e019e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.735+0000 7f96dde00640 1 -- 172.21.15.119:0/1542649895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e01a3760 con 0x7f96e0107e50 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96dde00640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c920 tx=0x7f96d000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96d7400640 1 -- 172.21.15.119:0/1542649895 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0007c70 con 0x7f96e0107e50 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96d7400640 1 -- 172.21.15.119:0/1542649895 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96d0007e10 con 0x7f96e0107e50 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96d7400640 1 -- 172.21.15.119:0/1542649895 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0018890 con 0x7f96e0107e50 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e01a3a50 con 0x7f96e0107e50 2024-03-28T18:53:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.736+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e01ab1c0 con 0x7f96e0107e50 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.740+0000 7f96e79c6740 1 Processor -- start 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96d7400640 1 -- 172.21.15.119:0/1542649895 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f96d0018a70 con 0x7f96e0107e50 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 -- start start 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f321810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f324230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f27c2c0 con 0x55600f321de0 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f224c90 con 0x55600f319260 2024-03-28T18:53:00.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96e79c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f278360 con 0x55600f3188b0 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96dde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f321810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96de800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96de800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35702/0 (socket says 172.21.15.119:35702) 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96de800640 1 -- 172.21.15.119:0/2640457606 learned_addr learned my addr 172.21.15.119:0/2640457606 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.741+0000 7f96d5400640 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f324230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 -- 172.21.15.119:0/2640457606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 msgr2=0x55600f321810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f321810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 -- 172.21.15.119:0/2640457606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 msgr2=0x55600f324230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f324230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 -- 172.21.15.119:0/2640457606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55600f249080 con 0x55600f3188b0 2024-03-28T18:53:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96de800640 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f96d8009f40 tx=0x7f96d80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c09877bfb180971 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96cf000640 1 -- 172.21.15.119:0/2640457606 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8004d20 con 0x55600f3188b0 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96cf000640 1 -- 172.21.15.119:0/2640457606 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96d8004050 con 0x55600f3188b0 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96cf000640 1 -- 172.21.15.119:0/2640457606 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8005280 con 0x55600f3188b0 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.742+0000 7f96d7400640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 0x7f96c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96d7400640 1 -- 172.21.15.119:0/1542649895 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f96d00cf050 con 0x7f96e0107e50 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96dde00640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 0x7f96c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 -- 172.21.15.119:0/2640457606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 msgr2=0x55600f318c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f96d8009f40 tx=0x7f96d80118d0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 -- 172.21.15.119:0/2640457606 shutdown_connections 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f324230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f321810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2640457606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f318c90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 -- 172.21.15.119:0/2640457606 >> 172.21.15.119:0/2640457606 conn(0x55600f312f80 msgr2=0x55600f313360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 -- 172.21.15.119:0/2640457606 shutdown_connections 2024-03-28T18:53:00.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96e79c6740 1 -- 172.21.15.119:0/2640457606 wait complete. 2024-03-28T18:53:00.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.743+0000 7f96dde00640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 0x7f96c40a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96d00cc7c0 tx=0x7f96d00d2b90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 -- start start 2024-03-28T18:53:00.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 Processor -- start 2024-03-28T18:53:00.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f3216a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f31ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f324d50 con 0x55600f321de0 2024-03-28T18:53:00.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.744+0000 7f96e79c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f2380d0 con 0x55600f319260 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96e79c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55600f324ed0 con 0x55600f3188b0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96e79c6740 10 client.0 _ll_register_callbacks cb 0x55600f23ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96de800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f3216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57890/0 (socket says 172.21.15.119:57890) 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 -- 172.21.15.119:0/1840694426 learned_addr learned my addr 172.21.15.119:0/1840694426 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96de800640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f31ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 -- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 msgr2=0x55600f3216a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f3216a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 -- 172.21.15.119:0/1840694426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 msgr2=0x55600f31ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f31ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 -- 172.21.15.119:0/1840694426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55600ef903f0 con 0x55600f319260 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.745+0000 7f96d5400640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f96c8005ab0 tx=0x7f96c8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f91e40947e7ace server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96c3400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:00.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c802cbd0 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f96c8005030 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96d8004050 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c8005230 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96e79c6740 10 client.28025 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55600f05a7d0 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.746+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96d8004050 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f96c804d030 con 0x55600f319260 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 10 client.28025 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 10 client.28025 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 20 client.28025 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f331c20 0x55600f334010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 4 client.28025 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55600f289800 con 0x55600f331c20 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96de800640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f331c20 0x55600f334010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96de800640 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f331c20 0x55600f334010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b364399824743323 server_cookie=7fc945b36edacbb5 in_seq=0 out_seq=0 2024-03-28T18:53:00.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.747+0000 7f96c3400640 10 client.28025 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96c3400640 1 -- 172.21.15.119:0/1840694426 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55600f289800 con 0x55600f331c20 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96c3400640 10 client.28025 handle_command_reply: tid=0 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 client.28025 shutdown 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 msgr2=0x55600f31ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f96c8005ab0 tx=0x7f96c8004610 comp rx=0 tx=0).stop 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 shutdown_connections 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f331c20 0x55600f334010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f31ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f31ea30 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96c3400640 0 client.28025 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 --2- 172.21.15.119:0/1840694426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f3216a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.748+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 >> 172.21.15.119:0/1840694426 conn(0x55600f312f80 msgr2=0x55600f314cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.749+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 shutdown_connections 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.749+0000 7f96e79c6740 1 -- 172.21.15.119:0/1840694426 wait complete. 2024-03-28T18:53:00.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.749+0000 7f96e79c6740 20 client.28025 trim_cache size 0 max 16384 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 Processor -- start 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 -- start start 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f321de0 0x55600f31bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b37e0 con 0x55600f319260 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b5160 con 0x55600f3188b0 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.762+0000 7f96e79c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b3e40 con 0x55600f321de0 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96de800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48388/0 (socket says 172.21.15.119:48388) 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96de800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57902/0 (socket says 172.21.15.119:57902) 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 -- 172.21.15.119:0/2763166814 learned_addr learned my addr 172.21.15.119:0/2763166814 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96de800640 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f321de0 0x55600f31bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 -- 172.21.15.119:0/2763166814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f321de0 msgr2=0x55600f31bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f321de0 0x55600f31bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 -- 172.21.15.119:0/2763166814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 msgr2=0x55600f321160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96d5400640 1 -- 172.21.15.119:0/2763166814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55600f319000 con 0x55600f319260 2024-03-28T18:53:00.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.763+0000 7f96de800640 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.764+0000 7f96d5400640 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f96c8009850 tx=0x7f96c8003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bdccb6d09f906d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.764+0000 7f96ce600640 1 -- 172.21.15.119:0/2763166814 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8005d50 con 0x55600f319260 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.764+0000 7f96ce600640 1 -- 172.21.15.119:0/2763166814 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96c80043e0 con 0x55600f319260 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.764+0000 7f96ce600640 1 -- 172.21.15.119:0/2763166814 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8003e20 con 0x55600f319260 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.764+0000 7f96e79c6740 1 -- 172.21.15.119:0/2763166814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 msgr2=0x55600f3216a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f96c8009850 tx=0x7f96c8003690 comp rx=0 tx=0).stop 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 -- 172.21.15.119:0/2763166814 shutdown_connections 2024-03-28T18:53:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f321de0 0x55600f31bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f319260 0x55600f3216a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 --2- 172.21.15.119:0/2763166814 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f3188b0 0x55600f321160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 -- 172.21.15.119:0/2763166814 >> 172.21.15.119:0/2763166814 conn(0x55600f312f80 msgr2=0x55600f312160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 -- 172.21.15.119:0/2763166814 shutdown_connections 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 -- 172.21.15.119:0/2763166814 wait complete. 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 -- start start 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.765+0000 7f96e79c6740 1 Processor -- start 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f329f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f326890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b4b00 con 0x55600f321de0 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b44a0 con 0x55600f319260 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e01b19c0 con 0x55600f3188b0 2024-03-28T18:53:00.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96e79c6740 10 client.0 _ll_register_callbacks cb 0x7f96c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96de800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f329f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96d5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57918/0 (socket says 172.21.15.119:57918) 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96d5400640 1 -- 172.21.15.119:0/589257850 learned_addr learned my addr 172.21.15.119:0/589257850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96de800640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f326890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96d5400640 1 -- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 msgr2=0x55600f329f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.766+0000 7f96d5400640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f329f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96d5400640 1 -- 172.21.15.119:0/589257850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 msgr2=0x55600f326890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96d5400640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f326890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96d5400640 1 -- 172.21.15.119:0/589257850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55600f3270c0 con 0x55600f319260 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96d5400640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f96c8004f10 tx=0x7f96c80051c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5913c17cdbf74f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96c3400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c805c070 con 0x55600f319260 2024-03-28T18:53:00.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55600f283b50 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f96c802cbd0 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c8005d50 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96e79c6740 10 client.28031 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.767+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55600f38e4b0 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55600f283b50 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f96c8056020 con 0x55600f319260 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 10 client.28031 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 10 client.28031 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 20 client.28031 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f38e940 0x55600f390d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 4 client.28031 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55600f201500 con 0x55600f38e940 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.768+0000 7f96de800640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f38e940 0x55600f390d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96de800640 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f38e940 0x55600f390d60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8264a734b2aa83f8 server_cookie=821ab1dbc4bd0cf2 in_seq=0 out_seq=0 2024-03-28T18:53:00.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96c3400640 10 client.28031 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:00.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96c3400640 1 -- 172.21.15.119:0/589257850 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55600f201500 con 0x55600f38e940 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96c3400640 10 client.28031 handle_command_reply: tid=0 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 client.28031 shutdown 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 msgr2=0x55600f326310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f96c8004f10 tx=0x7f96c80051c0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 shutdown_connections 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55600f38e940 0x55600f390d60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55600f321de0 0x55600f326890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55600f319260 0x55600f326310 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 --2- 172.21.15.119:0/589257850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55600f3188b0 0x55600f329f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 >> 172.21.15.119:0/589257850 conn(0x55600f312f80 msgr2=0x55600f319640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96c3400640 0 client.28031 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 shutdown_connections 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 1 -- 172.21.15.119:0/589257850 wait complete. 2024-03-28T18:53:00.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.769+0000 7f96e79c6740 20 client.28031 trim_cache size 0 max 16384 2024-03-28T18:53:00.774 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:00.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 msgr2=0x7f96c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 0x7f96c40a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f96d00cc7c0 tx=0x7f96d00d2b90 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 msgr2=0x7f96e019e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c920 tx=0x7f96d000cde0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 shutdown_connections 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f96c40a2970 0x7f96c40a4e30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f96e01a2c70 0x7f96e01a3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e010cea0 0x7f96e019e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 --2- 172.21.15.119:0/1542649895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96e0107e50 0x7f96e019e400 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 >> 172.21.15.119:0/1542649895 conn(0x7f96e00774c0 msgr2=0x7f96e0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 shutdown_connections 2024-03-28T18:53:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.770+0000 7f96df200640 1 -- 172.21.15.119:0/1542649895 wait complete. 2024-03-28T18:53:00.784 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:00 smithi119 ceph-mon[31813]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:00.784 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:00 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3072626049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:00.786 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:00.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:53:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:00 smithi196 ceph-mon[36300]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:00 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3072626049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:00 smithi131 ceph-mon[36065]: pgmap v275: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:00 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3072626049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 -- 172.21.15.119:0/3554156194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b01124e0 msgr2=0x7fc6b01148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 --2- 172.21.15.119:0/3554156194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b01124e0 0x7fc6b01148d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009cd0 tx=0x7fc6a802c0f0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 -- 172.21.15.119:0/3554156194 shutdown_connections 2024-03-28T18:53:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 --2- 172.21.15.119:0/3554156194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b01124e0 0x7fc6b01148d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 --2- 172.21.15.119:0/3554156194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b00764f0 0x7fc6b0111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 --2- 172.21.15.119:0/3554156194 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b0075bd0 0x7fc6b0075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 -- 172.21.15.119:0/3554156194 >> 172.21.15.119:0/3554156194 conn(0x7fc6b00ff2f0 msgr2=0x7fc6b0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:00.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 -- 172.21.15.119:0/3554156194 shutdown_connections 2024-03-28T18:53:00.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.987+0000 7fc6b7200640 1 -- 172.21.15.119:0/3554156194 wait complete. 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.988+0000 7fc6b7200640 1 Processor -- start 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.988+0000 7fc6b7200640 1 -- start start 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b0075bd0 0x7fc6b019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 0x7fc6b01a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b0118040 con 0x7fc6b0075bd0 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b0117ec0 con 0x7fc6b01124e0 2024-03-28T18:53:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01181c0 con 0x7fc6b00764f0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35748/0 (socket says 172.21.15.119:35748) 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 -- 172.21.15.119:0/164114425 learned_addr learned my addr 172.21.15.119:0/164114425 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b5200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 0x7fc6b01a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6afe00640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b0075bd0 0x7fc6b019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 -- 172.21.15.119:0/164114425 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 msgr2=0x7fc6b01a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 0x7fc6b01a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 -- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b0075bd0 msgr2=0x7fc6b019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b0075bd0 0x7fc6b019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6a7e00640 1 -- 172.21.15.119:0/164114425 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b01a3550 con 0x7fc6b00764f0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.989+0000 7fc6b5200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 0x7fc6b01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc69800e9f0 tx=0x7fc69800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6ad600640 1 -- 172.21.15.119:0/164114425 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69800cd80 con 0x7fc6b00764f0 2024-03-28T18:53:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6ad600640 1 -- 172.21.15.119:0/164114425 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc698010910 con 0x7fc6b00764f0 2024-03-28T18:53:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6ad600640 1 -- 172.21.15.119:0/164114425 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698018990 con 0x7fc6b00764f0 2024-03-28T18:53:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b01a3840 con 0x7fc6b00764f0 2024-03-28T18:53:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.990+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b01ab0d0 con 0x7fc6b00764f0 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.993+0000 7fc6ba024740 1 Processor -- start 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.993+0000 7fc6ad600640 1 -- 172.21.15.119:0/164114425 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc698010ac0 con 0x7fc6b00764f0 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.993+0000 7fc6ba024740 1 -- start start 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 0x56056150aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561516050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56056146e2c0 con 0x56056150a6d0 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x560561416c90 con 0x56056150b080 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6ba024740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x56056146a360 con 0x560561513c00 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57946/0 (socket says 172.21.15.119:57946) 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 -- 172.21.15.119:0/1846349861 learned_addr learned my addr 172.21.15.119:0/1846349861 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561516050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6b5200640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 0x56056150aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 -- 172.21.15.119:0/1846349861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 msgr2=0x560561516050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561516050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 -- 172.21.15.119:0/1846349861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 msgr2=0x56056150aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 0x56056150aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6afe00640 1 -- 172.21.15.119:0/1846349861 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56056143b080 con 0x56056150b080 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.994+0000 7fc6b5200640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 0x56056150aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:00.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009d20 tx=0x7fc6a0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1be1d2ba1e1cb7aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6ad600640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 0x7fc6900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6a5600640 1 -- 172.21.15.119:0/1846349861 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a001e070 con 0x56056150b080 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6a5600640 1 -- 172.21.15.119:0/1846349861 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0019b10 con 0x56056150b080 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6a5600640 1 -- 172.21.15.119:0/1846349861 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a000eb90 con 0x56056150b080 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6ad600640 1 -- 172.21.15.119:0/164114425 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc6980cf100 con 0x7fc6b00764f0 2024-03-28T18:53:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.995+0000 7fc6b5200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 0x7fc6900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- 172.21.15.119:0/1846349861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 msgr2=0x560561513630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009d20 tx=0x7fc6a0011810 comp rx=0 tx=0).stop 2024-03-28T18:53:01.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- 172.21.15.119:0/1846349861 shutdown_connections 2024-03-28T18:53:01.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561516050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150b080 0x560561513630 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1846349861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150a6d0 0x56056150aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- 172.21.15.119:0/1846349861 >> 172.21.15.119:0/1846349861 conn(0x560561504da0 msgr2=0x560561505180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- 172.21.15.119:0/1846349861 shutdown_connections 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- 172.21.15.119:0/1846349861 wait complete. 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6b5200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 0x7fc6900a4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009bf0 tx=0x7fc6a802e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.996+0000 7fc6ba024740 1 -- start start 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 Processor -- start 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x5605615108a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561510e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560561516b70 con 0x56056150b080 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x56056142a0d0 con 0x560561513c00 2024-03-28T18:53:01.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x560561516cf0 con 0x56056150a6d0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6ba024740 10 client.0 _ll_register_callbacks cb 0x56056142ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35760/0 (socket says 172.21.15.119:35760) 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6afe00640 1 -- 172.21.15.119:0/2350311700 learned_addr learned my addr 172.21.15.119:0/2350311700 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6b5200640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561510e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.997+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x5605615108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 -- 172.21.15.119:0/2350311700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 msgr2=0x560561510e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561510e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 -- 172.21.15.119:0/2350311700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 msgr2=0x5605615108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x5605615108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 -- 172.21.15.119:0/2350311700 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605611823f0 con 0x56056150a6d0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x5605615108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc6a001c040 tx=0x7fc6a0019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c05cc46893559583 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc69e000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:01.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0027020 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc6a000ead0 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6a0019b10 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.998+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a0032050 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 10 client.28018 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56056124c7d0 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6a0019b10 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc6a00046c0 con 0x56056150a6d0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 10 client.28018 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 10 client.28018 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 20 client.28018 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5605615239c0 0x560561525db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 4 client.28018 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56056147b800 con 0x5605615239c0 2024-03-28T18:53:01.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:00.999+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5605615239c0 0x560561525db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.000+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5605615239c0 0x560561525db0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7475393da666281 server_cookie=4f1d1af843380b6f in_seq=0 out_seq=0 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.000+0000 7fc69e000640 10 client.28018 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.000+0000 7fc69e000640 1 -- 172.21.15.119:0/2350311700 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56056147b800 con 0x5605615239c0 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.000+0000 7fc69e000640 10 client.28018 handle_command_reply: tid=0 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 client.28018 shutdown 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 msgr2=0x560561513510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc6a001c040 tx=0x7fc6a0019660 comp rx=0 tx=0).stop 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 shutdown_connections 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x5605615239c0 0x560561525db0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561510e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x5605615108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2350311700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x560561513510 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc69e000640 0 client.28018 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 >> 172.21.15.119:0/2350311700 conn(0x560561504da0 msgr2=0x560561506af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 shutdown_connections 2024-03-28T18:53:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 1 -- 172.21.15.119:0/2350311700 wait complete. 2024-03-28T18:53:01.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.001+0000 7fc6ba024740 20 client.28018 trim_cache size 0 max 16384 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 Processor -- start 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 -- start start 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561512fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561513510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b3aa0 con 0x56056150b080 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b5420 con 0x560561513c00 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6ba024740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b4100 con 0x56056150a6d0 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6b5200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561513510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561512fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35770/0 (socket says 172.21.15.119:35770) 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.015+0000 7fc6afe00640 1 -- 172.21.15.119:0/2727080082 learned_addr learned my addr 172.21.15.119:0/2727080082 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 -- 172.21.15.119:0/2727080082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 msgr2=0x560561513510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561513510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 -- 172.21.15.119:0/2727080082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 msgr2=0x560561512fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561512fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 -- 172.21.15.119:0/2727080082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605615067f0 con 0x56056150a6d0 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6afe00640 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011850 tx=0x7fc6a000e680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=466e4ca5cd1c272e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6a4c00640 1 -- 172.21.15.119:0/2727080082 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0027070 con 0x56056150a6d0 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6a4c00640 1 -- 172.21.15.119:0/2727080082 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6a001f040 con 0x56056150a6d0 2024-03-28T18:53:01.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.016+0000 7fc6a4c00640 1 -- 172.21.15.119:0/2727080082 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a001e070 con 0x56056150a6d0 2024-03-28T18:53:01.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- 172.21.15.119:0/2727080082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 msgr2=0x56056150fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011850 tx=0x7fc6a000e680 comp rx=0 tx=0).stop 2024-03-28T18:53:01.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- 172.21.15.119:0/2727080082 shutdown_connections 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560561513c00 0x560561513510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561512fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 --2- 172.21.15.119:0/2727080082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56056150a6d0 0x56056150fa00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- 172.21.15.119:0/2727080082 >> 172.21.15.119:0/2727080082 conn(0x560561504da0 msgr2=0x560561505180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- 172.21.15.119:0/2727080082 shutdown_connections 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- 172.21.15.119:0/2727080082 wait complete. 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 -- start start 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.017+0000 7fc6ba024740 1 Processor -- start 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561519a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561519f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b4dc0 con 0x56056150b080 2024-03-28T18:53:01.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b4760 con 0x56056150a6d0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b01b1c80 con 0x560561513c00 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6ba024740 10 client.0 _ll_register_callbacks cb 0x7fc6a001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:57970/0 (socket says 172.21.15.119:57970) 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 -- 172.21.15.119:0/1502399088 learned_addr learned my addr 172.21.15.119:0/1502399088 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561519a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6b5200640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561519f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 -- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 msgr2=0x560561519f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561519f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 -- 172.21.15.119:0/1502399088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 msgr2=0x560561519a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561519a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6afe00640 1 -- 172.21.15.119:0/1502399088 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605615111b0 con 0x56056150a6d0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.018+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561519a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc6afe00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011ef0 tx=0x7fc6a0019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0cb6c271328f0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc69e000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:01.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a001e040 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6a001f040 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc6a0019c60 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a0046050 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc6ba024740 10 client.28049 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56056157f310 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.019+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6a001f040 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc6a0005200 con 0x56056150a6d0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 10 client.28049 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 10 client.28049 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 20 client.28049 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x56056157f8c0 0x560561581cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 4 client.28049 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560561475b50 con 0x56056157f8c0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x56056157f8c0 0x560561581cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc6a7e00640 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x56056157f8c0 0x560561581cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aec530496cb85235 server_cookie=22493096dd05b578 in_seq=0 out_seq=0 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.020+0000 7fc69e000640 10 client.28049 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc69e000640 1 -- 172.21.15.119:0/1502399088 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560561475b50 con 0x56056157f8c0 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc69e000640 10 client.28049 handle_command_reply: tid=0 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 client.28049 shutdown 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 msgr2=0x5605615194c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0011ef0 tx=0x7fc6a0019640 comp rx=0 tx=0).stop 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 shutdown_connections 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x56056157f8c0 0x560561581cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560561513c00 0x560561519f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56056150b080 0x560561519a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 --2- 172.21.15.119:0/1502399088 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x56056150a6d0 0x5605615194c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc69e000640 0 client.28049 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 >> 172.21.15.119:0/1502399088 conn(0x560561504da0 msgr2=0x560561503eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 shutdown_connections 2024-03-28T18:53:01.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 1 -- 172.21.15.119:0/1502399088 wait complete. 2024-03-28T18:53:01.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.021+0000 7fc6ba024740 20 client.28049 trim_cache size 0 max 16384 2024-03-28T18:53:01.026 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 msgr2=0x7fc6900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 0x7fc6900a4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009bf0 tx=0x7fc6a802e040 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 msgr2=0x7fc6b019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc69800e9f0 tx=0x7fc69800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 shutdown_connections 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc6900a29f0 0x7fc6900a4eb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc6b01124e0 0x7fc6b01a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6b00764f0 0x7fc6b019eaa0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 --2- 172.21.15.119:0/164114425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b0075bd0 0x7fc6b019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 >> 172.21.15.119:0/164114425 conn(0x7fc6b00ff2f0 msgr2=0x7fc6b01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 shutdown_connections 2024-03-28T18:53:01.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.022+0000 7fc6b7200640 1 -- 172.21.15.119:0/164114425 wait complete. 2024-03-28T18:53:01.040 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:01.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.255+0000 7f3611400640 1 -- 172.21.15.119:0/1159308660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 msgr2=0x7f360c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.255+0000 7f3611400640 1 --2- 172.21.15.119:0/1159308660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c109c30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009ca0 tx=0x7f35fc02c100 comp rx=0 tx=0).stop 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 -- 172.21.15.119:0/1159308660 shutdown_connections 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 --2- 172.21.15.119:0/1159308660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 0x7f360c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 --2- 172.21.15.119:0/1159308660 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 --2- 172.21.15.119:0/1159308660 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c109c30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 -- 172.21.15.119:0/1159308660 >> 172.21.15.119:0/1159308660 conn(0x7f360c0ff350 msgr2=0x7f360c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 -- 172.21.15.119:0/1159308660 shutdown_connections 2024-03-28T18:53:01.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.256+0000 7f3611400640 1 -- 172.21.15.119:0/1159308660 wait complete. 2024-03-28T18:53:01.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 Processor -- start 2024-03-28T18:53:01.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 -- start start 2024-03-28T18:53:01.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 0x7f360c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c19ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c113c60 con 0x7f360c104200 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c113ae0 con 0x7f360c109850 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3611400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c113de0 con 0x7f360c103840 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f360a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f360a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35798/0 (socket says 172.21.15.119:35798) 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f360a800640 1 -- 172.21.15.119:0/2734938494 learned_addr learned my addr 172.21.15.119:0/2734938494 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f3609e00640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 0x7f360c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.257+0000 7f360b200640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c19ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 -- 172.21.15.119:0/2734938494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 msgr2=0x7f360c19ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c19ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 -- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 msgr2=0x7f360c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 0x7f360c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 -- 172.21.15.119:0/2734938494 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c19f160 con 0x7f360c103840 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f360a800640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f35fc005e30 tx=0x7f35fc0346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f3603400640 1 -- 172.21.15.119:0/2734938494 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc039050 con 0x7f360c103840 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c19f3f0 con 0x7f360c103840 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f3603400640 1 -- 172.21.15.119:0/2734938494 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35fc045070 con 0x7f360c103840 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f3603400640 1 -- 172.21.15.119:0/2734938494 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004630 con 0x7f360c103840 2024-03-28T18:53:01.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.258+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c1a6ce0 con 0x7f360c103840 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.261+0000 7f36141e6740 1 Processor -- start 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 -- start start 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f3603400640 1 -- 172.21.15.119:0/2734938494 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f35fc003970 con 0x7f360c103840 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a530630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 0x560b9a533050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a48b2c0 con 0x560b9a530c00 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a433c90 con 0x560b9a528080 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f36141e6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a487360 con 0x560b9a5276d0 2024-03-28T18:53:01.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f360b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f360b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35810/0 (socket says 172.21.15.119:35810) 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f360b200640 1 -- 172.21.15.119:0/183165234 learned_addr learned my addr 172.21.15.119:0/183165234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f360a800640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a530630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.262+0000 7f3609e00640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 0x560b9a533050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 -- 172.21.15.119:0/183165234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 msgr2=0x560b9a530630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a530630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 -- 172.21.15.119:0/183165234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 msgr2=0x560b9a533050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 0x560b9a533050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 -- 172.21.15.119:0/183165234 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b9a458080 con 0x560b9a5276d0 2024-03-28T18:53:01.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f3609e00640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 0x560b9a533050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360b200640 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f360400b6b0 tx=0x7f3604004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd2351be3f16272 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f3603400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f35fb400640 1 -- 172.21.15.119:0/183165234 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604003b20 con 0x560b9a5276d0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f35fb400640 1 -- 172.21.15.119:0/183165234 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3604015070 con 0x560b9a5276d0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f35fb400640 1 -- 172.21.15.119:0/183165234 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36040116a0 con 0x560b9a5276d0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f3603400640 1 -- 172.21.15.119:0/2734938494 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f35fc0f14c0 con 0x7f360c103840 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.263+0000 7f360a800640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 -- 172.21.15.119:0/183165234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 msgr2=0x560b9a527ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f360400b6b0 tx=0x7f3604004610 comp rx=0 tx=0).stop 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f360a800640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 0x7f35f00a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x560b9a5271a0 tx=0x7f35fc0f4f70 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 -- 172.21.15.119:0/183165234 shutdown_connections 2024-03-28T18:53:01.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a530c00 0x560b9a533050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a530630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 --2- 172.21.15.119:0/183165234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a527ab0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 -- 172.21.15.119:0/183165234 >> 172.21.15.119:0/183165234 conn(0x560b9a521da0 msgr2=0x560b9a522180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 -- 172.21.15.119:0/183165234 shutdown_connections 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.264+0000 7f36141e6740 1 -- 172.21.15.119:0/183165234 wait complete. 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 -- start start 2024-03-28T18:53:01.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 Processor -- start 2024-03-28T18:53:01.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a530560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a533b70 con 0x560b9a5276d0 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a4470d0 con 0x560b9a530c00 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b9a533cf0 con 0x560b9a528080 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.265+0000 7f36141e6740 10 client.0 _ll_register_callbacks cb 0x560b9a44be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f360b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a530560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35812/0 (socket says 172.21.15.119:35812) 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 -- 172.21.15.119:0/2352451168 learned_addr learned my addr 172.21.15.119:0/2352451168 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f360b200640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 -- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 msgr2=0x560b9a52df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 -- 172.21.15.119:0/2352451168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 msgr2=0x560b9a530560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a530560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 -- 172.21.15.119:0/2352451168 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b9a19f3f0 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f3609e00640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35f4012700 tx=0x7f35f4012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c475f5db26f09d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.266+0000 7f35efe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4004020 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f35f40041c0 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3604015070 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35f4004400 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 10 client.28036 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560b9a2697d0 con 0x560b9a528080 2024-03-28T18:53:01.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3604015070 con 0x560b9a528080 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f35f4003bc0 con 0x560b9a528080 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 10 client.28036 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 10 client.28036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 20 client.28036 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a540b20 0x560b9a542f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 4 client.28036 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.267+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b9a498800 con 0x560b9a540b20 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.268+0000 7f360b200640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a540b20 0x560b9a542f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.268+0000 7f360b200640 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a540b20 0x560b9a542f10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=129abbdd0edcb99e server_cookie=d6bb4e026b1340f9 in_seq=0 out_seq=0 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.268+0000 7f35efe00640 10 client.28036 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f35efe00640 1 -- 172.21.15.119:0/2352451168 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560b9a498800 con 0x560b9a540b20 2024-03-28T18:53:01.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f35efe00640 10 client.28036 handle_command_reply: tid=0 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 client.28036 shutdown 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 msgr2=0x560b9a52d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35f4012700 tx=0x7f35f4012bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 shutdown_connections 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a540b20 0x560b9a542f10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a528080 0x560b9a52d8f0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f35efe00640 0 client.28036 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 --2- 172.21.15.119:0/2352451168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a530560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 >> 172.21.15.119:0/2352451168 conn(0x560b9a521da0 msgr2=0x560b9a523af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.269+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 shutdown_connections 2024-03-28T18:53:01.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.270+0000 7f36141e6740 1 -- 172.21.15.119:0/2352451168 wait complete. 2024-03-28T18:53:01.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.270+0000 7f36141e6740 20 client.28036 trim_cache size 0 max 16384 2024-03-28T18:53:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 Processor -- start 2024-03-28T18:53:01.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 -- start start 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a530020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1af620 con 0x560b9a528080 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1b0fa0 con 0x560b9a530c00 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.283+0000 7f36141e6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1afc80 con 0x560b9a5276d0 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f360b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a530020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48480/0 (socket says 172.21.15.119:48480) 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 -- 172.21.15.119:0/3029331510 learned_addr learned my addr 172.21.15.119:0/3029331510 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f360b200640 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 -- 172.21.15.119:0/3029331510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 msgr2=0x560b9a530020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a530020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 -- 172.21.15.119:0/3029331510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 msgr2=0x560b9a52ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 -- 172.21.15.119:0/3029331510 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b9a533a50 con 0x560b9a528080 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.284+0000 7f3609e00640 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009d30 tx=0x7f35f4009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37611b5d33137da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f35faa00640 1 -- 172.21.15.119:0/3029331510 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f401c600 con 0x560b9a528080 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f35faa00640 1 -- 172.21.15.119:0/3029331510 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35f400fe80 con 0x560b9a528080 2024-03-28T18:53:01.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f35faa00640 1 -- 172.21.15.119:0/3029331510 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f401ccd0 con 0x560b9a528080 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 -- 172.21.15.119:0/3029331510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 msgr2=0x560b9a530560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009d30 tx=0x7f35f4009620 comp rx=0 tx=0).stop 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 -- 172.21.15.119:0/3029331510 shutdown_connections 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a530c00 0x560b9a52ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a528080 0x560b9a530560 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 --2- 172.21.15.119:0/3029331510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a5276d0 0x560b9a530020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 -- 172.21.15.119:0/3029331510 >> 172.21.15.119:0/3029331510 conn(0x560b9a521da0 msgr2=0x560b9a520eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 -- 172.21.15.119:0/3029331510 shutdown_connections 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.285+0000 7f36141e6740 1 -- 172.21.15.119:0/3029331510 wait complete. 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 -- start start 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 Processor -- start 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a5389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 0x560b9a537280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1b0940 con 0x560b9a5276d0 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1b02e0 con 0x560b9a528080 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1ad800 con 0x560b9a530c00 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.286+0000 7f36141e6740 10 client.0 _ll_register_callbacks cb 0x7f35f4009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f360b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a5389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58024/0 (socket says 172.21.15.119:58024) 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 -- 172.21.15.119:0/2872169135 learned_addr learned my addr 172.21.15.119:0/2872169135 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f360b200640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 0x560b9a537280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 -- 172.21.15.119:0/2872169135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 msgr2=0x560b9a537280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 0x560b9a537280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 -- 172.21.15.119:0/2872169135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 msgr2=0x560b9a5389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a5389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 -- 172.21.15.119:0/2872169135 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b9a5377c0 con 0x560b9a528080 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f360b200640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 0x560b9a537280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f3609e00640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f35f40058c0 tx=0x7f35f401c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7050b0f1c3ae0d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.287+0000 7f35efe00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:01.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f400fbb0 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f35f401c950 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b9a492b50 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35f403a050 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 10 client.28067 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560b9a59d0f0 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560b9a492b50 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f35f402a070 con 0x560b9a528080 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 10 client.28067 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 10 client.28067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 20 client.28067 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a59d700 0x560b9a59fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:01.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 4 client.28067 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.288+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560b9a410500 con 0x560b9a59d700 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f360b200640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a59d700 0x560b9a59fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f35efe00640 10 client.28067 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f360b200640 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a59d700 0x560b9a59fb20 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4366fe6baff7d94f server_cookie=96cf4a9eab1d5533 in_seq=0 out_seq=0 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f35efe00640 1 -- 172.21.15.119:0/2872169135 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560b9a410500 con 0x560b9a59d700 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f35efe00640 10 client.28067 handle_command_reply: tid=0 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 client.28067 shutdown 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 msgr2=0x560b9a536d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f35f40058c0 tx=0x7f35f401c7e0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 shutdown_connections 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x560b9a59d700 0x560b9a59fb20 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560b9a530c00 0x560b9a537280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x560b9a528080 0x560b9a536d40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.289+0000 7f36141e6740 1 --2- 172.21.15.119:0/2872169135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560b9a5276d0 0x560b9a5389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 >> 172.21.15.119:0/2872169135 conn(0x560b9a521da0 msgr2=0x560b9a5230a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f35efe00640 0 client.28067 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 shutdown_connections 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f36141e6740 1 -- 172.21.15.119:0/2872169135 wait complete. 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f36141e6740 20 client.28067 trim_cache size 0 max 16384 2024-03-28T18:53:01.296 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 msgr2=0x7f35f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 0x7f35f00a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x560b9a5271a0 tx=0x7f35fc0f4f70 comp rx=0 tx=0).stop 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 msgr2=0x7f360c19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f35fc005e30 tx=0x7f35fc0346f0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 shutdown_connections 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f360c109850 0x7f360c19ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f360c104200 0x7f360c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 --2- 172.21.15.119:0/2734938494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f360c103840 0x7f360c19a1c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:01.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.290+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 >> 172.21.15.119:0/2734938494 conn(0x7f360c0ff350 msgr2=0x7f360c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:01.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.291+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 shutdown_connections 2024-03-28T18:53:01.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:01.291+0000 7f3611400640 1 -- 172.21.15.119:0/2734938494 wait complete. 2024-03-28T18:53:02.309 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:53:02.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:53:02.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:53:02.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 -- 172.21.15.119:0/3335359291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 msgr2=0x7f78f0103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 --2- 172.21.15.119:0/3335359291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f0103cc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009890 tx=0x7f78d802bb70 comp rx=0 tx=0).stop 2024-03-28T18:53:02.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 -- 172.21.15.119:0/3335359291 shutdown_connections 2024-03-28T18:53:02.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 --2- 172.21.15.119:0/3335359291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0104200 0x7f78f0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 --2- 172.21.15.119:0/3335359291 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f0103cc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 --2- 172.21.15.119:0/3335359291 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0109850 0x7f78f0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.533+0000 7f78f5e00640 1 -- 172.21.15.119:0/3335359291 >> 172.21.15.119:0/3335359291 conn(0x7f78f00ff2f0 msgr2=0x7f78f0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.534+0000 7f78f5e00640 1 -- 172.21.15.119:0/3335359291 shutdown_connections 2024-03-28T18:53:02.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.534+0000 7f78f5e00640 1 -- 172.21.15.119:0/3335359291 wait complete. 2024-03-28T18:53:02.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.535+0000 7f78f5e00640 1 Processor -- start 2024-03-28T18:53:02.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.535+0000 7f78f5e00640 1 -- start start 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.535+0000 7f78f5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78f5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0104200 0x7f78f019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78f5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0109850 0x7f78f019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78f5e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0113c60 con 0x7f78f0109850 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78f5e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0113ae0 con 0x7f78f0104200 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78f5e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0113de0 con 0x7f78f0103840 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35852/0 (socket says 172.21.15.119:35852) 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 -- 172.21.15.119:0/1433252090 learned_addr learned my addr 172.21.15.119:0/1433252090 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78ee000640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0104200 0x7f78f019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78ef400640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0109850 0x7f78f019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 -- 172.21.15.119:0/1433252090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0104200 msgr2=0x7f78f019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0104200 0x7f78f019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 -- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0109850 msgr2=0x7f78f019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0109850 0x7f78f019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.536+0000 7f78eea00640 1 -- 172.21.15.119:0/1433252090 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f019f1d0 con 0x7f78f0103840 2024-03-28T18:53:02.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.537+0000 7f78eea00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f78e000e9f0 tx=0x7f78e000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.538+0000 7f78f4a00640 1 -- 172.21.15.119:0/1433252090 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e000ccc0 con 0x7f78f0103840 2024-03-28T18:53:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.538+0000 7f78f4a00640 1 -- 172.21.15.119:0/1433252090 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e0005500 con 0x7f78f0103840 2024-03-28T18:53:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.538+0000 7f78f4a00640 1 -- 172.21.15.119:0/1433252090 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0010690 con 0x7f78f0103840 2024-03-28T18:53:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.538+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f019f4c0 con 0x7f78f0103840 2024-03-28T18:53:02.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.539+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f01a6d50 con 0x7f78f0103840 2024-03-28T18:53:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.542+0000 7f78f4a00640 1 -- 172.21.15.119:0/1433252090 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f78e0004f20 con 0x7f78f0103840 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.542+0000 7f78f89ff740 1 Processor -- start 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 -- start start 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc91a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc923040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc925970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc87eeb0 con 0x5566fc923580 2024-03-28T18:53:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc827a60 con 0x5566fc91a080 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78f89ff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc87af50 con 0x5566fc91a9a0 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78eea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc923040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc91a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78ee000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc925970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc91a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58054/0 (socket says 172.21.15.119:58054) 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.543+0000 7f78ef400640 1 -- 172.21.15.119:0/2852738987 learned_addr learned my addr 172.21.15.119:0/2852738987 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 -- 172.21.15.119:0/2852738987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 msgr2=0x5566fc91a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc91a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 -- 172.21.15.119:0/2852738987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 msgr2=0x5566fc925970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc925970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78f4a00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 0x7f78d40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 -- 172.21.15.119:0/2852738987 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566fc65b7e0 con 0x5566fc91a9a0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78f4a00640 1 -- 172.21.15.119:0/1433252090 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f78e00cf140 con 0x7f78f0103840 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78ef400640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 0x7f78d40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78eea00640 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc923040 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f78e00cc800 tx=0x7f78e0097a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c20c774278a4d842 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78dfe00640 1 -- 172.21.15.119:0/2852738987 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e009b250 con 0x5566fc91a9a0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78dfe00640 1 -- 172.21.15.119:0/2852738987 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e009be10 con 0x5566fc91a9a0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.544+0000 7f78dfe00640 1 -- 172.21.15.119:0/2852738987 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e00056b0 con 0x5566fc91a9a0 2024-03-28T18:53:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 -- 172.21.15.119:0/2852738987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 msgr2=0x5566fc923040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78ef400640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 0x7f78d40a4eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78e80099a0 tx=0x7f78e8009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc923040 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f78e00cc800 tx=0x7f78e0097a70 comp rx=0 tx=0).stop 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 -- 172.21.15.119:0/2852738987 shutdown_connections 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc925970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc923040 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2852738987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc91a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 -- 172.21.15.119:0/2852738987 >> 172.21.15.119:0/2852738987 conn(0x5566fc9147b0 msgr2=0x5566fc914b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 -- 172.21.15.119:0/2852738987 shutdown_connections 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.545+0000 7f78f89ff740 1 -- 172.21.15.119:0/2852738987 wait complete. 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 -- start start 2024-03-28T18:53:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 Processor -- start 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc922e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc926370 con 0x5566fc91a080 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc83aeb0 con 0x5566fc91a9a0 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.546+0000 7f78f89ff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566fc9264f0 con 0x5566fc923580 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78f89ff740 10 client.0 _ll_register_callbacks cb 0x5566fc82f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78eea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc922e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58060/0 (socket says 172.21.15.119:58060) 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78eea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc922e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48516/0 (socket says 172.21.15.119:48516) 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35870/0 (socket says 172.21.15.119:35870) 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 -- 172.21.15.119:0/2884508280 learned_addr learned my addr 172.21.15.119:0/2884508280 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 -- 172.21.15.119:0/2884508280 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 msgr2=0x5566fc920730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 -- 172.21.15.119:0/2884508280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 msgr2=0x5566fc922e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc922e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 -- 172.21.15.119:0/2884508280 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566fc6bbf70 con 0x5566fc91a9a0 2024-03-28T18:53:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ef400640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78ee000640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f78d8000c00 tx=0x7f78d8009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5498cafaa40a3948 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.547+0000 7f78dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d8043070 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f78d8004450 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78e009be10 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78d804e050 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 10 client.28079 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5566fc553170 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e009be10 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f78d8032640 con 0x5566fc91a9a0 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 10 client.28079 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 10 client.28079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 20 client.28079 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc933510 0x5566fc935920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 4 client.28079 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.548+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566fc88c3f0 con 0x5566fc933510 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.549+0000 7f78eea00640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc933510 0x5566fc935920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.549+0000 7f78eea00640 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc933510 0x5566fc935920 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7a312e6080db22 server_cookie=57f8d1b9413fecf5 in_seq=0 out_seq=0 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.549+0000 7f78dcc00640 10 client.28079 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78dcc00640 1 -- 172.21.15.119:0/2884508280 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566fc88c3f0 con 0x5566fc933510 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78dcc00640 10 client.28079 handle_command_reply: tid=0 2024-03-28T18:53:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 client.28079 shutdown 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 msgr2=0x5566fc9201f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f78d8000c00 tx=0x7f78d8009520 comp rx=0 tx=0).stop 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 shutdown_connections 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc933510 0x5566fc935920 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc923580 0x5566fc920730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a9a0 0x5566fc9201f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78dcc00640 0 client.28079 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 --2- 172.21.15.119:0/2884508280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc922e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 >> 172.21.15.119:0/2884508280 conn(0x5566fc9147b0 msgr2=0x5566fc916500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 shutdown_connections 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 1 -- 172.21.15.119:0/2884508280 wait complete. 2024-03-28T18:53:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.550+0000 7f78f89ff740 20 client.28079 trim_cache size 0 max 16384 2024-03-28T18:53:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 Processor -- start 2024-03-28T18:53:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 -- start start 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc91b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc922920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01af520 con 0x5566fc91a080 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01b0ea0 con 0x5566fc923580 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78f89ff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01afb80 con 0x5566fc91a9a0 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78ef400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58072/0 (socket says 172.21.15.119:58072) 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78ef400640 1 -- 172.21.15.119:0/3355643816 learned_addr learned my addr 172.21.15.119:0/3355643816 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78ee000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc922920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.568+0000 7f78eea00640 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc91b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 -- 172.21.15.119:0/3355643816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 msgr2=0x5566fc922920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc922920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 -- 172.21.15.119:0/3355643816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 msgr2=0x5566fc91b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc91b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 -- 172.21.15.119:0/3355643816 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566fc925ce0 con 0x5566fc923580 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78eea00640 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc91b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78ef400640 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78e8010cf0 tx=0x7f78e8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b0bf3240f1eb47 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78df400640 1 -- 172.21.15.119:0/3355643816 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8010d20 con 0x5566fc923580 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78df400640 1 -- 172.21.15.119:0/3355643816 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e8003e50 con 0x5566fc923580 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.569+0000 7f78df400640 1 -- 172.21.15.119:0/3355643816 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8028420 con 0x5566fc923580 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- 172.21.15.119:0/3355643816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 msgr2=0x5566fc922e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78e8010cf0 tx=0x7f78e8019000 comp rx=0 tx=0).stop 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- 172.21.15.119:0/3355643816 shutdown_connections 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc923580 0x5566fc922e60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc922920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3355643816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc91a080 0x5566fc91b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- 172.21.15.119:0/3355643816 >> 172.21.15.119:0/3355643816 conn(0x5566fc9147b0 msgr2=0x5566fc91aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- 172.21.15.119:0/3355643816 shutdown_connections 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- 172.21.15.119:0/3355643816 wait complete. 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.570+0000 7f78f89ff740 1 -- start start 2024-03-28T18:53:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 Processor -- start 2024-03-28T18:53:02.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc928e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc997610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01b0840 con 0x5566fc923580 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01b01e0 con 0x5566fc91a080 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f01ad700 con 0x5566fc91a9a0 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78f89ff740 10 client.0 _ll_register_callbacks cb 0x7f78e0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78eea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78ef400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc997610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78ee000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc928e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78eea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58074/0 (socket says 172.21.15.119:58074) 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.571+0000 7f78eea00640 1 -- 172.21.15.119:0/3111257673 learned_addr learned my addr 172.21.15.119:0/3111257673 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 -- 172.21.15.119:0/3111257673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 msgr2=0x5566fc928e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc928e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 -- 172.21.15.119:0/3111257673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 msgr2=0x5566fc997610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc997610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 -- 172.21.15.119:0/3111257673 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566fc997b50 con 0x5566fc91a080 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78ee000640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc928e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78eea00640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78e0097b70 tx=0x7f78e0004ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6feae3328c2a3f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e009b070 con 0x5566fc91a080 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f78e00d2980 con 0x5566fc91a080 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78e8003e50 con 0x5566fc91a080 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78e00056b0 con 0x5566fc91a080 2024-03-28T18:53:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78f89ff740 10 client.28085 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.572+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5566fc998dc0 con 0x5566fc91a080 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78e8003e50 con 0x5566fc91a080 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f78e00cca80 con 0x5566fc91a080 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 10 client.28085 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 10 client.28085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 20 client.28085 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc999140 0x5566fc99b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 4 client.28085 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5566fc886740 con 0x5566fc999140 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78ee000640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc999140 0x5566fc99b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78ee000640 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc999140 0x5566fc99b530 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb665d3ded517185 server_cookie=3e551bc211cf50f6 in_seq=0 out_seq=0 2024-03-28T18:53:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.573+0000 7f78dcc00640 10 client.28085 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.574+0000 7f78dcc00640 1 -- 172.21.15.119:0/3111257673 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5566fc886740 con 0x5566fc999140 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.574+0000 7f78dcc00640 10 client.28085 handle_command_reply: tid=0 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 client.28085 shutdown 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 msgr2=0x5566fc928930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f78e0097b70 tx=0x7f78e0004ba0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 shutdown_connections 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5566fc999140 0x5566fc99b530 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5566fc923580 0x5566fc997610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566fc91a9a0 0x5566fc928e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 --2- 172.21.15.119:0/3111257673 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5566fc91a080 0x5566fc928930 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78dcc00640 0 client.28085 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 >> 172.21.15.119:0/3111257673 conn(0x5566fc9147b0 msgr2=0x5566fc91b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 shutdown_connections 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 1 -- 172.21.15.119:0/3111257673 wait complete. 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.575+0000 7f78f89ff740 20 client.28085 trim_cache size 0 max 16384 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "dbdfcb6a-7d46-45c6-85e9-334bd7da8ba5", 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:53:02.582 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:53:02.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 msgr2=0x7f78d40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 0x7f78d40a4eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78e80099a0 tx=0x7f78e8009500 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 msgr2=0x7f78f019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f78e000e9f0 tx=0x7f78e000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 shutdown_connections 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f78d40a29f0 0x7f78d40a4eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78f0109850 0x7f78f019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f78f0104200 0x7f78f019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 --2- 172.21.15.119:0/1433252090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f0103840 0x7f78f019a1e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 >> 172.21.15.119:0/1433252090 conn(0x7f78f00ff2f0 msgr2=0x7f78f00ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 shutdown_connections 2024-03-28T18:53:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.576+0000 7f78f5e00640 1 -- 172.21.15.119:0/1433252090 wait complete. 2024-03-28T18:53:02.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:53:02.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:53:02.671 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:02 smithi119 ceph-mon[31813]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 -- 172.21.15.119:0/660283948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c112640 msgr2=0x7f600c114a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 --2- 172.21.15.119:0/660283948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c112640 0x7f600c114a30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6004009ca0 tx=0x7f600402c190 comp rx=0 tx=0).stop 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 -- 172.21.15.119:0/660283948 shutdown_connections 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 --2- 172.21.15.119:0/660283948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c112640 0x7f600c114a30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 --2- 172.21.15.119:0/660283948 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c074360 0x7f600c112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 --2- 172.21.15.119:0/660283948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c073a40 0x7f600c073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 -- 172.21.15.119:0/660283948 >> 172.21.15.119:0/660283948 conn(0x7f600c078aa0 msgr2=0x7f600c078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 -- 172.21.15.119:0/660283948 shutdown_connections 2024-03-28T18:53:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.805+0000 7f6011200640 1 -- 172.21.15.119:0/660283948 wait complete. 2024-03-28T18:53:02.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 Processor -- start 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 -- start start 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 0x7f600c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 0x7f600c1a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c118040 con 0x7f600c112640 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c117ec0 con 0x7f600c073a40 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.806+0000 7f6011200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1181c0 con 0x7f600c074360 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58090/0 (socket says 172.21.15.119:58090) 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 0x7f600c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 -- 172.21.15.119:0/1626800300 learned_addr learned my addr 172.21.15.119:0/1626800300 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f600a800640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 0x7f600c1a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 -- 172.21.15.119:0/1626800300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 msgr2=0x7f600c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 0x7f600c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 -- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 msgr2=0x7f600c1a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 0x7f600c1a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 -- 172.21.15.119:0/1626800300 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c1a34f0 con 0x7f600c073a40 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f600a800640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 0x7f600c1a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6003e00640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 0x7f600c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.807+0000 7f6009e00640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00d950 tx=0x7f5ffc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.808+0000 7f6002a00640 1 -- 172.21.15.119:0/1626800300 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc017070 con 0x7f600c073a40 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.808+0000 7f6002a00640 1 -- 172.21.15.119:0/1626800300 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc009c30 con 0x7f600c073a40 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.808+0000 7f6002a00640 1 -- 172.21.15.119:0/1626800300 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc00f6b0 con 0x7f600c073a40 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.808+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c1a37e0 con 0x7f600c073a40 2024-03-28T18:53:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.808+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c1ab070 con 0x7f600c073a40 2024-03-28T18:53:02.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 Processor -- start 2024-03-28T18:53:02.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6002a00640 1 -- 172.21.15.119:0/1626800300 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5ffc0109d0 con 0x7f600c073a40 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 -- start start 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a975753a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 0x559a97580680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a974d9250 con 0x559a97574f20 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a97481a30 con 0x559a97575c00 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.811+0000 7f6013dfb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a974d52f0 con 0x559a9757e290 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6009e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a975753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f600a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f600a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:58096/0 (socket says 172.21.15.119:58096) 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 0x559a97580680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6009e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a975753a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48554/0 (socket says 172.21.15.119:48554) 2024-03-28T18:53:02.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f600a800640 1 -- 172.21.15.119:0/2455326999 learned_addr learned my addr 172.21.15.119:0/2455326999 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 -- 172.21.15.119:0/2455326999 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 msgr2=0x559a97574950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 -- 172.21.15.119:0/2455326999 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 msgr2=0x559a975753a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a975753a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 -- 172.21.15.119:0/2455326999 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a9746c690 con 0x559a9757e290 2024-03-28T18:53:02.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f600a800640 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.812+0000 7f6003e00640 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 0x559a97580680 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009f20 tx=0x7f5ff40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62e40a45472df4a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f6002a00640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 0x7f5ff00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f5ffb400640 1 -- 172.21.15.119:0/2455326999 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4004030 con 0x559a9757e290 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f5ffb400640 1 -- 172.21.15.119:0/2455326999 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ff4005500 con 0x559a9757e290 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f5ffb400640 1 -- 172.21.15.119:0/2455326999 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4004e10 con 0x559a9757e290 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f6002a00640 1 -- 172.21.15.119:0/1626800300 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5ffc0ceb80 con 0x7f600c073a40 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f600a800640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 0x7f5ff00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f6013dfb740 1 -- 172.21.15.119:0/2455326999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 msgr2=0x559a97580680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.813+0000 7f6013dfb740 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 0x559a97580680 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009f20 tx=0x7f5ff40118d0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 -- 172.21.15.119:0/2455326999 shutdown_connections 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a9757e290 0x559a97580680 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a975753a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 --2- 172.21.15.119:0/2455326999 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a97574950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 -- 172.21.15.119:0/2455326999 >> 172.21.15.119:0/2455326999 conn(0x559a9753c200 msgr2=0x559a9756f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 -- 172.21.15.119:0/2455326999 shutdown_connections 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f600a800640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 0x7f5ff00a4e30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6004006010 tx=0x7f6004004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 -- 172.21.15.119:0/2455326999 wait complete. 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 -- start start 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.814+0000 7f6013dfb740 1 Processor -- start 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a9757ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a9757b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a97581100 con 0x559a9757e290 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a97494e80 con 0x559a97575c00 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a97581280 con 0x559a97574f20 2024-03-28T18:53:02.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6013dfb740 10 client.0 _ll_register_callbacks cb 0x559a974a7130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35934/0 (socket says 172.21.15.119:35934) 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 -- 172.21.15.119:0/730793782 learned_addr learned my addr 172.21.15.119:0/730793782 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f600a800640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a9757b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6003e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a9757ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 -- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 msgr2=0x559a9757ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a9757ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 -- 172.21.15.119:0/730793782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 msgr2=0x559a9757b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a9757b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.815+0000 7f6009e00640 1 -- 172.21.15.119:0/730793782 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a972b77d0 con 0x559a97574f20 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f6009e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc010fd0 tx=0x7f5ffc010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61fbe66739b8fb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f5fefe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc020050 con 0x559a97574f20 2024-03-28T18:53:02.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5ffc09a540 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ff4005500 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ffc09aa50 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f6013dfb740 10 client.28072 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559a971ed3f0 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.816+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ff4005500 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5ffc099fb0 con 0x559a97574f20 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 10 client.28072 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 10 client.28072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 20 client.28072 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a9758e130 0x559a97590520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 4 client.28072 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a974e6790 con 0x559a9758e130 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.817+0000 7f6003e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a9758e130 0x559a97590520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.818+0000 7f6003e00640 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a9758e130 0x559a97590520 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=983d18ce455d3ee5 server_cookie=dca5a95d0d914059 in_seq=0 out_seq=0 2024-03-28T18:53:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.818+0000 7f5fefe00640 10 client.28072 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.818+0000 7f5fefe00640 1 -- 172.21.15.119:0/730793782 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559a974e6790 con 0x559a9758e130 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.818+0000 7f5fefe00640 10 client.28072 handle_command_reply: tid=0 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.818+0000 7f6013dfb740 1 client.28072 shutdown 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 msgr2=0x559a9757d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc010fd0 tx=0x7f5ffc010d70 comp rx=0 tx=0).stop 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 shutdown_connections 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a9758e130 0x559a97590520 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a9757b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97575c00 0x559a9757ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 --2- 172.21.15.119:0/730793782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97574f20 0x559a9757d8e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f5fefe00640 0 client.28072 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 >> 172.21.15.119:0/730793782 conn(0x559a9753c200 msgr2=0x559a975716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 shutdown_connections 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 1 -- 172.21.15.119:0/730793782 wait complete. 2024-03-28T18:53:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.819+0000 7f6013dfb740 20 client.28072 trim_cache size 0 max 16384 2024-03-28T18:53:02.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.836+0000 7f6013dfb740 1 Processor -- start 2024-03-28T18:53:02.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.836+0000 7f6013dfb740 1 -- start start 2024-03-28T18:53:02.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.836+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a9757d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 0x559a9757f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6013dfb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b3b10 con 0x559a97574f20 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6013dfb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b5490 con 0x559a9757e290 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6013dfb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b4170 con 0x559a97575c00 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f600a800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 0x559a9757f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35950/0 (socket says 172.21.15.119:35950) 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 -- 172.21.15.119:0/3280844706 learned_addr learned my addr 172.21.15.119:0/3280844706 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6009e00640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a9757d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 -- 172.21.15.119:0/3280844706 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 msgr2=0x559a9757f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 0x559a9757f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 -- 172.21.15.119:0/3280844706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 msgr2=0x559a9757d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a9757d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6003e00640 1 -- 172.21.15.119:0/3280844706 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a971ad0f0 con 0x559a97575c00 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f600a800640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 0x559a9757f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.837+0000 7f6009e00640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a9757d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:02.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6003e00640 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4000c00 tx=0x7f5ff40048d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=651fbc8df8aadb05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f5ffaa00640 1 -- 172.21.15.119:0/3280844706 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff402b2d0 con 0x559a97575c00 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f5ffaa00640 1 -- 172.21.15.119:0/3280844706 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ff4030070 con 0x559a97575c00 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f5ffaa00640 1 -- 172.21.15.119:0/3280844706 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4003aa0 con 0x559a97575c00 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 -- 172.21.15.119:0/3280844706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 msgr2=0x559a9757d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4000c00 tx=0x7f5ff40048d0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 -- 172.21.15.119:0/3280844706 shutdown_connections 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a9757e290 0x559a9757f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a9757d8e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 --2- 172.21.15.119:0/3280844706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a97574f20 0x559a9757d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.838+0000 7f6013dfb740 1 -- 172.21.15.119:0/3280844706 >> 172.21.15.119:0/3280844706 conn(0x559a9753c200 msgr2=0x559a9756f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.839+0000 7f6013dfb740 1 -- 172.21.15.119:0/3280844706 shutdown_connections 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.839+0000 7f6013dfb740 1 -- 172.21.15.119:0/3280844706 wait complete. 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.839+0000 7f6013dfb740 1 -- start start 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.839+0000 7f6013dfb740 1 Processor -- start 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.839+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 0x559a97578940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a97577500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b4e30 con 0x559a9757e290 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b47d0 con 0x559a97574f20 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1b1cf0 con 0x559a97575c00 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6013dfb740 10 client.0 _ll_register_callbacks cb 0x7f5ffc00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6009e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 0x559a97578940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:35964/0 (socket says 172.21.15.119:35964) 2024-03-28T18:53:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 -- 172.21.15.119:0/1750025731 learned_addr learned my addr 172.21.15.119:0/1750025731 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f600a800640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a97577500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 -- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 msgr2=0x559a97578940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 0x559a97578940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 -- 172.21.15.119:0/1750025731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 msgr2=0x559a97577500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a97577500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 -- 172.21.15.119:0/1750025731 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a97577c40 con 0x559a97575c00 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6009e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 0x559a97578940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f6003e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009700 tx=0x7f5ff4004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3006f13b6f73e243 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.840+0000 7f5fefe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40043c0 con 0x559a97575c00 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ff4030070 con 0x559a97575c00 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5ff4016050 con 0x559a97575c00 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ff4042050 con 0x559a97575c00 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 10 client.28078 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:02.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559a975e9910 con 0x559a97575c00 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ff4030070 con 0x559a97575c00 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5ff4036830 con 0x559a97575c00 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 10 client.28078 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 10 client.28078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 20 client.28078 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a975e9d60 0x559a975ec180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 4 client.28078 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.841+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559a974e0ae0 con 0x559a975e9d60 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f6009e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a975e9d60 0x559a975ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f6009e00640 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a975e9d60 0x559a975ec180 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4ef3315744c2f9f server_cookie=825d205256779ef7 in_seq=0 out_seq=0 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f5fefe00640 10 client.28078 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f5fefe00640 1 -- 172.21.15.119:0/1750025731 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559a974e0ae0 con 0x559a975e9d60 2024-03-28T18:53:02.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f5fefe00640 10 client.28078 handle_command_reply: tid=0 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f6013dfb740 1 client.28078 shutdown 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 msgr2=0x559a97576fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.842+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4009700 tx=0x7f5ff4004040 comp rx=0 tx=0).stop 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 shutdown_connections 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x559a975e9d60 0x559a975ec180 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559a9757e290 0x559a97577500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559a97575c00 0x559a97576fc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 --2- 172.21.15.119:0/1750025731 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x559a97574f20 0x559a97578940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f5fefe00640 0 client.28078 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 >> 172.21.15.119:0/1750025731 conn(0x559a9753c200 msgr2=0x559a97579580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 shutdown_connections 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 1 -- 172.21.15.119:0/1750025731 wait complete. 2024-03-28T18:53:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6013dfb740 20 client.28078 trim_cache size 0 max 16384 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "dbdfcb6a-7d46-45c6-85e9-334bd7da8ba5": { 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "dbdfcb6a-7d46-45c6-85e9-334bd7da8ba5", 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:53:02.848 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 msgr2=0x7f5ff00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 0x7f5ff00a4e30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6004006010 tx=0x7f6004004a40 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 msgr2=0x7f600c19e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00d950 tx=0x7f5ffc00de10 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.843+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 shutdown_connections 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5ff00a2970 0x7f5ff00a4e30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f600c112640 0x7f600c1a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c074360 0x7f600c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 --2- 172.21.15.119:0/1626800300 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f600c073a40 0x7f600c19e500 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 >> 172.21.15.119:0/1626800300 conn(0x7f600c078aa0 msgr2=0x7f600c10e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:02.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 shutdown_connections 2024-03-28T18:53:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:02.844+0000 7f6011200640 1 -- 172.21.15.119:0/1626800300 wait complete. 2024-03-28T18:53:02.861 INFO:tasks.cephfs.filesystem:scrub status for tag:dbdfcb6a-7d46-45c6-85e9-334bd7da8ba5 - {'path': '/', 'tag': 'dbdfcb6a-7d46-45c6-85e9-334bd7da8ba5', 'options': 'recursive,force'} 2024-03-28T18:53:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:02 smithi196 ceph-mon[36300]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:02 smithi131 ceph-mon[36065]: pgmap v276: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:03 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:53:03.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:03 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:03 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:53:03.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:03 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:03 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:53:03.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:03 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:53:03.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:03 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:03 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:53:03.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:03 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:03 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:53:03.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:03 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:53:03.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:03 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:03 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:53:03.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:03 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:03.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:03 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:53:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:04 smithi119 ceph-mon[31813]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:04 smithi196 ceph-mon[36300]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:04 smithi131 ceph-mon[36065]: pgmap v277: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:06.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:06 smithi119 ceph-mon[31813]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:53:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:06 smithi196 ceph-mon[36300]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:53:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:06 smithi131 ceph-mon[36065]: pgmap v278: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:53:08.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:08 smithi119 ceph-mon[31813]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:53:08.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:08 smithi196 ceph-mon[36300]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:53:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:08 smithi131 ceph-mon[36065]: pgmap v279: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:53:09.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:09 smithi196 ceph-mon[36300]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:53:09.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:09 smithi196 ceph-mon[36300]: pgmap v281: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:09 smithi131 ceph-mon[36065]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:53:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:09 smithi131 ceph-mon[36065]: pgmap v281: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:09 smithi119 ceph-mon[31813]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T18:53:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:09 smithi119 ceph-mon[31813]: pgmap v281: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:12.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:12 smithi119 ceph-mon[31813]: pgmap v282: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:12.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:12 smithi196 ceph-mon[36300]: pgmap v282: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:12.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:12 smithi131 ceph-mon[36065]: pgmap v282: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:14 smithi119 ceph-mon[31813]: pgmap v283: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:14 smithi196 ceph-mon[36300]: pgmap v283: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:14 smithi131 ceph-mon[36065]: pgmap v283: 97 pgs: 22 active+clean+snaptrim, 35 active+clean, 40 active+clean+snaptrim_wait; 5.3 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-03-28T18:53:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:16 smithi119 ceph-mon[31813]: pgmap v284: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:16 smithi196 ceph-mon[36300]: pgmap v284: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:16 smithi131 ceph-mon[36065]: pgmap v284: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:18.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:18 smithi119 ceph-mon[31813]: pgmap v285: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:18.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:18 smithi196 ceph-mon[36300]: pgmap v285: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:18 smithi131 ceph-mon[36065]: pgmap v285: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T18:53:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:20 smithi119 ceph-mon[31813]: pgmap v286: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 188 B/s wr, 14 op/s 2024-03-28T18:53:20.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:20 smithi119 ceph-mon[31813]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:53:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:20 smithi196 ceph-mon[36300]: pgmap v286: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 188 B/s wr, 14 op/s 2024-03-28T18:53:20.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:20 smithi196 ceph-mon[36300]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:53:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:20 smithi131 ceph-mon[36065]: pgmap v286: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 188 B/s wr, 14 op/s 2024-03-28T18:53:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:20 smithi131 ceph-mon[36065]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T18:53:22.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:22 smithi119 ceph-mon[31813]: pgmap v288: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:22.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:22 smithi196 ceph-mon[36300]: pgmap v288: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:22.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:22 smithi131 ceph-mon[36065]: pgmap v288: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:24 smithi119 ceph-mon[31813]: pgmap v289: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:24 smithi196 ceph-mon[36300]: pgmap v289: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:24 smithi131 ceph-mon[36065]: pgmap v289: 97 pgs: 97 active+clean; 2.2 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:53:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:25 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:25.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:25 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:25.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:25 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:53:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:26 smithi119 ceph-mon[31813]: pgmap v290: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:26 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:26 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:26 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:53:26.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:26 smithi196 ceph-mon[36300]: pgmap v290: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:26.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:26 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:26.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:26 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:26.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:26 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:53:26.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:26 smithi131 ceph-mon[36065]: pgmap v290: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:26.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:26 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:53:26.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:26 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:53:26.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:26 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:53:28.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:28 smithi119 ceph-mon[31813]: pgmap v291: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:28.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:28 smithi196 ceph-mon[36300]: pgmap v291: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:28.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:28 smithi131 ceph-mon[36065]: pgmap v291: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:30 smithi119 ceph-mon[31813]: pgmap v292: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:30.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:30 smithi196 ceph-mon[36300]: pgmap v292: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:30.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:30 smithi131 ceph-mon[36065]: pgmap v292: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T18:53:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:32 smithi119 ceph-mon[31813]: pgmap v293: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:32.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:53:32.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:53:32.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:32 smithi196 ceph-mon[36300]: pgmap v293: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:32.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:32 smithi131 ceph-mon[36065]: pgmap v293: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:33.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1403138956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 msgr2=0x7fd6e410d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1403138956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e410d860 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009900 tx=0x7fd6cc02bcd0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1403138956 shutdown_connections 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1403138956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 0x7fd6e4110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1403138956 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e410d860 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1403138956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 0x7fd6e4073e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.087+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1403138956 >> 172.21.15.119:0/1403138956 conn(0x7fd6e40fd010 msgr2=0x7fd6e40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.088+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1403138956 shutdown_connections 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.088+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1403138956 wait complete. 2024-03-28T18:53:33.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 Processor -- start 2024-03-28T18:53:33.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 -- start start 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 0x7fd6e419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 0x7fd6e419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4113ad0 con 0x7fd6e4073a50 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4113950 con 0x7fd6e4074370 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.089+0000 7fd6e9a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4113c50 con 0x7fd6e410dda0 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 0x7fd6e419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37880/0 (socket says 172.21.15.119:37880) 2024-03-28T18:53:33.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 -- 172.21.15.119:0/1430152363 learned_addr learned my addr 172.21.15.119:0/1430152363 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 0x7fd6e419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 -- 172.21.15.119:0/1430152363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 msgr2=0x7fd6e419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 0x7fd6e419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 -- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 msgr2=0x7fd6e419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 0x7fd6e419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 -- 172.21.15.119:0/1430152363 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e419f050 con 0x7fd6e4074370 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.090+0000 7fd6db400640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc000c00 tx=0x7fd6cc004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.091+0000 7fd6e1600640 1 -- 172.21.15.119:0/1430152363 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc042020 con 0x7fd6e4074370 2024-03-28T18:53:33.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.091+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e419f2e0 con 0x7fd6e4074370 2024-03-28T18:53:33.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.091+0000 7fd6e1600640 1 -- 172.21.15.119:0/1430152363 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6cc0054e0 con 0x7fd6e4074370 2024-03-28T18:53:33.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.091+0000 7fd6e1600640 1 -- 172.21.15.119:0/1430152363 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc003af0 con 0x7fd6e4074370 2024-03-28T18:53:33.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.091+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e41a6bd0 con 0x7fd6e4074370 2024-03-28T18:53:33.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6e1600640 1 -- 172.21.15.119:0/1430152363 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fd6cc045020 con 0x7fd6e4074370 2024-03-28T18:53:33.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 Processor -- start 2024-03-28T18:53:33.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 -- start start 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 0x562397e07570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 0x562397e12850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397d6b250 con 0x562397e10460 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397d13a30 con 0x562397e07dd0 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.094+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397d672f0 con 0x562397e070f0 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 0x562397e07570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6db400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 0x562397e12850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37884/0 (socket says 172.21.15.119:37884) 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 -- 172.21.15.119:0/2972110889 learned_addr learned my addr 172.21.15.119:0/2972110889 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 -- 172.21.15.119:0/2972110889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 msgr2=0x562397e07570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 0x562397e07570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 -- 172.21.15.119:0/2972110889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 msgr2=0x562397e12850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 0x562397e12850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e9000640 1 -- 172.21.15.119:0/2972110889 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562397cfe690 con 0x562397e07dd0 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 0x562397e07570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6db400640 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 0x562397e12850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.095+0000 7fd6e1600640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 0x7fd6c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6e9000640 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009f60 tx=0x7fd6dc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=361581b92399e540 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6e1600640 1 -- 172.21.15.119:0/1430152363 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd6cc0f09e0 con 0x7fd6e4074370 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6d9600640 1 -- 172.21.15.119:0/2972110889 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc01e070 con 0x562397e07dd0 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 0x7fd6c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6d9600640 1 -- 172.21.15.119:0/2972110889 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6dc003c30 con 0x562397e07dd0 2024-03-28T18:53:33.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.096+0000 7fd6d9600640 1 -- 172.21.15.119:0/2972110889 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc00fa20 con 0x562397e07dd0 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2972110889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 msgr2=0x562397e06b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009f60 tx=0x7fd6dc009f90 comp rx=0 tx=0).stop 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 0x7fd6c80a4e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x562397e08320 tx=0x7fd6d4005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2972110889 shutdown_connections 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e10460 0x562397e12850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e070f0 0x562397e07570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2972110889 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e07dd0 0x562397e06b20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2972110889 >> 172.21.15.119:0/2972110889 conn(0x562397dce200 msgr2=0x562397e01b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2972110889 shutdown_connections 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2972110889 wait complete. 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 -- start start 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.097+0000 7fd6eddc5740 1 Processor -- start 2024-03-28T18:53:33.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 0x562397e0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397e132d0 con 0x562397e07dd0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397d26e80 con 0x562397e070f0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x562397e13450 con 0x562397e10460 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6eddc5740 10 client.0 _ll_register_callbacks cb 0x562397d39130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37890/0 (socket says 172.21.15.119:37890) 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e3e00640 1 -- 172.21.15.119:0/1846385722 learned_addr learned my addr 172.21.15.119:0/1846385722 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 0x562397e0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6db400640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 -- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 msgr2=0x562397e0fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 -- 172.21.15.119:0/1846385722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 msgr2=0x562397e0cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.098+0000 7fd6e9000640 1 -- 172.21.15.119:0/1846385722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562397b497d0 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 0x562397e0d430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc011d60 tx=0x7fd6dc0048a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff79571c70aa9bed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6d2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc027020 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6dc003c30 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd6dc00f760 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6dc032050 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6eddc5740 10 client.28090 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.099+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562397a7f3f0 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6dc003c30 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd6dc0043a0 con 0x562397e10460 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 10 client.28090 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 10 client.28090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 20 client.28090 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e202a0 0x562397e22690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 4 client.28090 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:33.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562397d78790 con 0x562397e202a0 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.100+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e202a0 0x562397e22690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.101+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e202a0 0x562397e22690 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d402438e71d909 server_cookie=224943e7e6edbde2 in_seq=0 out_seq=0 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.101+0000 7fd6d2000640 10 client.28090 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.101+0000 7fd6d2000640 1 -- 172.21.15.119:0/1846385722 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562397d78790 con 0x562397e202a0 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.101+0000 7fd6d2000640 10 client.28090 handle_command_reply: tid=0 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.101+0000 7fd6eddc5740 1 client.28090 shutdown 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 msgr2=0x562397e0d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 0x562397e0d430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc011d60 tx=0x7fd6dc0048a0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 shutdown_connections 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e202a0 0x562397e22690 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e10460 0x562397e0d430 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1846385722 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6d2000640 0 client.28090 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 >> 172.21.15.119:0/1846385722 conn(0x562397dce200 msgr2=0x562397e03890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 shutdown_connections 2024-03-28T18:53:33.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1846385722 wait complete. 2024-03-28T18:53:33.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.102+0000 7fd6eddc5740 20 client.28090 trim_cache size 0 max 16384 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.119+0000 7fd6eddc5740 1 Processor -- start 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.119+0000 7fd6eddc5740 1 -- start start 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.119+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41af4a0 con 0x562397e07dd0 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41b0e20 con 0x562397e070f0 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41afb00 con 0x562397dce200 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43440/0 (socket says 172.21.15.119:43440) 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 -- 172.21.15.119:0/2268731666 learned_addr learned my addr 172.21.15.119:0/2268731666 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e9000640 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6db400640 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 -- 172.21.15.119:0/2268731666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 msgr2=0x562397e0f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 -- 172.21.15.119:0/2268731666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 msgr2=0x562397e0fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 -- 172.21.15.119:0/2268731666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562397e0dee0 con 0x562397dce200 2024-03-28T18:53:33.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.120+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009c90 tx=0x7fd6d401b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2cb5873fd0e563e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6d8c00640 1 -- 172.21.15.119:0/2268731666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40044b0 con 0x562397dce200 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6d8c00640 1 -- 172.21.15.119:0/2268731666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4030070 con 0x562397dce200 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6d8c00640 1 -- 172.21.15.119:0/2268731666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d401f970 con 0x562397dce200 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2268731666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 msgr2=0x562397e0bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009c90 tx=0x7fd6d401b000 comp rx=0 tx=0).stop 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2268731666 shutdown_connections 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e07dd0 0x562397e0fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397e070f0 0x562397e0f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/2268731666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397dce200 0x562397e0bdc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.121+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2268731666 >> 172.21.15.119:0/2268731666 conn(0x562397e0d590 msgr2=0x562397e0b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.122+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2268731666 shutdown_connections 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.122+0000 7fd6eddc5740 1 -- 172.21.15.119:0/2268731666 wait complete. 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.122+0000 7fd6eddc5740 1 -- start start 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.122+0000 7fd6eddc5740 1 Processor -- start 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 0x562397e0a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 0x562397e1b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41b07c0 con 0x562397e070f0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41b0160 con 0x562397dce200 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41ad680 con 0x562397e07dd0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6eddc5740 10 client.0 _ll_register_callbacks cb 0x7fd6dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6db400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 0x562397e0a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e9000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 0x562397e1b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37896/0 (socket says 172.21.15.119:37896) 2024-03-28T18:53:33.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 -- 172.21.15.119:0/1992273804 learned_addr learned my addr 172.21.15.119:0/1992273804 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 -- 172.21.15.119:0/1992273804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 msgr2=0x562397e1b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 0x562397e1b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 -- 172.21.15.119:0/1992273804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 msgr2=0x562397e0a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 0x562397e0a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 -- 172.21.15.119:0/1992273804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562397a3f0f0 con 0x562397dce200 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6db400640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 0x562397e0a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e9000640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 0x562397e1b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.123+0000 7fd6e3e00640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4003fe0 tx=0x7fd6d4004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d531edafad13300d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4005240 con 0x562397dce200 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd6d40053e0 con 0x562397dce200 2024-03-28T18:53:33.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6d4030070 con 0x562397dce200 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6d40055e0 con 0x562397dce200 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6eddc5740 10 client.28121 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562397e7c120 con 0x562397dce200 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d4030070 con 0x562397dce200 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.124+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd6d402baf0 con 0x562397dce200 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 10 client.28121 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 10 client.28121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 20 client.28121 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e10460 0x562397e7e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 4 client.28121 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562397d72ae0 con 0x562397e10460 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6db400640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e10460 0x562397e7e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6db400640 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e10460 0x562397e7e830 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea436c67b7fa0ed9 server_cookie=ded66c07a346c761 in_seq=0 out_seq=0 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6d2000640 10 client.28121 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6d2000640 1 -- 172.21.15.119:0/1992273804 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562397d72ae0 con 0x562397e10460 2024-03-28T18:53:33.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.125+0000 7fd6d2000640 10 client.28121 handle_command_reply: tid=0 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 client.28121 shutdown 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 msgr2=0x562397e0a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4003fe0 tx=0x7fd6d4004540 comp rx=0 tx=0).stop 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 shutdown_connections 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x562397e10460 0x562397e7e830 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562397e07dd0 0x562397e1b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562397e070f0 0x562397e0a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 --2- 172.21.15.119:0/1992273804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x562397dce200 0x562397e0a3c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6d2000640 0 client.28121 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 >> 172.21.15.119:0/1992273804 conn(0x562397e0d590 msgr2=0x562397e0bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 shutdown_connections 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 1 -- 172.21.15.119:0/1992273804 wait complete. 2024-03-28T18:53:33.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.126+0000 7fd6eddc5740 20 client.28121 trim_cache size 0 max 16384 2024-03-28T18:53:33.130 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:53:33.130 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:53:33.130 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:53:33.130 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:53:33.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 msgr2=0x7fd6c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 0x7fd6c80a4e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x562397e08320 tx=0x7fd6d4005f90 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 msgr2=0x7fd6e419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc000c00 tx=0x7fd6cc004360 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 shutdown_connections 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd6c80a2970 0x7fd6c80a4e30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6e410dda0 0x7fd6e419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd6e4074370 0x7fd6e419a5f0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 --2- 172.21.15.119:0/1430152363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e4073a50 0x7fd6e419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 >> 172.21.15.119:0/1430152363 conn(0x7fd6e40fd010 msgr2=0x7fd6e40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 shutdown_connections 2024-03-28T18:53:33.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.127+0000 7fd6e9a00640 1 -- 172.21.15.119:0/1430152363 wait complete. 2024-03-28T18:53:33.146 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:53:33.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:53:33.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 -- 172.21.15.119:0/1264220947 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 msgr2=0x7fc9f4101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 --2- 172.21.15.119:0/1264220947 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4101ab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e002bc30 comp rx=0 tx=0).stop 2024-03-28T18:53:33.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 -- 172.21.15.119:0/1264220947 shutdown_connections 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 --2- 172.21.15.119:0/1264220947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 0x7fc9f410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 --2- 172.21.15.119:0/1264220947 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4101ab0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 --2- 172.21.15.119:0/1264220947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 -- 172.21.15.119:0/1264220947 >> 172.21.15.119:0/1264220947 conn(0x7fc9f40fd140 msgr2=0x7fc9f40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 -- 172.21.15.119:0/1264220947 shutdown_connections 2024-03-28T18:53:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.357+0000 7fc9f9600640 1 -- 172.21.15.119:0/1264220947 wait complete. 2024-03-28T18:53:33.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 Processor -- start 2024-03-28T18:53:33.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 -- start start 2024-03-28T18:53:33.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 0x7fc9f4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4111a60 con 0x7fc9f4107640 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f41118e0 con 0x7fc9f4101630 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f9600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f4111be0 con 0x7fc9f4101ff0 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.359+0000 7fc9f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37920/0 (socket says 172.21.15.119:37920) 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 -- 172.21.15.119:0/3162028756 learned_addr learned my addr 172.21.15.119:0/3162028756 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2000640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 0x7fc9f4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f3400640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 -- 172.21.15.119:0/3162028756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 msgr2=0x7fc9f4198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 0x7fc9f4198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 -- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 msgr2=0x7fc9f419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 -- 172.21.15.119:0/3162028756 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f419cf30 con 0x7fc9f4101630 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f3400640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:33.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9f2a00640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400e9f0 tx=0x7fc9e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.360+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400cd80 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.361+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e4010910 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.361+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f419d220 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.361+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4018990 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.361+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f41a4ab0 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.362+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f40688e0 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.364+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fc9e4010ac0 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.365+0000 7fc9eb400640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 0x7fc9d80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.365+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fc9e40cf2c0 con 0x7fc9f4101630 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.365+0000 7fc9f2000640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 0x7fc9d80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.366+0000 7fc9f2000640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 0x7fc9d80a4e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc9e002efd0 tx=0x7fc9e0000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.374+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc9e409bf30 con 0x7fc9f4101630 2024-03-28T18:53:33.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.616+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f4199290 con 0x7fc9f4101630 2024-03-28T18:53:33.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.617+0000 7fc9eb400640 1 -- 172.21.15.119:0/3162028756 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7fc9e409b830 con 0x7fc9f4101630 2024-03-28T18:53:33.617 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:53:33.617 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:53:33.617 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:53:33.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 msgr2=0x7fc9d80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 0x7fc9d80a4e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc9e002efd0 tx=0x7fc9e0000bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 msgr2=0x7fc9f4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400e9f0 tx=0x7fc9e400eeb0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 shutdown_connections 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fc9d80a2970 0x7fc9d80a4e30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9f4107640 0x7fc9f419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9f4101ff0 0x7fc9f4198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 --2- 172.21.15.119:0/3162028756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc9f4101630 0x7fc9f4197f40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.619+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 >> 172.21.15.119:0/3162028756 conn(0x7fc9f40fd140 msgr2=0x7fc9f40fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.620+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 shutdown_connections 2024-03-28T18:53:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.620+0000 7fc9f9600640 1 -- 172.21.15.119:0/3162028756 wait complete. 2024-03-28T18:53:33.639 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:33.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 -- 172.21.15.119:0/3452420978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 msgr2=0x7f8a9c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/3452420978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c109c30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009930 tx=0x7f8a8c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 -- 172.21.15.119:0/3452420978 shutdown_connections 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/3452420978 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/3452420978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/3452420978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c109c30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 -- 172.21.15.119:0/3452420978 >> 172.21.15.119:0/3452420978 conn(0x7f8a9c0ff2f0 msgr2=0x7f8a9c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 -- 172.21.15.119:0/3452420978 shutdown_connections 2024-03-28T18:53:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.854+0000 7f8aa2a00640 1 -- 172.21.15.119:0/3452420978 wait complete. 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.855+0000 7f8aa2a00640 1 Processor -- start 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 -- start start 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c113cb0 con 0x7f8a9c103840 2024-03-28T18:53:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c113b30 con 0x7f8a9c104200 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa2a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c113e30 con 0x7f8a9c109850 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8a9be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8a9be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54346/0 (socket says 172.21.15.119:54346) 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8a9be00640 1 -- 172.21.15.119:0/312124864 learned_addr learned my addr 172.21.15.119:0/312124864 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.856+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37936/0 (socket says 172.21.15.119:37936) 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 -- 172.21.15.119:0/312124864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 msgr2=0x7f8a9c19ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c19ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 -- 172.21.15.119:0/312124864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 msgr2=0x7f8a9c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 -- 172.21.15.119:0/312124864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c19f140 con 0x7f8a9c103840 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a93e00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a9be00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c005ab0 tx=0x7f8a8c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a99600640 1 -- 172.21.15.119:0/312124864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c043020 con 0x7f8a9c103840 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c19f3d0 con 0x7f8a9c103840 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a99600640 1 -- 172.21.15.119:0/312124864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a8c0054e0 con 0x7f8a9c103840 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8a99600640 1 -- 172.21.15.119:0/312124864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c003c50 con 0x7f8a9c103840 2024-03-28T18:53:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.857+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c1a6cc0 con 0x7f8a9c103840 2024-03-28T18:53:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.860+0000 7f8aa5766740 1 Processor -- start 2024-03-28T18:53:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.860+0000 7f8a99600640 1 -- 172.21.15.119:0/312124864 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f8a8c046020 con 0x7f8a9c103840 2024-03-28T18:53:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 -- start start 2024-03-28T18:53:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 0x565174d35630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d38050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174c902c0 con 0x565174d35c00 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174c38c90 con 0x565174d2d080 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa5766740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174c8c360 con 0x565174d2c6d0 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8a9be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 0x565174d35630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43490/0 (socket says 172.21.15.119:43490) 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8a9be00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 0x565174d35630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37942/0 (socket says 172.21.15.119:37942) 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8aa0a00640 1 -- 172.21.15.119:0/1996036089 learned_addr learned my addr 172.21.15.119:0/1996036089 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.861+0000 7f8a93e00640 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d38050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 -- 172.21.15.119:0/1996036089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 msgr2=0x565174d35630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 0x565174d35630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 -- 172.21.15.119:0/1996036089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 msgr2=0x565174d38050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d38050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 -- 172.21.15.119:0/1996036089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565174c5d080 con 0x565174d2c6d0 2024-03-28T18:53:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8a94010710 tx=0x7f8a94010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81e72b16d7e148cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8a91200640 1 -- 172.21.15.119:0/1996036089 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400e8d0 con 0x565174d2c6d0 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8a91200640 1 -- 172.21.15.119:0/1996036089 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a94012450 con 0x565174d2c6d0 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.862+0000 7f8a91200640 1 -- 172.21.15.119:0/1996036089 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9401c610 con 0x565174d2c6d0 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8a99600640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 0x7f8a7c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8a99600640 1 -- 172.21.15.119:0/312124864 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f8a8c0f0e40 con 0x7f8a9c103840 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8a9be00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 0x7f8a7c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 -- 172.21.15.119:0/1996036089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 msgr2=0x565174d2cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8a94010710 tx=0x7f8a94010bd0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 -- 172.21.15.119:0/1996036089 shutdown_connections 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d38050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2d080 0x565174d35630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1996036089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2c6d0 0x565174d2cab0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 -- 172.21.15.119:0/1996036089 >> 172.21.15.119:0/1996036089 conn(0x565174d26da0 msgr2=0x565174d27180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 -- 172.21.15.119:0/1996036089 shutdown_connections 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.863+0000 7f8aa5766740 1 -- 172.21.15.119:0/1996036089 wait complete. 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8a9be00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 0x7f8a7c0a4e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x565174d2c1a0 tx=0x7f8a8c0dd040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 -- start start 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 Processor -- start 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d32860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.864+0000 7f8aa5766740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174d38b70 con 0x565174d2d080 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa5766740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174c4c0d0 con 0x565174d2c6d0 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa5766740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x565174d38cf0 con 0x565174d35c00 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa5766740 10 client.0 _ll_register_callbacks cb 0x565174c50e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d32860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37948/0 (socket says 172.21.15.119:37948) 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa0a00640 1 -- 172.21.15.119:0/110809364 learned_addr learned my addr 172.21.15.119:0/110809364 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d32dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 -- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 msgr2=0x565174d32dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d32dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 -- 172.21.15.119:0/110809364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 msgr2=0x565174d354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 -- 172.21.15.119:0/110809364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651749a43f0 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a93e00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d32860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005d00 tx=0x7f8a84011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1abafbb96917851 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.865+0000 7f8a89600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84010c30 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f8a84010dd0 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a94012450 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a84004020 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8aa5766740 10 client.17898 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565174a6e7d0 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a94012450 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.866+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8a8401a410 con 0x565174d2d080 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 10 client.17898 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 10 client.17898 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:53:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 20 client.17898 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174d45a90 0x565174d47eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 4 client.17898 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565174c9d800 con 0x565174d45a90 2024-03-28T18:53:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174d45a90 0x565174d47eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174d45a90 0x565174d47eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae562b917d36cc93 server_cookie=cbb2a1f4ca864f9b in_seq=0 out_seq=0 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.867+0000 7f8a89600640 10 client.17898 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8a89600640 1 -- 172.21.15.119:0/110809364 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x565174c9d800 con 0x565174d45a90 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8a89600640 10 client.17898 handle_command_reply: tid=0 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 client.17898 shutdown 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 msgr2=0x565174d32860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d32860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005d00 tx=0x7f8a84011bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 shutdown_connections 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174d45a90 0x565174d47eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d32860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 --2- 172.21.15.119:0/110809364 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8a89600640 0 client.17898 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 >> 172.21.15.119:0/110809364 conn(0x565174d26da0 msgr2=0x565174d28af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 shutdown_connections 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 1 -- 172.21.15.119:0/110809364 wait complete. 2024-03-28T18:53:33.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.869+0000 7f8aa5766740 20 client.17898 trim_cache size 0 max 16384 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 Processor -- start 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 -- start start 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d34f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d354d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1af690 con 0x565174d2d080 2024-03-28T18:53:33.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1b1010 con 0x565174d2c6d0 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa5766740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1afcf0 con 0x565174d35c00 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d34f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37954/0 (socket says 172.21.15.119:37954) 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa0a00640 1 -- 172.21.15.119:0/1647395309 learned_addr learned my addr 172.21.15.119:0/1647395309 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.883+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 -- 172.21.15.119:0/1647395309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 msgr2=0x565174d354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d354d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 -- 172.21.15.119:0/1647395309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 msgr2=0x565174d260f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 -- 172.21.15.119:0/1647395309 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565174d2e470 con 0x565174d2d080 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a93e00640 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d34f90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8a8401a800 tx=0x7f8a84010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985e53564febc257 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.884+0000 7f8a8be00640 1 -- 172.21.15.119:0/1647395309 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8402e070 con 0x565174d2d080 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8a8be00640 1 -- 172.21.15.119:0/1647395309 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a8401a3f0 con 0x565174d2d080 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8a8be00640 1 -- 172.21.15.119:0/1647395309 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a840052a0 con 0x565174d2d080 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 -- 172.21.15.119:0/1647395309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 msgr2=0x565174d34f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d34f90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8a8401a800 tx=0x7f8a84010f20 comp rx=0 tx=0).stop 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 -- 172.21.15.119:0/1647395309 shutdown_connections 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d35c00 0x565174d354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d2d080 0x565174d34f90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 --2- 172.21.15.119:0/1647395309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d260f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 -- 172.21.15.119:0/1647395309 >> 172.21.15.119:0/1647395309 conn(0x565174d26da0 msgr2=0x565174d28930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 -- 172.21.15.119:0/1647395309 shutdown_connections 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.885+0000 7f8aa5766740 1 -- 172.21.15.119:0/1647395309 wait complete. 2024-03-28T18:53:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 -- start start 2024-03-28T18:53:33.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 Processor -- start 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d39b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1b09b0 con 0x565174d35c00 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1b0350 con 0x565174d2c6d0 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c1ad870 con 0x565174d2d080 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.886+0000 7f8aa5766740 10 client.0 _ll_register_callbacks cb 0x7f8a8400bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43518/0 (socket says 172.21.15.119:43518) 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 -- 172.21.15.119:0/2669226733 learned_addr learned my addr 172.21.15.119:0/2669226733 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d39b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 -- 172.21.15.119:0/2669226733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 msgr2=0x565174d39b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d39b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 -- 172.21.15.119:0/2669226733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 msgr2=0x565174d9a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d9a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 -- 172.21.15.119:0/2669226733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565174d9a900 con 0x565174d2d080 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d9a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a93e00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8a84011480 tx=0x7f8a84005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b176e1ee21fcdc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a89600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8402e050 con 0x565174d2d080 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.887+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a8401a3f0 con 0x565174d2d080 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f8a84004660 con 0x565174d2d080 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a84010c30 con 0x565174d2d080 2024-03-28T18:53:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 10 client.28120 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565174da1ca0 con 0x565174d2d080 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a8401a3f0 con 0x565174d2d080 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8a84041030 con 0x565174d2d080 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 10 client.28120 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 10 client.28120 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 20 client.28120 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174da23a0 0x565174da47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 4 client.28120 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.888+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565174c97b50 con 0x565174da23a0 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174da23a0 0x565174da47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa0a00640 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174da23a0 0x565174da47c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=16cb26c30b8f5666 server_cookie=6283b46bdf66d6de in_seq=0 out_seq=0 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8a89600640 10 client.28120 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8a89600640 1 -- 172.21.15.119:0/2669226733 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565174c97b50 con 0x565174da23a0 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8a89600640 10 client.28120 handle_command_reply: tid=0 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 client.28120 shutdown 2024-03-28T18:53:33.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 msgr2=0x565174d3a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8a84011480 tx=0x7f8a84005270 comp rx=0 tx=0).stop 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 shutdown_connections 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x565174da23a0 0x565174da47c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x565174d35c00 0x565174d9a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x565174d2d080 0x565174d3a0d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 --2- 172.21.15.119:0/2669226733 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x565174d2c6d0 0x565174d39b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8a89600640 0 client.28120 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.889+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 >> 172.21.15.119:0/2669226733 conn(0x565174d26da0 msgr2=0x565174d28050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 shutdown_connections 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa5766740 1 -- 172.21.15.119:0/2669226733 wait complete. 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa5766740 20 client.28120 trim_cache size 0 max 16384 2024-03-28T18:53:33.893 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 msgr2=0x7f8a7c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 0x7f8a7c0a4e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x565174d2c1a0 tx=0x7f8a8c0dd040 comp rx=0 tx=0).stop 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 msgr2=0x7f8a9c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.890+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c005ab0 tx=0x7f8a8c0043b0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 shutdown_connections 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f8a7c0a2970 0x7f8a7c0a4e30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a9c109850 0x7f8a9c19ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f8a9c104200 0x7f8a9c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 --2- 172.21.15.119:0/312124864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a9c103840 0x7f8a9c19a150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:33.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 >> 172.21.15.119:0/312124864 conn(0x7f8a9c0ff2f0 msgr2=0x7f8a9c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:33.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 shutdown_connections 2024-03-28T18:53:33.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:33.891+0000 7f8aa2a00640 1 -- 172.21.15.119:0/312124864 wait complete. 2024-03-28T18:53:33.909 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:33.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.123+0000 7f661f600640 1 -- 172.21.15.119:0/4273752377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6618100490 msgr2=0x7f6618100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.123+0000 7f661f600640 1 --2- 172.21.15.119:0/4273752377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6618100490 0x7f6618100870 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f660c009900 tx=0x7f660c02bc50 comp rx=0 tx=0).stop 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.123+0000 7f661f600640 1 -- 172.21.15.119:0/4273752377 shutdown_connections 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.123+0000 7f661f600640 1 --2- 172.21.15.119:0/4273752377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f661810e040 0x7f6618110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.123+0000 7f661f600640 1 --2- 172.21.15.119:0/4273752377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661810db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.124+0000 7f661f600640 1 --2- 172.21.15.119:0/4273752377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6618100490 0x7f6618100870 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.124+0000 7f661f600640 1 -- 172.21.15.119:0/4273752377 >> 172.21.15.119:0/4273752377 conn(0x7f6618077700 msgr2=0x7f6618077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.124+0000 7f661f600640 1 -- 172.21.15.119:0/4273752377 shutdown_connections 2024-03-28T18:53:34.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.124+0000 7f661f600640 1 -- 172.21.15.119:0/4273752377 wait complete. 2024-03-28T18:53:34.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 Processor -- start 2024-03-28T18:53:34.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 -- start start 2024-03-28T18:53:34.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 0x7f661819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 0x7f661819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618113d30 con 0x7f6618100490 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618113bb0 con 0x7f6618100e40 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661f600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618113eb0 con 0x7f661810e040 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 0x7f661819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f6617e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f6617e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37982/0 (socket says 172.21.15.119:37982) 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 0x7f661819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54410/0 (socket says 172.21.15.119:54410) 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f6617e00640 1 -- 172.21.15.119:0/3254805377 learned_addr learned my addr 172.21.15.119:0/3254805377 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.125+0000 7f661d600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 0x7f661819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 -- 172.21.15.119:0/3254805377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 msgr2=0x7f661819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 0x7f661819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 -- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 msgr2=0x7f661819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 0x7f661819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 -- 172.21.15.119:0/3254805377 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661819f2a0 con 0x7f6618100e40 2024-03-28T18:53:34.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f661d600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 0x7f661819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6617e00640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f660400ea60 tx=0x7f660400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6615600640 1 -- 172.21.15.119:0/3254805377 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660400cde0 con 0x7f6618100e40 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661819f590 con 0x7f6618100e40 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6615600640 1 -- 172.21.15.119:0/3254805377 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66040035f0 con 0x7f6618100e40 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.126+0000 7f6615600640 1 -- 172.21.15.119:0/3254805377 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604010670 con 0x7f6618100e40 2024-03-28T18:53:34.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.127+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66181a6e20 con 0x7f6618100e40 2024-03-28T18:53:34.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6615600640 1 -- 172.21.15.119:0/3254805377 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f66040037a0 con 0x7f6618100e40 2024-03-28T18:53:34.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 Processor -- start 2024-03-28T18:53:34.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 -- start start 2024-03-28T18:53:34.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12d080 0x55992b135630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b138050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b0902c0 con 0x55992b135c00 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b038c90 con 0x55992b12c6d0 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f6622429740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b08c360 con 0x55992b12d080 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.130+0000 7f661d600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:37984/0 (socket says 172.21.15.119:37984) 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 -- 172.21.15.119:0/3738741234 learned_addr learned my addr 172.21.15.119:0/3738741234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f6617e00640 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b138050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661cc00640 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12d080 0x55992b135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 -- 172.21.15.119:0/3738741234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12d080 msgr2=0x55992b135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12d080 0x55992b135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 -- 172.21.15.119:0/3738741234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 msgr2=0x55992b138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b138050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 -- 172.21.15.119:0/3738741234 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55992b05d080 con 0x55992b12c6d0 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f6615600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 0x7f66000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f661d600640 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6610011790 tx=0x7f6610011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e2b75eb5620d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.131+0000 7f6615600640 1 -- 172.21.15.119:0/3254805377 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f66040cf240 con 0x7f6618100e40 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f661cc00640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 0x7f66000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f6609c00640 1 -- 172.21.15.119:0/3738741234 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610003c30 con 0x55992b12c6d0 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f6609c00640 1 -- 172.21.15.119:0/3738741234 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6610003dd0 con 0x55992b12c6d0 2024-03-28T18:53:34.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f6609c00640 1 -- 172.21.15.119:0/3738741234 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66100193e0 con 0x55992b12c6d0 2024-03-28T18:53:34.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f6622429740 1 -- 172.21.15.119:0/3738741234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 msgr2=0x55992b12cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f661cc00640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 0x7f66000a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f660c006010 tx=0x7f660c000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.132+0000 7f6622429740 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6610011790 tx=0x7f6610011c50 comp rx=0 tx=0).stop 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 -- 172.21.15.119:0/3738741234 shutdown_connections 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b138050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12d080 0x55992b135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 --2- 172.21.15.119:0/3738741234 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12c6d0 0x55992b12cab0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 -- 172.21.15.119:0/3738741234 >> 172.21.15.119:0/3738741234 conn(0x55992b126da0 msgr2=0x55992b127180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 -- 172.21.15.119:0/3738741234 shutdown_connections 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 -- 172.21.15.119:0/3738741234 wait complete. 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 -- start start 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.133+0000 7f6622429740 1 Processor -- start 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12d080 0x55992b132890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 0x55992b132e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b138b70 con 0x55992b12d080 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b04c0d0 con 0x55992b135c00 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55992b138cf0 con 0x55992b12c6d0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6622429740 10 client.0 _ll_register_callbacks cb 0x55992b050e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f6617e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12d080 0x55992b132890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43546/0 (socket says 172.21.15.119:43546) 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661cc00640 1 -- 172.21.15.119:0/925971030 learned_addr learned my addr 172.21.15.119:0/925971030 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661d600640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 0x55992b132e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661cc00640 1 -- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 msgr2=0x55992b132e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.134+0000 7f661cc00640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 0x55992b132e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f661cc00640 1 -- 172.21.15.119:0/925971030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12d080 msgr2=0x55992b132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f661cc00640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12d080 0x55992b132890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f661cc00640 1 -- 172.21.15.119:0/925971030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55992ada43f0 con 0x55992b12c6d0 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f661d600640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 0x55992b132e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f661cc00640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f660c035000 tx=0x7f660c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ad74623cb5769e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f65fe000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c005160 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f660c005300 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6610003dd0 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f660c004c80 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.135+0000 7f6622429740 10 client.28132 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55992ae6e7d0 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6610003dd0 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f660c051070 con 0x55992b12c6d0 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 10 client.28132 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 10 client.28132 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 20 client.28132 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b145940 0x55992b147d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 4 client.28132 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55992b09d800 con 0x55992b145940 2024-03-28T18:53:34.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.136+0000 7f6617e00640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b145940 0x55992b147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.137+0000 7f6617e00640 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b145940 0x55992b147d30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=295d78074970f4c0 server_cookie=a4e2b99aae866c7b in_seq=0 out_seq=0 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.137+0000 7f65fe000640 10 client.28132 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.137+0000 7f65fe000640 1 -- 172.21.15.119:0/925971030 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55992b09d800 con 0x55992b145940 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.137+0000 7f65fe000640 10 client.28132 handle_command_reply: tid=0 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.137+0000 7f6622429740 1 client.28132 shutdown 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 msgr2=0x55992b135500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f660c035000 tx=0x7f660c009260 comp rx=0 tx=0).stop 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 shutdown_connections 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b135c00 0x55992b132e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b145940 0x55992b147d30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12d080 0x55992b132890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 --2- 172.21.15.119:0/925971030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b135500 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 >> 172.21.15.119:0/925971030 conn(0x55992b126da0 msgr2=0x55992b128af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f65fe000640 0 client.28132 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:34.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 shutdown_connections 2024-03-28T18:53:34.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 1 -- 172.21.15.119:0/925971030 wait complete. 2024-03-28T18:53:34.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.138+0000 7f6622429740 20 client.28132 trim_cache size 0 max 16384 2024-03-28T18:53:34.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.151+0000 7f6622429740 1 Processor -- start 2024-03-28T18:53:34.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.151+0000 7f6622429740 1 -- start start 2024-03-28T18:53:34.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b134fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b135500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181af570 con 0x55992b135c00 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181b0ef0 con 0x55992b12d080 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6622429740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181afbd0 con 0x55992b12c6d0 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43562/0 (socket says 172.21.15.119:43562) 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 -- 172.21.15.119:0/2051289563 learned_addr learned my addr 172.21.15.119:0/2051289563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f6617e00640 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b134fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661d600640 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 -- 172.21.15.119:0/2051289563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 msgr2=0x55992b134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b134fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 -- 172.21.15.119:0/2051289563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 msgr2=0x55992b135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 -- 172.21.15.119:0/2051289563 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55992b133180 con 0x55992b12c6d0 2024-03-28T18:53:34.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.152+0000 7f661cc00640 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f660c009230 tx=0x7f660c048630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9327bb4f5e5835f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6609200640 1 -- 172.21.15.119:0/2051289563 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c048780 con 0x55992b12c6d0 2024-03-28T18:53:34.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6609200640 1 -- 172.21.15.119:0/2051289563 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f660c004460 con 0x55992b12c6d0 2024-03-28T18:53:34.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6609200640 1 -- 172.21.15.119:0/2051289563 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c0035f0 con 0x55992b12c6d0 2024-03-28T18:53:34.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 -- 172.21.15.119:0/2051289563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 msgr2=0x55992b125d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f660c009230 tx=0x7f660c048630 comp rx=0 tx=0).stop 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 -- 172.21.15.119:0/2051289563 shutdown_connections 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b135c00 0x55992b135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b134fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 --2- 172.21.15.119:0/2051289563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b12c6d0 0x55992b125d30 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 -- 172.21.15.119:0/2051289563 >> 172.21.15.119:0/2051289563 conn(0x55992b126da0 msgr2=0x55992b125620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.153+0000 7f6622429740 1 -- 172.21.15.119:0/2051289563 shutdown_connections 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 -- 172.21.15.119:0/2051289563 wait complete. 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 -- start start 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 Processor -- start 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b135c00 0x55992b1a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181b0890 con 0x55992b12c6d0 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181b0230 con 0x55992b12d080 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.154+0000 7f6622429740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66181ad750 con 0x55992b135c00 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6622429740 10 client.0 _ll_register_callbacks cb 0x7f660c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f661cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f661cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54450/0 (socket says 172.21.15.119:54450) 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f661cc00640 1 -- 172.21.15.119:0/3982004556 learned_addr learned my addr 172.21.15.119:0/3982004556 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f661d600640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b135c00 0x55992b1a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 -- 172.21.15.119:0/3982004556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b135c00 msgr2=0x55992b1a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b135c00 0x55992b1a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 -- 172.21.15.119:0/3982004556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 msgr2=0x55992b13ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 -- 172.21.15.119:0/3982004556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55992b1a06f0 con 0x55992b12d080 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f661cc00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f6617e00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b13bfd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f66040cc8a0 tx=0x7f66040d2cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7511a0c7dc9bcef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f65fe000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.155+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660409b2f0 con 0x55992b12d080 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f6604003b30 con 0x55992b12d080 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f660c004460 con 0x55992b12d080 2024-03-28T18:53:34.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f660409b110 con 0x55992b12d080 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 10 client.28163 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55992b1a1a80 con 0x55992b12d080 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f660c004460 con 0x55992b12d080 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6604097e20 con 0x55992b12d080 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 10 client.28163 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 10 client.28163 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 20 client.28163 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b1a2110 0x55992b1a4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 4 client.28163 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55992b097b50 con 0x55992b1a2110 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.156+0000 7f661cc00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b1a2110 0x55992b1a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f661cc00640 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b1a2110 0x55992b1a4530 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64b99931970c4c5f server_cookie=87018dc452009cae in_seq=0 out_seq=0 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f65fe000640 10 client.28163 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:34.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f65fe000640 1 -- 172.21.15.119:0/3982004556 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55992b097b50 con 0x55992b1a2110 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f65fe000640 10 client.28163 handle_command_reply: tid=0 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 client.28163 shutdown 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 msgr2=0x55992b13bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b13bfd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f66040cc8a0 tx=0x7f66040d2cf0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 shutdown_connections 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55992b135c00 0x55992b1a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55992b12d080 0x55992b13bfd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x55992b1a2110 0x55992b1a4530 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 --2- 172.21.15.119:0/3982004556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55992b12c6d0 0x55992b13ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 >> 172.21.15.119:0/3982004556 conn(0x55992b126da0 msgr2=0x55992b1288f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f65fe000640 0 client.28163 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 shutdown_connections 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.157+0000 7f6622429740 1 -- 172.21.15.119:0/3982004556 wait complete. 2024-03-28T18:53:34.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f6622429740 20 client.28163 trim_cache size 0 max 16384 2024-03-28T18:53:34.167 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:34.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 msgr2=0x7f66000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 0x7f66000a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f660c006010 tx=0x7f660c000bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 msgr2=0x7f661819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f660400ea60 tx=0x7f660400ef20 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 shutdown_connections 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f66000a2970 0x7f66000a4e30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f661810e040 0x7f661819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6618100e40 0x7f661819a7f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 --2- 172.21.15.119:0/3254805377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6618100490 0x7f661819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 >> 172.21.15.119:0/3254805377 conn(0x7f6618077700 msgr2=0x7f661810f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 shutdown_connections 2024-03-28T18:53:34.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.158+0000 7f661f600640 1 -- 172.21.15.119:0/3254805377 wait complete. 2024-03-28T18:53:34.176 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:34.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:53:34.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 -- 172.21.15.119:0/1148700247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 msgr2=0x7f47a0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 --2- 172.21.15.119:0/1148700247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a0109c30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4794009930 tx=0x7f479402bc20 comp rx=0 tx=0).stop 2024-03-28T18:53:34.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 -- 172.21.15.119:0/1148700247 shutdown_connections 2024-03-28T18:53:34.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 --2- 172.21.15.119:0/1148700247 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 --2- 172.21.15.119:0/1148700247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 0x7f47a0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 --2- 172.21.15.119:0/1148700247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a0109c30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 -- 172.21.15.119:0/1148700247 >> 172.21.15.119:0/1148700247 conn(0x7f47a00ff2f0 msgr2=0x7f47a0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 -- 172.21.15.119:0/1148700247 shutdown_connections 2024-03-28T18:53:34.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.379+0000 7f47a7400640 1 -- 172.21.15.119:0/1148700247 wait complete. 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.380+0000 7f47a7400640 1 Processor -- start 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.380+0000 7f47a7400640 1 -- start start 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 0x7f47a019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0113c70 con 0x7f47a0103840 2024-03-28T18:53:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0113af0 con 0x7f47a0104200 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a7400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0113df0 con 0x7f47a0109850 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 0x7f47a019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38042/0 (socket says 172.21.15.119:38042) 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 -- 172.21.15.119:0/4140972769 learned_addr learned my addr 172.21.15.119:0/4140972769 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f47a5400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 -- 172.21.15.119:0/4140972769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 msgr2=0x7f47a019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 -- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 msgr2=0x7f47a019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 0x7f47a019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.381+0000 7f479fe00640 1 -- 172.21.15.119:0/4140972769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a019f1b0 con 0x7f47a0104200 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f479fe00640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f478c00e9f0 tx=0x7f478c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f479d600640 1 -- 172.21.15.119:0/4140972769 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c00cd80 con 0x7f47a0104200 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f479d600640 1 -- 172.21.15.119:0/4140972769 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f478c010910 con 0x7f47a0104200 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a019f4a0 con 0x7f47a0104200 2024-03-28T18:53:34.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f479d600640 1 -- 172.21.15.119:0/4140972769 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c018990 con 0x7f47a0104200 2024-03-28T18:53:34.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.382+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a01a6d30 con 0x7f47a0104200 2024-03-28T18:53:34.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.385+0000 7f479d600640 1 -- 172.21.15.119:0/4140972769 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f478c010ac0 con 0x7f47a0104200 2024-03-28T18:53:34.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.385+0000 7f47aa224740 1 Processor -- start 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 -- start start 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5348080 0x5560d5350630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5350c00 0x5560d5353050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d52ab2c0 con 0x5560d5348080 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d5253c90 con 0x5560d5350c00 2024-03-28T18:53:34.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47aa224740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d52a7360 con 0x5560d53476d0 2024-03-28T18:53:34.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47a5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47a5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43608/0 (socket says 172.21.15.119:43608) 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5348080 0x5560d5350630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f47a5400640 1 -- 172.21.15.119:0/3595132201 learned_addr learned my addr 172.21.15.119:0/3595132201 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.386+0000 7f479fe00640 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5350c00 0x5560d5353050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 -- 172.21.15.119:0/3595132201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5350c00 msgr2=0x5560d5353050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5350c00 0x5560d5353050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 -- 172.21.15.119:0/3595132201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5348080 msgr2=0x5560d5350630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5348080 0x5560d5350630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 -- 172.21.15.119:0/3595132201 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560d5278080 con 0x5560d53476d0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a5400640 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4798009f50 tx=0x7f47980118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9133e56d879552e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f479d600640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 0x7f47880a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f4791c00640 1 -- 172.21.15.119:0/3595132201 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798004d20 con 0x5560d53476d0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f479d600640 1 -- 172.21.15.119:0/4140972769 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f478c0cf190 con 0x7f47a0104200 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f4791c00640 1 -- 172.21.15.119:0/3595132201 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4798004050 con 0x5560d53476d0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f4791c00640 1 -- 172.21.15.119:0/3595132201 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798005220 con 0x5560d53476d0 2024-03-28T18:53:34.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.387+0000 7f47a4a00640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 0x7f47880a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47a4a00640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 0x7f47880a4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4794009900 tx=0x7f4794005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 -- 172.21.15.119:0/3595132201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 msgr2=0x5560d5347ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4798009f50 tx=0x7f47980118d0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 -- 172.21.15.119:0/3595132201 shutdown_connections 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5350c00 0x5560d5353050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5348080 0x5560d5350630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 --2- 172.21.15.119:0/3595132201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5347ab0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 -- 172.21.15.119:0/3595132201 >> 172.21.15.119:0/3595132201 conn(0x5560d5341da0 msgr2=0x5560d5342180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 -- 172.21.15.119:0/3595132201 shutdown_connections 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.388+0000 7f47aa224740 1 -- 172.21.15.119:0/3595132201 wait complete. 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 -- start start 2024-03-28T18:53:34.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 Processor -- start 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5350440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d534d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d534dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d5353b70 con 0x5560d5350c00 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d52670d0 con 0x5560d5348080 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560d5353cf0 con 0x5560d53476d0 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47aa224740 10 client.0 _ll_register_callbacks cb 0x5560d526be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.389+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5350440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d534d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5350440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43622/0 (socket says 172.21.15.119:43622) 2024-03-28T18:53:34.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f47a4a00640 1 -- 172.21.15.119:0/3251914607 learned_addr learned my addr 172.21.15.119:0/3251914607 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f47a5400640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d534dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 -- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 msgr2=0x5560d5350440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5350440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 -- 172.21.15.119:0/3251914607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 msgr2=0x5560d534dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d534dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 -- 172.21.15.119:0/3251914607 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560d4fbf3f0 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f479fe00640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d534d7d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f478c0cc850 tx=0x7f478c0974b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9f1bf9474ece8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f4786000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c09b5e0 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f478c018b30 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.390+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4798004050 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f478c09b170 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 10 client.28175 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5560d50897d0 con 0x5560d5348080 2024-03-28T18:53:34.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4798004050 con 0x5560d5348080 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f478c0afbd0 con 0x5560d5348080 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 10 client.28175 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 10 client.28175 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 20 client.28175 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d5360b50 0x5560d5362f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 4 client.28175 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560d52b8800 con 0x5560d5360b50 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.391+0000 7f47a4a00640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d5360b50 0x5560d5362f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.392+0000 7f47a4a00640 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d5360b50 0x5560d5362f40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e4d187db5d3fc58 server_cookie=b3b47375c94187a2 in_seq=0 out_seq=0 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.392+0000 7f4786000640 10 client.28175 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f4786000640 1 -- 172.21.15.119:0/3251914607 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5560d52b8800 con 0x5560d5360b50 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f4786000640 10 client.28175 handle_command_reply: tid=0 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 client.28175 shutdown 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 msgr2=0x5560d534d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d534d7d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f478c0cc850 tx=0x7f478c0974b0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 shutdown_connections 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d5360b50 0x5560d5362f40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d534dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f4786000640 0 client.28175 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d534d7d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 --2- 172.21.15.119:0/3251914607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5350440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 >> 172.21.15.119:0/3251914607 conn(0x5560d5341da0 msgr2=0x5560d5343af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 shutdown_connections 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.393+0000 7f47aa224740 1 -- 172.21.15.119:0/3251914607 wait complete. 2024-03-28T18:53:34.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.394+0000 7f47aa224740 20 client.28175 trim_cache size 0 max 16384 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 Processor -- start 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 -- start start 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d5348080 0x5560d5350440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d535c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01af800 con 0x5560d5350c00 2024-03-28T18:53:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01b1180 con 0x5560d53476d0 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.407+0000 7f47aa224740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01afe60 con 0x5560d5348080 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38080/0 (socket says 172.21.15.119:38080) 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 -- 172.21.15.119:0/608349272 learned_addr learned my addr 172.21.15.119:0/608349272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f479fe00640 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d5348080 0x5560d5350440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a5400640 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d535c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 -- 172.21.15.119:0/608349272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d5348080 msgr2=0x5560d5350440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d5348080 0x5560d5350440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 -- 172.21.15.119:0/608349272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 msgr2=0x5560d535c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d535c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 -- 172.21.15.119:0/608349272 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560d53472b0 con 0x5560d53476d0 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f47a4a00640 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f479402c230 tx=0x7f4794042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2cbc3b0f40def59 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f4791200640 1 -- 172.21.15.119:0/608349272 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794046070 con 0x5560d53476d0 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.408+0000 7f4791200640 1 -- 172.21.15.119:0/608349272 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4794058110 con 0x5560d53476d0 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f4791200640 1 -- 172.21.15.119:0/608349272 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940586f0 con 0x5560d53476d0 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 -- 172.21.15.119:0/608349272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 msgr2=0x5560d534ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f479402c230 tx=0x7f4794042000 comp rx=0 tx=0).stop 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 -- 172.21.15.119:0/608349272 shutdown_connections 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d535c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d5348080 0x5560d5350440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 --2- 172.21.15.119:0/608349272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d53476d0 0x5560d534ff00 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 -- 172.21.15.119:0/608349272 >> 172.21.15.119:0/608349272 conn(0x5560d5341da0 msgr2=0x5560d5348cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 -- 172.21.15.119:0/608349272 shutdown_connections 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.409+0000 7f47aa224740 1 -- 172.21.15.119:0/608349272 wait complete. 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 -- start start 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 Processor -- start 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d5354f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d53554b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01b0b20 con 0x5560d5350c00 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01b04c0 con 0x5560d5348080 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01ad9e0 con 0x5560d53476d0 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.410+0000 7f47aa224740 10 client.0 _ll_register_callbacks cb 0x7f478c0d29d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43638/0 (socket says 172.21.15.119:43638) 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 -- 172.21.15.119:0/1833531682 learned_addr learned my addr 172.21.15.119:0/1833531682 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a5400640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d53554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f479fe00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d5354f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 -- 172.21.15.119:0/1833531682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 msgr2=0x5560d5354f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d5354f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 -- 172.21.15.119:0/1833531682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 msgr2=0x5560d53554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d53554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 -- 172.21.15.119:0/1833531682 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560d5355bf0 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f47a4a00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4794009930 tx=0x7f4794004c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54ae2ea39c65e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f4786000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.411+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479402ca70 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4794058110 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f479402cc10 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47940586f0 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 10 client.28156 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5560d53bd040 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4794058110 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f47940053e0 con 0x5560d53476d0 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 10 client.28156 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 10 client.28156 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 20 client.28156 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d53bd660 0x5560d53bfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 4 client.28156 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.412+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5560d52b2b50 con 0x5560d53bd660 2024-03-28T18:53:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f479fe00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d53bd660 0x5560d53bfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f479fe00640 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d53bd660 0x5560d53bfa50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d468de57101c41c server_cookie=658b0fab8338fa7a in_seq=0 out_seq=0 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f4786000640 10 client.28156 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f4786000640 1 -- 172.21.15.119:0/1833531682 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5560d52b2b50 con 0x5560d53bd660 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f4786000640 10 client.28156 handle_command_reply: tid=0 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 client.28156 shutdown 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 msgr2=0x5560d5358bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4794009930 tx=0x7f4794004c00 comp rx=0 tx=0).stop 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 shutdown_connections 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x5560d53bd660 0x5560d53bfa50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5560d5350c00 0x5560d53554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5560d5348080 0x5560d5354f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 --2- 172.21.15.119:0/1833531682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5560d53476d0 0x5560d5358bf0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f4786000640 0 client.28156 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:53:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 >> 172.21.15.119:0/1833531682 conn(0x5560d5341da0 msgr2=0x5560d5340620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.413+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 shutdown_connections 2024-03-28T18:53:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47aa224740 1 -- 172.21.15.119:0/1833531682 wait complete. 2024-03-28T18:53:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47aa224740 20 client.28156 trim_cache size 0 max 16384 2024-03-28T18:53:34.418 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 msgr2=0x7f47880a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 0x7f47880a4eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4794009900 tx=0x7f4794005a20 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 msgr2=0x7f47a019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f478c00e9f0 tx=0x7f478c00eeb0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 shutdown_connections 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f47880a29f0 0x7f47880a4eb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47a0109850 0x7f47a019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47a0104200 0x7f47a019a750 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 --2- 172.21.15.119:0/4140972769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47a0103840 0x7f47a019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 >> 172.21.15.119:0/4140972769 conn(0x7f47a00ff2f0 msgr2=0x7f47a00ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 shutdown_connections 2024-03-28T18:53:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.414+0000 7f47a7400640 1 -- 172.21.15.119:0/4140972769 wait complete. 2024-03-28T18:53:34.431 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:34.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 -- 172.21.15.119:0/1617336862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa8107eb0 msgr2=0x7f5aa8108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 --2- 172.21.15.119:0/1617336862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa8107eb0 0x7f5aa8108290 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009900 tx=0x7f5a9c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 -- 172.21.15.119:0/1617336862 shutdown_connections 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 --2- 172.21.15.119:0/1617336862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa810cf80 0x7f5aa8114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 --2- 172.21.15.119:0/1617336862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa8108860 0x7f5aa810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 --2- 172.21.15.119:0/1617336862 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa8107eb0 0x7f5aa8108290 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 -- 172.21.15.119:0/1617336862 >> 172.21.15.119:0/1617336862 conn(0x7f5aa8075be0 msgr2=0x7f5aa8078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 -- 172.21.15.119:0/1617336862 shutdown_connections 2024-03-28T18:53:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.635+0000 7f5ab0200640 1 -- 172.21.15.119:0/1617336862 wait complete. 2024-03-28T18:53:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 Processor -- start 2024-03-28T18:53:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 -- start start 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa810cf80 0x7f5aa819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa81a2e30 0x7f5aa81a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8118080 con 0x7f5aa810cf80 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8117f00 con 0x7f5aa81a2e30 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.636+0000 7f5ab0200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8118200 con 0x7f5aa8107eb0 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43656/0 (socket says 172.21.15.119:43656) 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aae200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa81a2e30 0x7f5aa81a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 -- 172.21.15.119:0/3097955683 learned_addr learned my addr 172.21.15.119:0/3097955683 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aace00640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa810cf80 0x7f5aa819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 -- 172.21.15.119:0/3097955683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa81a2e30 msgr2=0x7f5aa81a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa81a2e30 0x7f5aa81a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 -- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa810cf80 msgr2=0x7f5aa819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa810cf80 0x7f5aa819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 -- 172.21.15.119:0/3097955683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa81a3920 con 0x7f5aa8107eb0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.637+0000 7f5aad800640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c02c240 tx=0x7f5a9c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.638+0000 7f5aa6000640 1 -- 172.21.15.119:0/3097955683 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c043020 con 0x7f5aa8107eb0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.638+0000 7f5aa6000640 1 -- 172.21.15.119:0/3097955683 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a9c0054e0 con 0x7f5aa8107eb0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.638+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa81a3bb0 con 0x7f5aa8107eb0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.638+0000 7f5aa6000640 1 -- 172.21.15.119:0/3097955683 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c003ab0 con 0x7f5aa8107eb0 2024-03-28T18:53:34.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.638+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa81ab400 con 0x7f5aa8107eb0 2024-03-28T18:53:34.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5aa6000640 1 -- 172.21.15.119:0/3097955683 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5a9c046020 con 0x7f5aa8107eb0 2024-03-28T18:53:34.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 Processor -- start 2024-03-28T18:53:34.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 -- start start 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c75050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86bcd2c0 con 0x55aa86c6a080 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86b75c90 con 0x55aa86c696d0 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.641+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86bc9360 con 0x55aa86c72c00 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38110/0 (socket says 172.21.15.119:38110) 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 -- 172.21.15.119:0/2021876538 learned_addr learned my addr 172.21.15.119:0/2021876538 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aace00640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c75050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aad800640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 -- 172.21.15.119:0/2021876538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 msgr2=0x55aa86c75050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c75050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 -- 172.21.15.119:0/2021876538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 msgr2=0x55aa86c72630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 -- 172.21.15.119:0/2021876538 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa86b9a080 con 0x55aa86c696d0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aad800640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aace00640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c75050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aa6000640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 0x7f5a900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.642+0000 7f5aae200640 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009f60 tx=0x7f5aa0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d3012eb4994bca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5aa6000640 1 -- 172.21.15.119:0/3097955683 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5a9c0f0cd0 con 0x7f5aa8107eb0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5a9aa00640 1 -- 172.21.15.119:0/2021876538 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa001e070 con 0x55aa86c696d0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5a9aa00640 1 -- 172.21.15.119:0/2021876538 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa0004030 con 0x55aa86c696d0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5a9aa00640 1 -- 172.21.15.119:0/2021876538 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa000f720 con 0x55aa86c696d0 2024-03-28T18:53:34.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5aad800640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 0x7f5a900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5aad800640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 0x7f5a900a4eb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55aa86c691a0 tx=0x7f5a9c0dd040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/2021876538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 msgr2=0x55aa86c69ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009f60 tx=0x7f5aa0009f90 comp rx=0 tx=0).stop 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/2021876538 shutdown_connections 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c75050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/2021876538 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c696d0 0x55aa86c69ab0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/2021876538 >> 172.21.15.119:0/2021876538 conn(0x55aa86c63da0 msgr2=0x55aa86c64180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/2021876538 shutdown_connections 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.643+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/2021876538 wait complete. 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.644+0000 7f5ab2e7d740 1 -- start start 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.644+0000 7f5ab2e7d740 1 Processor -- start 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.644+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 0x55aa86c72520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.644+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c6fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86c75b70 con 0x55aa86c696d0 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86b890d0 con 0x55aa86c6a080 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa86c75cf0 con 0x55aa86c72c00 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5ab2e7d740 10 client.0 _ll_register_callbacks cb 0x55aa86b8de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aae200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 0x55aa86c72520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38112/0 (socket says 172.21.15.119:38112) 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 -- 172.21.15.119:0/3532247873 learned_addr learned my addr 172.21.15.119:0/3532247873 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aae200640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c6fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 -- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 msgr2=0x55aa86c6fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c6fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 -- 172.21.15.119:0/3532247873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 msgr2=0x55aa86c72520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 0x55aa86c72520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 -- 172.21.15.119:0/3532247873 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa868e13f0 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aae200640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 0x55aa86c72520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5aace00640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5a94012700 tx=0x7f5a94012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c6150d14091949 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.645+0000 7f5a8f400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94004030 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5a940041d0 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5aa0004030 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a94004410 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5ab2e7d740 10 client.28193 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55aa869ab7d0 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa0004030 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.646+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5a94022030 con 0x55aa86c6a080 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 10 client.28193 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 10 client.28193 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 20 client.28193 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86c82b30 0x55aa86c84f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 4 client.28193 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:34.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa86bda800 con 0x55aa86c82b30 2024-03-28T18:53:34.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5aae200640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86c82b30 0x55aa86c84f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5aae200640 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86c82b30 0x55aa86c84f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fc24c00fb10da4b server_cookie=182aac6ad290f25f in_seq=0 out_seq=0 2024-03-28T18:53:34.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.647+0000 7f5a8f400640 10 client.28193 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.648+0000 7f5a8f400640 1 -- 172.21.15.119:0/3532247873 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aa86bda800 con 0x55aa86c82b30 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.648+0000 7f5a8f400640 10 client.28193 handle_command_reply: tid=0 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.648+0000 7f5ab2e7d740 1 client.28193 shutdown 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 msgr2=0x55aa86c6f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5a94012700 tx=0x7f5a94012bc0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 shutdown_connections 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86c82b30 0x55aa86c84f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c72c00 0x55aa86c6fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c6a080 0x55aa86c6f8b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/3532247873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c696d0 0x55aa86c72520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5a8f400640 0 client.28193 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 >> 172.21.15.119:0/3532247873 conn(0x55aa86c63da0 msgr2=0x55aa86c65af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 shutdown_connections 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/3532247873 wait complete. 2024-03-28T18:53:34.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.649+0000 7f5ab2e7d740 20 client.28193 trim_cache size 0 max 16384 2024-03-28T18:53:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.662+0000 7f5ab2e7d740 1 Processor -- start 2024-03-28T18:53:34.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.662+0000 7f5ab2e7d740 1 -- start start 2024-03-28T18:53:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c71fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c6cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b3990 con 0x55aa86c6a080 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b5310 con 0x55aa86c72c00 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b3ff0 con 0x55aa86c696d0 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5aace00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5aace00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54532/0 (socket says 172.21.15.119:54532) 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5aace00640 1 -- 172.21.15.119:0/179576762 learned_addr learned my addr 172.21.15.119:0/179576762 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.663+0000 7f5aae200640 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c71fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 -- 172.21.15.119:0/179576762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 msgr2=0x55aa86c71fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c71fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 -- 172.21.15.119:0/179576762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 msgr2=0x55aa86c6cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c6cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 -- 172.21.15.119:0/179576762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa86c6de00 con 0x55aa86c6a080 2024-03-28T18:53:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aae200640 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c71fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5aace00640 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5a94012420 tx=0x7f5a940096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55610ac9d693da91 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5a9a000640 1 -- 172.21.15.119:0/179576762 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94010430 con 0x55aa86c6a080 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5a9a000640 1 -- 172.21.15.119:0/179576762 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a940035f0 con 0x55aa86c6a080 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.664+0000 7f5a9a000640 1 -- 172.21.15.119:0/179576762 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9400fcc0 con 0x55aa86c6a080 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/179576762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 msgr2=0x55aa86c72520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5a94012420 tx=0x7f5a940096c0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/179576762 shutdown_connections 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c6cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c72520 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/179576762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c71fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/179576762 >> 172.21.15.119:0/179576762 conn(0x55aa86c63da0 msgr2=0x55aa86c65a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/179576762 shutdown_connections 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/179576762 wait complete. 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.665+0000 7f5ab2e7d740 1 -- start start 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 Processor -- start 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c7acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c795d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b4cb0 con 0x55aa86c6a080 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b4650 con 0x55aa86c72c00 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81b1b70 con 0x55aa86c696d0 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5ab2e7d740 10 client.0 _ll_register_callbacks cb 0x7f5a94009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5aae200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c7acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5aace00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5aace00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54542/0 (socket says 172.21.15.119:54542) 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5aace00640 1 -- 172.21.15.119:0/1189056494 learned_addr learned my addr 172.21.15.119:0/1189056494 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.666+0000 7f5aae200640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c795d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 -- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 msgr2=0x55aa86c7acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c7acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 -- 172.21.15.119:0/1189056494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 msgr2=0x55aa86c795d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c795d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 -- 172.21.15.119:0/1189056494 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa86c79e00 con 0x55aa86c6a080 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aae200640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5aace00640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5a940126a0 tx=0x7f5a94003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7bf43055cd3a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5a8f400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a94031070 con 0x55aa86c6a080 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5a9400f9c0 con 0x55aa86c6a080 2024-03-28T18:53:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.667+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a940035f0 con 0x55aa86c6a080 2024-03-28T18:53:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a9400fbc0 con 0x55aa86c6a080 2024-03-28T18:53:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 10 client.17958 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55aa86cdf4f0 con 0x55aa86c6a080 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a940035f0 con 0x55aa86c6a080 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5a9403b030 con 0x55aa86c6a080 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 10 client.17958 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 10 client.17958 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 20 client.17958 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86cdfaa0 0x55aa86ce1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 4 client.17958 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55aa86bd4b50 con 0x55aa86cdfaa0 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.668+0000 7f5aae200640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86cdfaa0 0x55aa86ce1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5aae200640 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86cdfaa0 0x55aa86ce1ec0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea1c6bf1377748f0 server_cookie=b1f90145dda707d in_seq=0 out_seq=0 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5a8f400640 10 client.17958 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5a8f400640 1 -- 172.21.15.119:0/1189056494 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55aa86bd4b50 con 0x55aa86cdfaa0 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5a8f400640 10 client.17958 handle_command_reply: tid=0 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 client.17958 shutdown 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 msgr2=0x55aa86c79070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5a940126a0 tx=0x7f5a94003e90 comp rx=0 tx=0).stop 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 shutdown_connections 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55aa86cdfaa0 0x55aa86ce1ec0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55aa86c72c00 0x55aa86c795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55aa86c6a080 0x55aa86c79070 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5a8f400640 0 client.17958 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 --2- 172.21.15.119:0/1189056494 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa86c696d0 0x55aa86c7acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.669+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 >> 172.21.15.119:0/1189056494 conn(0x55aa86c63da0 msgr2=0x55aa86c651a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 shutdown_connections 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab2e7d740 1 -- 172.21.15.119:0/1189056494 wait complete. 2024-03-28T18:53:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab2e7d740 20 client.17958 trim_cache size 0 max 16384 2024-03-28T18:53:34.674 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 msgr2=0x7f5a900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 0x7f5a900a4eb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55aa86c691a0 tx=0x7f5a9c0dd040 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 msgr2=0x7f5aa819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c02c240 tx=0x7f5a9c0042f0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 shutdown_connections 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a900a29f0 0x7f5a900a4eb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5aa81a2e30 0x7f5aa81a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa810cf80 0x7f5aa819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 --2- 172.21.15.119:0/3097955683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5aa8107eb0 0x7f5aa819e5c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 >> 172.21.15.119:0/3097955683 conn(0x7f5aa8075be0 msgr2=0x7f5aa8077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.670+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 shutdown_connections 2024-03-28T18:53:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.671+0000 7f5ab0200640 1 -- 172.21.15.119:0/3097955683 wait complete. 2024-03-28T18:53:34.685 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:34 smithi119 ceph-mon[31813]: pgmap v294: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:34.685 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:34 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3162028756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:34.687 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:53:34.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:53:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 -- 172.21.15.119:0/2958118880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580103840 msgr2=0x7f4580103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 --2- 172.21.15.119:0/2958118880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580103840 0x7f4580103cc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4568009900 tx=0x7f456802bc40 comp rx=0 tx=0).stop 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 -- 172.21.15.119:0/2958118880 shutdown_connections 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 --2- 172.21.15.119:0/2958118880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580104200 0x7f4580110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 --2- 172.21.15.119:0/2958118880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580103840 0x7f4580103cc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 --2- 172.21.15.119:0/2958118880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580109850 0x7f4580109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 -- 172.21.15.119:0/2958118880 >> 172.21.15.119:0/2958118880 conn(0x7f45800ff350 msgr2=0x7f4580101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 -- 172.21.15.119:0/2958118880 shutdown_connections 2024-03-28T18:53:34.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.894+0000 7f4586600640 1 -- 172.21.15.119:0/2958118880 wait complete. 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 Processor -- start 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 -- start start 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580103840 0x7f458019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 0x7f458019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580113cb0 con 0x7f4580103840 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580113b30 con 0x7f4580104200 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f4586600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580113e30 con 0x7f4580109850 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f457ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f457ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38164/0 (socket says 172.21.15.119:38164) 2024-03-28T18:53:34.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.896+0000 7f457ea00640 1 -- 172.21.15.119:0/1793441593 learned_addr learned my addr 172.21.15.119:0/1793441593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457f400640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580103840 0x7f458019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457fe00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 0x7f458019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 -- 172.21.15.119:0/1793441593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 msgr2=0x7f458019ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 0x7f458019ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 -- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580103840 msgr2=0x7f458019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580103840 0x7f458019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 -- 172.21.15.119:0/1793441593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458019f150 con 0x7f4580104200 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457fe00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 0x7f458019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f457ea00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4568005ab0 tx=0x7f45680042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.897+0000 7f4577e00640 1 -- 172.21.15.119:0/1793441593 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568041070 con 0x7f4580104200 2024-03-28T18:53:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.898+0000 7f4577e00640 1 -- 172.21.15.119:0/1793441593 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f456802ccd0 con 0x7f4580104200 2024-03-28T18:53:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.898+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458019f3e0 con 0x7f4580104200 2024-03-28T18:53:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.898+0000 7f4577e00640 1 -- 172.21.15.119:0/1793441593 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003d10 con 0x7f4580104200 2024-03-28T18:53:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.898+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45801a6cd0 con 0x7f4580104200 2024-03-28T18:53:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 Processor -- start 2024-03-28T18:53:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 -- start start 2024-03-28T18:53:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e29080 0x55ef86e31630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 0x55ef86e34050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f4577e00640 1 -- 172.21.15.119:0/1793441593 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4568044070 con 0x7f4580104200 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86d8c2c0 con 0x55ef86e29080 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86d34c90 con 0x55ef86e31c00 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.902+0000 7f45893d2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86d88360 con 0x55ef86e286d0 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43708/0 (socket says 172.21.15.119:43708) 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 -- 172.21.15.119:0/1493298813 learned_addr learned my addr 172.21.15.119:0/1493298813 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457f400640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e29080 0x55ef86e31630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457ea00640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 0x55ef86e34050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 -- 172.21.15.119:0/1493298813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 msgr2=0x55ef86e34050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 0x55ef86e34050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 -- 172.21.15.119:0/1493298813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e29080 msgr2=0x55ef86e31630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e29080 0x55ef86e31630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 -- 172.21.15.119:0/1493298813 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef86d59080 con 0x55ef86e286d0 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457ea00640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 0x55ef86e34050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.903+0000 7f457fe00640 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4578011540 tx=0x7f4578011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23c309a5247e02d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f4574c00640 1 -- 172.21.15.119:0/1493298813 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457800f5c0 con 0x55ef86e286d0 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f4574c00640 1 -- 172.21.15.119:0/1493298813 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f457800fc40 con 0x55ef86e286d0 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f4574c00640 1 -- 172.21.15.119:0/1493298813 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578019d20 con 0x55ef86e286d0 2024-03-28T18:53:34.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f4577e00640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 0x7f45640a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f4577e00640 1 -- 172.21.15.119:0/1793441593 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f45680f0810 con 0x7f4580104200 2024-03-28T18:53:34.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 -- 172.21.15.119:0/1493298813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 msgr2=0x55ef86e28ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4578011540 tx=0x7f4578011a00 comp rx=0 tx=0).stop 2024-03-28T18:53:34.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f457f400640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 0x7f45640a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 -- 172.21.15.119:0/1493298813 shutdown_connections 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e31c00 0x55ef86e34050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e29080 0x55ef86e31630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 --2- 172.21.15.119:0/1493298813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e28ab0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 -- 172.21.15.119:0/1493298813 >> 172.21.15.119:0/1493298813 conn(0x55ef86e22da0 msgr2=0x55ef86e23180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.904+0000 7f45893d2740 1 -- 172.21.15.119:0/1493298813 shutdown_connections 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.905+0000 7f45893d2740 1 -- 172.21.15.119:0/1493298813 wait complete. 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.905+0000 7f45893d2740 1 -- start start 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.905+0000 7f457f400640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 0x7f45640a4eb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55ef86e23e10 tx=0x7f4570009680 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.905+0000 7f45893d2740 1 Processor -- start 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e2e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86e34b70 con 0x55ef86e31c00 2024-03-28T18:53:34.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86d480d0 con 0x55ef86e29080 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef86e34cf0 con 0x55ef86e286d0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e2e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f45893d2740 10 client.0 _ll_register_callbacks cb 0x55ef86d4ce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43712/0 (socket says 172.21.15.119:43712) 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457f400640 1 -- 172.21.15.119:0/2618029593 learned_addr learned my addr 172.21.15.119:0/2618029593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457fe00640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 -- 172.21.15.119:0/2618029593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 msgr2=0x55ef86e314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e314f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 -- 172.21.15.119:0/2618029593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 msgr2=0x55ef86e2ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 -- 172.21.15.119:0/2618029593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef86aa03f0 con 0x55ef86e29080 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.906+0000 7f457ea00640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e2e880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f45680edf80 tx=0x7f45680044e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b6c45cdd505fbec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456804b040 con 0x55ef86e29080 2024-03-28T18:53:34.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f45680ee0f0 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f457800fc40 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45680da050 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 10 client.28208 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ef86b6a7d0 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f457800fc40 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f45680c51a0 con 0x55ef86e29080 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 10 client.28208 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 10 client.28208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 20 client.28208 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e419f0 0x55ef86e43e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 4 client.28208 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.907+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef86d99800 con 0x55ef86e419f0 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.908+0000 7f457f400640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e419f0 0x55ef86e43e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.908+0000 7f457f400640 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e419f0 0x55ef86e43e10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=249ff924f2bc9c90 server_cookie=ba05e782cb2fc0df in_seq=0 out_seq=0 2024-03-28T18:53:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.908+0000 7f456d600640 10 client.28208 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f456d600640 1 -- 172.21.15.119:0/2618029593 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ef86d99800 con 0x55ef86e419f0 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f456d600640 10 client.28208 handle_command_reply: tid=0 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 client.28208 shutdown 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 msgr2=0x55ef86e2e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e2e880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f45680edf80 tx=0x7f45680044e0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 shutdown_connections 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e419f0 0x55ef86e43e10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e2e880 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 --2- 172.21.15.119:0/2618029593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f456d600640 0 client.28208 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 >> 172.21.15.119:0/2618029593 conn(0x55ef86e22da0 msgr2=0x55ef86e24af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 shutdown_connections 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 1 -- 172.21.15.119:0/2618029593 wait complete. 2024-03-28T18:53:34.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.909+0000 7f45893d2740 20 client.28208 trim_cache size 0 max 16384 2024-03-28T18:53:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.923+0000 7f45893d2740 1 Processor -- start 2024-03-28T18:53:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.923+0000 7f45893d2740 1 -- start start 2024-03-28T18:53:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801af600 con 0x55ef86e31c00 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801b0f80 con 0x55ef86e29080 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f45893d2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801afc60 con 0x55ef86e286d0 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43728/0 (socket says 172.21.15.119:43728) 2024-03-28T18:53:34.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 -- 172.21.15.119:0/3438689930 learned_addr learned my addr 172.21.15.119:0/3438689930 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457fe00640 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 -- 172.21.15.119:0/3438689930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 msgr2=0x55ef86e314f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e314f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 -- 172.21.15.119:0/3438689930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 msgr2=0x55ef86e2bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457f400640 1 -- 172.21.15.119:0/3438689930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef86e2d830 con 0x55ef86e286d0 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.924+0000 7f457ea00640 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f457f400640 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4570005fd0 tx=0x7f4570016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e0df646d510ec99 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f456fe00640 1 -- 172.21.15.119:0/3438689930 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457002b050 con 0x55ef86e286d0 2024-03-28T18:53:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f456fe00640 1 -- 172.21.15.119:0/3438689930 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4570031070 con 0x55ef86e286d0 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f456fe00640 1 -- 172.21.15.119:0/3438689930 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700040e0 con 0x55ef86e286d0 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 -- 172.21.15.119:0/3438689930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 msgr2=0x55ef86e30fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4570005fd0 tx=0x7f4570016000 comp rx=0 tx=0).stop 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 -- 172.21.15.119:0/3438689930 shutdown_connections 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e2bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e314f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 --2- 172.21.15.119:0/3438689930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e30fb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.925+0000 7f45893d2740 1 -- 172.21.15.119:0/3438689930 >> 172.21.15.119:0/3438689930 conn(0x55ef86e22da0 msgr2=0x55ef86e24980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.926+0000 7f45893d2740 1 -- 172.21.15.119:0/3438689930 shutdown_connections 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.926+0000 7f45893d2740 1 -- 172.21.15.119:0/3438689930 wait complete. 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.926+0000 7f45893d2740 1 -- start start 2024-03-28T18:53:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.926+0000 7f45893d2740 1 Processor -- start 2024-03-28T18:53:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e37d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801b0920 con 0x55ef86e31c00 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801b02c0 con 0x55ef86e29080 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45801ad7e0 con 0x55ef86e286d0 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f45893d2740 10 client.0 _ll_register_callbacks cb 0x7f4568005670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e37d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e382c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54624/0 (socket says 172.21.15.119:54624) 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457fe00640 1 -- 172.21.15.119:0/2474731003 learned_addr learned my addr 172.21.15.119:0/2474731003 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457f400640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 -- 172.21.15.119:0/2474731003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 msgr2=0x55ef86e399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 -- 172.21.15.119:0/2474731003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 msgr2=0x55ef86e382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 -- 172.21.15.119:0/2474731003 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef86e38800 con 0x55ef86e29080 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457f400640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f457ea00640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e37d80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f45680bced0 tx=0x7f45680bb450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b1e5a31bf085f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.927+0000 7f456d600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45680a0060 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f45680bc390 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4570031070 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f456804b070 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 10 client.28214 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ef86e9dfd0 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4570031070 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f45680b7ab0 con 0x55ef86e29080 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 10 client.28214 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 10 client.28214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 20 client.28214 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e9e560 0x55ef86ea0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 4 client.28214 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:53:34.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.928+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ef86d93b50 con 0x55ef86e9e560 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f457f400640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e9e560 0x55ef86ea0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f457f400640 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e9e560 0x55ef86ea0980 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4a99b34e37570c4 server_cookie=14a052b900aceb49 in_seq=0 out_seq=0 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f456d600640 10 client.28214 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f456d600640 1 -- 172.21.15.119:0/2474731003 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ef86d93b50 con 0x55ef86e9e560 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f456d600640 10 client.28214 handle_command_reply: tid=0 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.929+0000 7f45893d2740 1 client.28214 shutdown 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 msgr2=0x55ef86e37d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e37d80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f45680bced0 tx=0x7f45680bb450 comp rx=0 tx=0).stop 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 shutdown_connections 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55ef86e9e560 0x55ef86ea0980 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ef86e31c00 0x55ef86e382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ef86e29080 0x55ef86e37d80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f456d600640 0 client.28214 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 --2- 172.21.15.119:0/2474731003 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef86e286d0 0x55ef86e399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 >> 172.21.15.119:0/2474731003 conn(0x55ef86e22da0 msgr2=0x55ef86e240f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 shutdown_connections 2024-03-28T18:53:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 1 -- 172.21.15.119:0/2474731003 wait complete. 2024-03-28T18:53:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.930+0000 7f45893d2740 20 client.28214 trim_cache size 0 max 16384 2024-03-28T18:53:34.934 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 msgr2=0x7f45640a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 0x7f45640a4eb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55ef86e23e10 tx=0x7f4570009680 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 msgr2=0x7f458019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4568005ab0 tx=0x7f45680042f0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 shutdown_connections 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f45640a29f0 0x7f45640a4eb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4580109850 0x7f458019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4580104200 0x7f458019a6f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 --2- 172.21.15.119:0/1793441593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4580103840 0x7f458019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 >> 172.21.15.119:0/1793441593 conn(0x7f45800ff350 msgr2=0x7f45800ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 shutdown_connections 2024-03-28T18:53:34.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:34.931+0000 7f4586600640 1 -- 172.21.15.119:0/1793441593 wait complete. 2024-03-28T18:53:34.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:34 smithi196 ceph-mon[36300]: pgmap v294: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:34.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:34 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3162028756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:34 smithi131 ceph-mon[36065]: pgmap v294: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:34.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:34 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3162028756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:53:35.949 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-28T18:53:35.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:53:35.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-28T18:53:36.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.180+0000 7f43aca00640 1 -- 172.21.15.119:0/3806144593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8109850 msgr2=0x7f43a8109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.180+0000 7f43aca00640 1 --2- 172.21.15.119:0/3806144593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8109850 0x7f43a8109c30 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4398009900 tx=0x7f439802bc50 comp rx=0 tx=0).stop 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 -- 172.21.15.119:0/3806144593 shutdown_connections 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 --2- 172.21.15.119:0/3806144593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8104200 0x7f43a8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 --2- 172.21.15.119:0/3806144593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8103840 0x7f43a8103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 --2- 172.21.15.119:0/3806144593 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8109850 0x7f43a8109c30 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 -- 172.21.15.119:0/3806144593 >> 172.21.15.119:0/3806144593 conn(0x7f43a80ff2f0 msgr2=0x7f43a8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 -- 172.21.15.119:0/3806144593 shutdown_connections 2024-03-28T18:53:36.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.181+0000 7f43aca00640 1 -- 172.21.15.119:0/3806144593 wait complete. 2024-03-28T18:53:36.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 Processor -- start 2024-03-28T18:53:36.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 -- start start 2024-03-28T18:53:36.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 0x7f43a819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 0x7f43a819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8113c70 con 0x7f43a8109850 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8113af0 con 0x7f43a8103840 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43aca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8113df0 con 0x7f43a8104200 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.182+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 0x7f43a819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38222/0 (socket says 172.21.15.119:38222) 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 -- 172.21.15.119:0/3204533784 learned_addr learned my addr 172.21.15.119:0/3204533784 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a6800640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 0x7f43a819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 -- 172.21.15.119:0/3204533784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 msgr2=0x7f43a819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 0x7f43a819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 -- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 msgr2=0x7f43a819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 0x7f43a819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 -- 172.21.15.119:0/3204533784 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a819f170 con 0x7f43a8103840 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a6800640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 0x7f43a819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:36.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5400640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 0x7f43a819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4398000c00 tx=0x7f43980042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.183+0000 7f439ea00640 1 -- 172.21.15.119:0/3204533784 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398041070 con 0x7f43a8103840 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.184+0000 7f439ea00640 1 -- 172.21.15.119:0/3204533784 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f439802cce0 con 0x7f43a8103840 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.184+0000 7f439ea00640 1 -- 172.21.15.119:0/3204533784 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398003d10 con 0x7f43a8103840 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.184+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a819f400 con 0x7f43a8103840 2024-03-28T18:53:36.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.184+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a81a6cf0 con 0x7f43a8103840 2024-03-28T18:53:36.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 Processor -- start 2024-03-28T18:53:36.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 -- start start 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f439ea00640 1 -- 172.21.15.119:0/3204533784 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4398044070 con 0x7f43a8103840 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea129a0 0x557caea1b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cae976eb0 con 0x557caea12080 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cae91fa60 con 0x557caea1b580 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.187+0000 7f43af822740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cae972f50 con 0x557caea129a0 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54642/0 (socket says 172.21.15.119:54642) 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a6800640 1 -- 172.21.15.119:0/773339467 learned_addr learned my addr 172.21.15.119:0/773339467 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea129a0 0x557caea1b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38232/0 (socket says 172.21.15.119:38232) 2024-03-28T18:53:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 -- 172.21.15.119:0/773339467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea129a0 msgr2=0x557caea1b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea129a0 0x557caea1b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 -- 172.21.15.119:0/773339467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 msgr2=0x557caea12460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 -- 172.21.15.119:0/773339467 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cae7537e0 con 0x557caea1b580 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a6800640 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.188+0000 7f43a5400640 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4390009fa0 tx=0x7f43900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0298da2af914a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f4397000640 1 -- 172.21.15.119:0/773339467 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390004030 con 0x557caea1b580 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f4397000640 1 -- 172.21.15.119:0/773339467 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4390005500 con 0x557caea1b580 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f4397000640 1 -- 172.21.15.119:0/773339467 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390004e10 con 0x557caea1b580 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f439ea00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f439ea00640 1 -- 172.21.15.119:0/3204533784 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f43980f08d0 con 0x7f43a8103840 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f43af822740 1 -- 172.21.15.119:0/773339467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 msgr2=0x557caea1d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f43af822740 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4390009fa0 tx=0x7f43900118d0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.189+0000 7f43a6800640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 -- 172.21.15.119:0/773339467 shutdown_connections 2024-03-28T18:53:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea1b580 0x557caea1d970 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea129a0 0x557caea1b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 --2- 172.21.15.119:0/773339467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea12460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 -- 172.21.15.119:0/773339467 >> 172.21.15.119:0/773339467 conn(0x557caea0c7b0 msgr2=0x557caea0cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 -- 172.21.15.119:0/773339467 shutdown_connections 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 -- 172.21.15.119:0/773339467 wait complete. 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43a6800640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 0x7f438c0a4e30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009d90 tx=0x7f43a0009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 -- start start 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.190+0000 7f43af822740 1 Processor -- start 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557caea1e370 con 0x557caea12080 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cae932eb0 con 0x557caea129a0 2024-03-28T18:53:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x557caea1e4f0 con 0x557caea1b580 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43af822740 10 client.0 _ll_register_callbacks cb 0x557cae927150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43a5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43a5400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38246/0 (socket says 172.21.15.119:38246) 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43a5400640 1 -- 172.21.15.119:0/1473214038 learned_addr learned my addr 172.21.15.119:0/1473214038 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43a6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.191+0000 7f43a5e00640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 -- 172.21.15.119:0/1473214038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 msgr2=0x557caea18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea18760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 -- 172.21.15.119:0/1473214038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 msgr2=0x557caea1ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 -- 172.21.15.119:0/1473214038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cae7b3f70 con 0x557caea129a0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a6800640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5e00640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43a5400640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4390009350 tx=0x7f4390005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a79077eb5edc3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f438b400640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:36.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390027020 con 0x557caea129a0 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4390004b10 con 0x557caea129a0 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.192+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4390005500 con 0x557caea129a0 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4390032050 con 0x557caea129a0 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 10 client.28226 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557cae64b170 con 0x557caea129a0 2024-03-28T18:53:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4390005500 con 0x557caea129a0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f4390003960 con 0x557caea129a0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 10 client.28226 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 10 client.28226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 20 client.28226 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea2b5b0 0x557caea2d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 4 client.28226 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cae9843f0 con 0x557caea2b5b0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.193+0000 7f43a5e00640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea2b5b0 0x557caea2d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.194+0000 7f43a5e00640 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea2b5b0 0x557caea2d9a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba1c5e8dd098cb24 server_cookie=8cc09aa83ed3087b in_seq=0 out_seq=0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.194+0000 7f438b400640 10 client.28226 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f438b400640 1 -- 172.21.15.119:0/1473214038 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557cae9843f0 con 0x557caea2b5b0 2024-03-28T18:53:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f438b400640 10 client.28226 handle_command_reply: tid=0 2024-03-28T18:53:36.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 client.28226 shutdown 2024-03-28T18:53:36.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 msgr2=0x557caea18220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4390009350 tx=0x7f4390005d90 comp rx=0 tx=0).stop 2024-03-28T18:53:36.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 shutdown_connections 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea2b5b0 0x557caea2d9a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea18220 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 --2- 172.21.15.119:0/1473214038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f438b400640 0 client.28226 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 >> 172.21.15.119:0/1473214038 conn(0x557caea0c7b0 msgr2=0x557caea0e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 shutdown_connections 2024-03-28T18:53:36.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 1 -- 172.21.15.119:0/1473214038 wait complete. 2024-03-28T18:53:36.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.195+0000 7f43af822740 20 client.28226 trim_cache size 0 max 16384 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 Processor -- start 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 -- start start 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea1ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea26c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81af650 con 0x557caea12080 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81b0fd0 con 0x557caea129a0 2024-03-28T18:53:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43af822740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81afcb0 con 0x557caea1b580 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43a6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea26c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54648/0 (socket says 172.21.15.119:54648) 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43a5e00640 1 -- 172.21.15.119:0/3341481698 learned_addr learned my addr 172.21.15.119:0/3341481698 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.213+0000 7f43a5400640 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea1ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 -- 172.21.15.119:0/3341481698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 msgr2=0x557caea26c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea26c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 -- 172.21.15.119:0/3341481698 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 msgr2=0x557caea1ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea1ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 -- 172.21.15.119:0/3341481698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557caea2dd10 con 0x557caea12080 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ec0 tx=0x7f43980f6530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66de622c8886d820 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f4396600640 1 -- 172.21.15.119:0/3341481698 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439804b040 con 0x557caea12080 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f4396600640 1 -- 172.21.15.119:0/3341481698 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43980bd650 con 0x557caea12080 2024-03-28T18:53:36.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.214+0000 7f4396600640 1 -- 172.21.15.119:0/3341481698 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980bcf50 con 0x557caea12080 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- 172.21.15.119:0/3341481698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 msgr2=0x557caea1a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ec0 tx=0x7f43980f6530 comp rx=0 tx=0).stop 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- 172.21.15.119:0/3341481698 shutdown_connections 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea26c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea1ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 --2- 172.21.15.119:0/3341481698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea1a950 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- 172.21.15.119:0/3341481698 >> 172.21.15.119:0/3341481698 conn(0x557caea0c7b0 msgr2=0x557caea0b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- 172.21.15.119:0/3341481698 shutdown_connections 2024-03-28T18:53:36.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- 172.21.15.119:0/3341481698 wait complete. 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.215+0000 7f43af822740 1 -- start start 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 Processor -- start 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81b0970 con 0x557caea12080 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81b0310 con 0x557caea129a0 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81ad830 con 0x557caea1b580 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43af822740 10 client.0 _ll_register_callbacks cb 0x7f43a0009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54658/0 (socket says 172.21.15.119:54658) 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43a5e00640 1 -- 172.21.15.119:0/3253193070 learned_addr learned my addr 172.21.15.119:0/3253193070 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43a5400640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.216+0000 7f43a6800640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 -- 172.21.15.119:0/3253193070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 msgr2=0x557caea158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 -- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 msgr2=0x557caea153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 -- 172.21.15.119:0/3253193070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557caea15e50 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5400640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43a5e00640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ec0 tx=0x7f43980095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f8e26c4bd211c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f438b400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439804b070 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f43980bb250 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.217+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43980bd650 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43980c10a0 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 10 client.17994 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557caea87ad0 con 0x557caea12080 2024-03-28T18:53:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43980bd650 con 0x557caea12080 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f43980c5260 con 0x557caea12080 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 10 client.17994 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 10 client.17994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 20 client.17994 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea88220 0x557caea8a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 4 client.17994 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557cae97e740 con 0x557caea88220 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.218+0000 7f43a5400640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea88220 0x557caea8a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.219+0000 7f43a5400640 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea88220 0x557caea8a640 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2646a9e7399b816 server_cookie=5c2fa145854c93bf in_seq=0 out_seq=0 2024-03-28T18:53:36.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.219+0000 7f438b400640 10 client.17994 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f438b400640 1 -- 172.21.15.119:0/3253193070 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557cae97e740 con 0x557caea88220 2024-03-28T18:53:36.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f438b400640 10 client.17994 handle_command_reply: tid=0 2024-03-28T18:53:36.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 client.17994 shutdown 2024-03-28T18:53:36.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 msgr2=0x557caea23240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ec0 tx=0x7f43980095e0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 shutdown_connections 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x557caea88220 0x557caea8a640 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557caea1b580 0x557caea158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x557caea129a0 0x557caea153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 --2- 172.21.15.119:0/3253193070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557caea12080 0x557caea23240 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f438b400640 0 client.17994 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.220+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 >> 172.21.15.119:0/3253193070 conn(0x557caea0c7b0 msgr2=0x557caea0afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 shutdown_connections 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43af822740 1 -- 172.21.15.119:0/3253193070 wait complete. 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43af822740 20 client.17994 trim_cache size 0 max 16384 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:53:36.223 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "052fbe83-d7d4-4f55-8d58-6e3388d99924", 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 msgr2=0x7f438c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 0x7f438c0a4e30 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009d90 tx=0x7f43a0009500 comp rx=0 tx=0).stop 2024-03-28T18:53:36.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 msgr2=0x7f43a819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4398000c00 tx=0x7f43980042f0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 shutdown_connections 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.221+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f43a8109850 0x7f43a819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43a8104200 0x7f43a819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 --2- 172.21.15.119:0/3204533784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f43a8103840 0x7f43a819a1d0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 >> 172.21.15.119:0/3204533784 conn(0x7f43a80ff2f0 msgr2=0x7f43a80ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 shutdown_connections 2024-03-28T18:53:36.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.222+0000 7f43aca00640 1 -- 172.21.15.119:0/3204533784 wait complete. 2024-03-28T18:53:36.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:53:36.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 -- 172.21.15.119:0/1279786206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 msgr2=0x7f7a30102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/1279786206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30102ab0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009900 tx=0x7f7a2002bcd0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 -- 172.21.15.119:0/1279786206 shutdown_connections 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/1279786206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30107c30 0x7f7a3010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/1279786206 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30102ff0 0x7f7a30107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/1279786206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30102ab0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 -- 172.21.15.119:0/1279786206 >> 172.21.15.119:0/1279786206 conn(0x7f7a300fbef0 msgr2=0x7f7a300fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 -- 172.21.15.119:0/1279786206 shutdown_connections 2024-03-28T18:53:36.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.453+0000 7f7a2fe00640 1 -- 172.21.15.119:0/1279786206 wait complete. 2024-03-28T18:53:36.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.454+0000 7f7a2fe00640 1 Processor -- start 2024-03-28T18:53:36.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.454+0000 7f7a2fe00640 1 -- start start 2024-03-28T18:53:36.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 0x7f7a301984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30107c30 0x7f7a3019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30111ad0 con 0x7f7a30102ff0 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30111950 con 0x7f7a30107c30 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30111c50 con 0x7f7a301026d0 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 0x7f7a301984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30107c30 0x7f7a3019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43818/0 (socket says 172.21.15.119:43818) 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 0x7f7a301984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:54676/0 (socket says 172.21.15.119:54676) 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.455+0000 7f7a2ea00640 1 -- 172.21.15.119:0/2177931804 learned_addr learned my addr 172.21.15.119:0/2177931804 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 -- 172.21.15.119:0/2177931804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30107c30 msgr2=0x7f7a3019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30107c30 0x7f7a3019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 -- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 msgr2=0x7f7a301984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 0x7f7a301984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 -- 172.21.15.119:0/2177931804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3019cfa0 con 0x7f7a301026d0 2024-03-28T18:53:36.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a20000c00 tx=0x7f7a200042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a27400640 1 -- 172.21.15.119:0/2177931804 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20042020 con 0x7f7a301026d0 2024-03-28T18:53:36.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a27400640 1 -- 172.21.15.119:0/2177931804 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a200054e0 con 0x7f7a301026d0 2024-03-28T18:53:36.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3019d230 con 0x7f7a301026d0 2024-03-28T18:53:36.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a27400640 1 -- 172.21.15.119:0/2177931804 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20003ab0 con 0x7f7a301026d0 2024-03-28T18:53:36.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.456+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a301a4b20 con 0x7f7a301026d0 2024-03-28T18:53:36.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.460+0000 7f7a3899d740 1 Processor -- start 2024-03-28T18:53:36.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.460+0000 7f7a3899d740 1 -- start start 2024-03-28T18:53:36.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b344e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b345b870 0x5598b345dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b33b2250 con 0x5598b344e0f0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b335aa30 con 0x5598b344edd0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a3899d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b33ae2f0 con 0x5598b345b870 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a27400640 1 -- 172.21.15.119:0/2177931804 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f7a20045020 con 0x7f7a301026d0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38282/0 (socket says 172.21.15.119:38282) 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a2f400640 1 -- 172.21.15.119:0/218999173 learned_addr learned my addr 172.21.15.119:0/218999173 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a2e000640 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b345b870 0x5598b345dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.461+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b344e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 -- 172.21.15.119:0/218999173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b345b870 msgr2=0x5598b345dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b345b870 0x5598b345dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 -- 172.21.15.119:0/218999173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 msgr2=0x5598b344e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b344e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 -- 172.21.15.119:0/218999173 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598b3345690 con 0x5598b344edd0 2024-03-28T18:53:36.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a2f400640 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009f50 tx=0x7f7a280118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e615f56ebef43b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a1be00640 1 -- 172.21.15.119:0/218999173 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28004d20 con 0x5598b344edd0 2024-03-28T18:53:36.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a1be00640 1 -- 172.21.15.119:0/218999173 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a28004050 con 0x5598b344edd0 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.462+0000 7f7a1be00640 1 -- 172.21.15.119:0/218999173 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28005280 con 0x5598b344edd0 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a27400640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 0x7f7a140a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 -- 172.21.15.119:0/218999173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 msgr2=0x5598b344db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009f50 tx=0x7f7a280118d0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a27400640 1 -- 172.21.15.119:0/2177931804 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f7a200f09c0 con 0x7f7a301026d0 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 -- 172.21.15.119:0/218999173 shutdown_connections 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b345b870 0x5598b345dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b344e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 --2- 172.21.15.119:0/218999173 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b344db20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 -- 172.21.15.119:0/218999173 >> 172.21.15.119:0/218999173 conn(0x5598b3415200 msgr2=0x5598b3448b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 0x7f7a140a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 -- 172.21.15.119:0/218999173 shutdown_connections 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.463+0000 7f7a3899d740 1 -- 172.21.15.119:0/218999173 wait complete. 2024-03-28T18:53:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a2ea00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 0x7f7a140a4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a200ee240 tx=0x7f7a200dd040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 -- start start 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 Processor -- start 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b3458830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b345e6e0 con 0x5598b345b870 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b336de80 con 0x5598b344edd0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598b345e860 con 0x5598b344e0f0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.464+0000 7f7a3899d740 10 client.0 _ll_register_callbacks cb 0x5598b3380130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:38284/0 (socket says 172.21.15.119:38284) 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 -- 172.21.15.119:0/4122106677 learned_addr learned my addr 172.21.15.119:0/4122106677 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2f400640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b3458830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 -- 172.21.15.119:0/4122106677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 msgr2=0x5598b345af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 -- 172.21.15.119:0/4122106677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 msgr2=0x5598b3458830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b3458830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 -- 172.21.15.119:0/4122106677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598b31907d0 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2f400640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a2e000640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0126e0 tx=0x7f7a1c012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7702c9eed450c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a18c00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0120d0 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a28004050 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.465+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f7a1c00f920 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1c012270 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 10 client.28244 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598b30c63f0 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a28004050 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a1c01c420 con 0x5598b344edd0 2024-03-28T18:53:36.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 10 client.28244 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 10 client.28244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 20 client.28244 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b346b7d0 0x5598b346dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 4 client.28244 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598b33bf790 con 0x5598b346b7d0 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.466+0000 7f7a2f400640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b346b7d0 0x5598b346dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.467+0000 7f7a2f400640 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b346b7d0 0x5598b346dbf0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad4da17f5217ac50 server_cookie=c480411fee698d4d in_seq=0 out_seq=0 2024-03-28T18:53:36.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.467+0000 7f7a18c00640 10 client.28244 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a18c00640 1 -- 172.21.15.119:0/4122106677 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598b33bf790 con 0x5598b346b7d0 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a18c00640 10 client.28244 handle_command_reply: tid=0 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 client.28244 shutdown 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 msgr2=0x5598b34582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0126e0 tx=0x7f7a1c012ba0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 shutdown_connections 2024-03-28T18:53:36.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b346b7d0 0x5598b346dbf0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b3458830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b34582f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 --2- 172.21.15.119:0/4122106677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a18c00640 0 client.28244 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 >> 172.21.15.119:0/4122106677 conn(0x5598b3415200 msgr2=0x5598b344a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 shutdown_connections 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.468+0000 7f7a3899d740 1 -- 172.21.15.119:0/4122106677 wait complete. 2024-03-28T18:53:36.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.469+0000 7f7a3899d740 20 client.28244 trim_cache size 0 max 16384 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 Processor -- start 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 -- start start 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b345af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b345cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301aafd0 con 0x5598b345b870 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301ac950 con 0x5598b344edd0 2024-03-28T18:53:36.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.486+0000 7f7a3899d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301ab630 con 0x5598b344e0f0 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43830/0 (socket says 172.21.15.119:43830) 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2f400640 1 -- 172.21.15.119:0/3237644083 learned_addr learned my addr 172.21.15.119:0/3237644083 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b345af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2f400640 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b345cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 -- 172.21.15.119:0/3237644083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 msgr2=0x5598b345aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 -- 172.21.15.119:0/3237644083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 msgr2=0x5598b345cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b345cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 -- 172.21.15.119:0/3237644083 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598b30860f0 con 0x5598b344edd0 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a2e000640 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b345af60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x5598b34598c0 tx=0x7f7a1c00f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da209e4fbe871d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.487+0000 7f7a1b400640 1 -- 172.21.15.119:0/3237644083 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c01c420 con 0x5598b344edd0 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a1b400640 1 -- 172.21.15.119:0/3237644083 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c01c5c0 con 0x5598b344edd0 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a1b400640 1 -- 172.21.15.119:0/3237644083 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c005330 con 0x5598b344edd0 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 -- 172.21.15.119:0/3237644083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 msgr2=0x5598b345af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b345af60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x5598b34598c0 tx=0x7f7a1c00f810 comp rx=0 tx=0).stop 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 -- 172.21.15.119:0/3237644083 shutdown_connections 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b345b870 0x5598b345cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b344edd0 0x5598b345af60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 --2- 172.21.15.119:0/3237644083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344e0f0 0x5598b345aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 -- 172.21.15.119:0/3237644083 >> 172.21.15.119:0/3237644083 conn(0x5598b3415200 msgr2=0x5598b34582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 -- 172.21.15.119:0/3237644083 shutdown_connections 2024-03-28T18:53:36.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.488+0000 7f7a3899d740 1 -- 172.21.15.119:0/3237644083 wait complete. 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 -- start start 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 Processor -- start 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b3463860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 0x5598b345f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301ac2f0 con 0x5598b344e0f0 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301abc90 con 0x5598b345b870 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301a91b0 con 0x5598b344edd0 2024-03-28T18:53:36.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.489+0000 7f7a3899d740 10 client.0 _ll_register_callbacks cb 0x7f7a1c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b3463860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:43842/0 (socket says 172.21.15.119:43842) 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 -- 172.21.15.119:0/903648307 learned_addr learned my addr 172.21.15.119:0/903648307 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2f400640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 0x5598b345f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 -- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 msgr2=0x5598b345f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 0x5598b345f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 -- 172.21.15.119:0/903648307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 msgr2=0x5598b3463860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b3463860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 -- 172.21.15.119:0/903648307 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598b345f940 con 0x5598b344edd0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2f400640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 0x5598b345f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a2e000640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c01c7c0 tx=0x7f7a1c012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f0e841a87eab54 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.490+0000 7f7a18c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c01c420 con 0x5598b344edd0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a1c01c5c0 con 0x5598b344edd0 2024-03-28T18:53:36.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f7a1c005040 con 0x5598b344edd0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a1c038050 con 0x5598b344edd0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 10 client.28228 fetch_fsmap learned FSMap version 28 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598b34c7080 con 0x5598b344edd0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c01c5c0 con 0x5598b344edd0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7a1c033030 con 0x5598b344edd0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 10 client.28228 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 10 client.28228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 20 client.28228 populate_metadata read hostname 'smithi119' 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.491+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b34c76e0 0x5598b34c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a3899d740 4 client.28228 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5598b33b9ae0 con 0x5598b34c76e0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a2f400640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b34c76e0 0x5598b34c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a18c00640 10 client.28228 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a2f400640 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b34c76e0 0x5598b34c9ad0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df2be4d178cace77 server_cookie=c2693dea0d1dfb8c in_seq=0 out_seq=0 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a18c00640 1 -- 172.21.15.119:0/903648307 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5598b33b9ae0 con 0x5598b34c76e0 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a18c00640 10 client.28228 handle_command_reply: tid=0 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.492+0000 7f7a3899d740 1 client.28228 shutdown 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 msgr2=0x5598b345ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c01c7c0 tx=0x7f7a1c012ba0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 shutdown_connections 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5598b34c76e0 0x5598b34c9ad0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5598b345b870 0x5598b345f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5598b344edd0 0x5598b345ebd0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 --2- 172.21.15.119:0/903648307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5598b344e0f0 0x5598b3463860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a18c00640 0 client.28228 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 >> 172.21.15.119:0/903648307 conn(0x5598b3415200 msgr2=0x5598b3456b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 shutdown_connections 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 1 -- 172.21.15.119:0/903648307 wait complete. 2024-03-28T18:53:36.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.493+0000 7f7a3899d740 20 client.28228 trim_cache size 0 max 16384 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "052fbe83-d7d4-4f55-8d58-6e3388d99924": { 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "052fbe83-d7d4-4f55-8d58-6e3388d99924", 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,force" 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:53:36.497 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 msgr2=0x7f7a140a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 0x7f7a140a4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a200ee240 tx=0x7f7a200dd040 comp rx=0 tx=0).stop 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 msgr2=0x7f7a30197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a20000c00 tx=0x7f7a200042f0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 shutdown_connections 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7a140a29f0 0x7f7a140a4eb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a30107c30 0x7f7a3019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a30102ff0 0x7f7a301984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 --2- 172.21.15.119:0/2177931804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a301026d0 0x7f7a30197fb0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:53:36.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 >> 172.21.15.119:0/2177931804 conn(0x7f7a300fbef0 msgr2=0x7f7a300fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:53:36.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 shutdown_connections 2024-03-28T18:53:36.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:53:36.494+0000 7f7a2fe00640 1 -- 172.21.15.119:0/2177931804 wait complete. 2024-03-28T18:53:36.510 INFO:tasks.cephfs.filesystem:scrub status for tag:052fbe83-d7d4-4f55-8d58-6e3388d99924 - {'path': '/', 'tag': '052fbe83-d7d4-4f55-8d58-6e3388d99924', 'options': 'recursive,force'} 2024-03-28T18:53:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:36 smithi119 ceph-mon[31813]: pgmap v295: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:36 smithi119 ceph-mon[31813]: scrub queued for path: / 2024-03-28T18:53:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:36 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:36 smithi119 ceph-mon[31813]: scrub summary: active paths [/] 2024-03-28T18:53:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:36 smithi119 ceph-mon[31813]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:36.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:36 smithi196 ceph-mon[36300]: pgmap v295: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:36 smithi196 ceph-mon[36300]: scrub queued for path: / 2024-03-28T18:53:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:36 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:36 smithi196 ceph-mon[36300]: scrub summary: active paths [/] 2024-03-28T18:53:36.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:36 smithi196 ceph-mon[36300]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:36 smithi131 ceph-mon[36065]: pgmap v295: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:36 smithi131 ceph-mon[36065]: scrub queued for path: / 2024-03-28T18:53:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:36 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:36 smithi131 ceph-mon[36065]: scrub summary: active paths [/] 2024-03-28T18:53:36.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:36 smithi131 ceph-mon[36065]: scrub summary: idle+waiting paths [/] 2024-03-28T18:53:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:38 smithi119 ceph-mon[31813]: pgmap v296: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:38 smithi119 ceph-mon[31813]: scrub summary: idle 2024-03-28T18:53:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:38 smithi196 ceph-mon[36300]: pgmap v296: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:38 smithi196 ceph-mon[36300]: scrub summary: idle 2024-03-28T18:53:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:38 smithi131 ceph-mon[36065]: pgmap v296: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:38.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:38 smithi131 ceph-mon[36065]: scrub summary: idle 2024-03-28T18:53:40.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:40 smithi119 ceph-mon[31813]: pgmap v297: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:40.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:40 smithi196 ceph-mon[36300]: pgmap v297: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:40 smithi131 ceph-mon[36065]: pgmap v297: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:42 smithi119 ceph-mon[31813]: pgmap v298: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:42.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:42 smithi196 ceph-mon[36300]: pgmap v298: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:42.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:42 smithi131 ceph-mon[36065]: pgmap v298: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:44 smithi119 ceph-mon[31813]: pgmap v299: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:44.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:44 smithi196 ceph-mon[36300]: pgmap v299: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:44.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:44 smithi131 ceph-mon[36065]: pgmap v299: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:46 smithi119 ceph-mon[31813]: pgmap v300: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:46.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:46 smithi196 ceph-mon[36300]: pgmap v300: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:46 smithi131 ceph-mon[36065]: pgmap v300: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:48 smithi119 ceph-mon[31813]: pgmap v301: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:48.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:48 smithi196 ceph-mon[36300]: pgmap v301: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:48 smithi131 ceph-mon[36065]: pgmap v301: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-28T18:53:50.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:50 smithi196 ceph-mon[36300]: pgmap v302: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:50 smithi131 ceph-mon[36065]: pgmap v302: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:51.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:50 smithi119 ceph-mon[31813]: pgmap v302: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-28T18:53:51.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:51 smithi196 ceph-mon[36300]: pgmap v303: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:51.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:51 smithi131 ceph-mon[36065]: pgmap v303: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:52.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:51 smithi119 ceph-mon[31813]: pgmap v303: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:54 smithi119 ceph-mon[31813]: pgmap v304: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:54 smithi196 ceph-mon[36300]: pgmap v304: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:54.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:54 smithi131 ceph-mon[36065]: pgmap v304: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:56 smithi119 ceph-mon[31813]: pgmap v305: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:56 smithi196 ceph-mon[36300]: pgmap v305: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:56 smithi131 ceph-mon[36065]: pgmap v305: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:53:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:53:58 smithi119 ceph-mon[31813]: pgmap v306: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:58.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:53:58 smithi196 ceph-mon[36300]: pgmap v306: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:53:58.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:53:58 smithi131 ceph-mon[36065]: pgmap v306: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:00 smithi119 ceph-mon[31813]: pgmap v307: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:00.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:00 smithi196 ceph-mon[36300]: pgmap v307: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:00.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:00 smithi131 ceph-mon[36065]: pgmap v307: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:02 smithi119 ceph-mon[31813]: pgmap v308: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:02.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:02 smithi196 ceph-mon[36300]: pgmap v308: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:02.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:02 smithi131 ceph-mon[36065]: pgmap v308: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:04.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:04 smithi119 ceph-mon[31813]: pgmap v309: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:04.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:04 smithi196 ceph-mon[36300]: pgmap v309: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:04.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:04 smithi131 ceph-mon[36065]: pgmap v309: 97 pgs: 97 active+clean; 2.2 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:05.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T18:54:05.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 -- 172.21.15.119:0/2553703433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 msgr2=0x7f5a84111fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2553703433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a84111fc0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009930 tx=0x7f5a6c02bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 -- 172.21.15.119:0/2553703433 shutdown_connections 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2553703433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 0x7f5a841148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2553703433 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a84111fc0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2553703433 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a84075fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 -- 172.21.15.119:0/2553703433 >> 172.21.15.119:0/2553703433 conn(0x7f5a840ff390 msgr2=0x7f5a841017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 -- 172.21.15.119:0/2553703433 shutdown_connections 2024-03-28T18:54:05.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.370+0000 7f5a8b200640 1 -- 172.21.15.119:0/2553703433 wait complete. 2024-03-28T18:54:05.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.371+0000 7f5a8b200640 1 Processor -- start 2024-03-28T18:54:05.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.371+0000 7f5a8b200640 1 -- start start 2024-03-28T18:54:05.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a8419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a8419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 0x7f5a841a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a841180c0 con 0x7f5a84112500 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a84117f40 con 0x7f5a84075bd0 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a8b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a84118240 con 0x7f5a840764f0 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a8419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a8419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a83e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a8419e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48196/0 (socket says 172.21.15.119:48196) 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a83e00640 1 -- 172.21.15.119:0/2133074447 learned_addr learned my addr 172.21.15.119:0/2133074447 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a89200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 0x7f5a841a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 -- 172.21.15.119:0/2133074447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 msgr2=0x7f5a8419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a8419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 -- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 msgr2=0x7f5a841a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 0x7f5a841a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.372+0000 7f5a7be00640 1 -- 172.21.15.119:0/2133074447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a841a3520 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a7be00640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a8419eac0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c036040 tx=0x7f5a6c004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c043020 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a6c005000 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a841a37b0 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c003a80 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.373+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a841ab0a0 con 0x7f5a840764f0 2024-03-28T18:54:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.374+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8404f400 con 0x7f5a840764f0 2024-03-28T18:54:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.377+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f5a6c046020 con 0x7f5a840764f0 2024-03-28T18:54:05.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.378+0000 7f5a81600640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 0x7f5a640a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.378+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f5a6c0f0c50 con 0x7f5a840764f0 2024-03-28T18:54:05.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.378+0000 7f5a83e00640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 0x7f5a640a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.379+0000 7f5a83e00640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 0x7f5a640a4f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009cd0 tx=0x7f5a74005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.384+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5a6c04d020 con 0x7f5a840764f0 2024-03-28T18:54:05.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.610+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a84061ae0 con 0x7f5a640a2a90 2024-03-28T18:54:05.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.622+0000 7f5a81600640 1 -- 172.21.15.119:0/2133074447 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f5a84061ae0 con 0x7f5a640a2a90 2024-03-28T18:54:05.624 INFO:teuthology.orchestra.run.smithi119.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T18:44:04", "first": "2024-03-28T18:45:00", "last": "2024-03-28T18:54:00", "last_pruned": "2024-03-28T18:54:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 msgr2=0x7f5a640a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 0x7f5a640a4f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009cd0 tx=0x7f5a74005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 msgr2=0x7f5a8419eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a8419eac0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c036040 tx=0x7f5a6c004250 comp rx=0 tx=0).stop 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 shutdown_connections 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f5a640a2a90 0x7f5a640a4f50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a84112500 0x7f5a841a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a840764f0 0x7f5a8419eac0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 --2- 172.21.15.119:0/2133074447 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5a84075bd0 0x7f5a8419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.625+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 >> 172.21.15.119:0/2133074447 conn(0x7f5a840ff390 msgr2=0x7f5a84101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:05.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.626+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 shutdown_connections 2024-03-28T18:54:05.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.626+0000 7f5a8b200640 1 -- 172.21.15.119:0/2133074447 wait complete. 2024-03-28T18:54:05.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 -- 172.21.15.119:0/149798526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c107660 msgr2=0x7fdd9c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 --2- 172.21.15.119:0/149798526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c107660 0x7fdd9c107a40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009930 tx=0x7fdd9002bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 -- 172.21.15.119:0/149798526 shutdown_connections 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 --2- 172.21.15.119:0/149798526 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c102010 0x7fdd9c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 --2- 172.21.15.119:0/149798526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c101650 0x7fdd9c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 --2- 172.21.15.119:0/149798526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c107660 0x7fdd9c107a40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.846+0000 7fdda4600640 1 -- 172.21.15.119:0/149798526 >> 172.21.15.119:0/149798526 conn(0x7fdd9c0fd160 msgr2=0x7fdd9c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:05.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.847+0000 7fdda4600640 1 -- 172.21.15.119:0/149798526 shutdown_connections 2024-03-28T18:54:05.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.847+0000 7fdda4600640 1 -- 172.21.15.119:0/149798526 wait complete. 2024-03-28T18:54:05.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.848+0000 7fdda4600640 1 Processor -- start 2024-03-28T18:54:05.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.848+0000 7fdda4600640 1 -- start start 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.848+0000 7fdda4600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.848+0000 7fdda4600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c102010 0x7fdd9c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.848+0000 7fdda4600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c107660 0x7fdd9c19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda4600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c111ac0 con 0x7fdd9c102010 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda4600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c111940 con 0x7fdd9c107660 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda4600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c111c40 con 0x7fdd9c101650 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41654/0 (socket says 172.21.15.119:41654) 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 -- 172.21.15.119:0/4180115246 learned_addr learned my addr 172.21.15.119:0/4180115246 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1200640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c102010 0x7fdd9c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda2600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c107660 0x7fdd9c19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 -- 172.21.15.119:0/4180115246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c107660 msgr2=0x7fdd9c19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c107660 0x7fdd9c19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 -- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c102010 msgr2=0x7fdd9c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c102010 0x7fdd9c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 -- 172.21.15.119:0/4180115246 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c19ced0 con 0x7fdd9c101650 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.849+0000 7fdda1c00640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdd90005ab0 tx=0x7fdd90005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.850+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90043020 con 0x7fdd9c101650 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.850+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd90005000 con 0x7fdd9c101650 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.850+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c19d160 con 0x7fdd9c101650 2024-03-28T18:54:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.850+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90003a80 con 0x7fdd9c101650 2024-03-28T18:54:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.850+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c1a4a50 con 0x7fdd9c101650 2024-03-28T18:54:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.851+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c04f400 con 0x7fdd9c101650 2024-03-28T18:54:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.853+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fdd90046020 con 0x7fdd9c101650 2024-03-28T18:54:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.854+0000 7fdd9a000640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 0x7fdd840a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.854+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fdd900f0cc0 con 0x7fdd9c101650 2024-03-28T18:54:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.854+0000 7fdda1200640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 0x7fdd840a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.855+0000 7fdda1200640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 0x7fdd840a4f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0ff6d0 tx=0x7fdd88005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:05.858+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd9004d020 con 0x7fdd9c101650 2024-03-28T18:54:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.081+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 --> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdd9c061ae0 con 0x7fdd840a2a90 2024-03-28T18:54:06.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.095+0000 7fdd9a000640 1 -- 172.21.15.119:0/4180115246 <== mgr.25418 v2:172.21.15.119:6800/212877697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fdd9c061ae0 con 0x7fdd840a2a90 2024-03-28T18:54:06.097 INFO:teuthology.orchestra.run.smithi119.stdout:/ 1m 6m3h 2024-03-28T18:54:06.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 msgr2=0x7fdd840a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 0x7fdd840a4f50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c0ff6d0 tx=0x7fdd88005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 msgr2=0x7fdd9c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdd90005ab0 tx=0x7fdd90005e80 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 shutdown_connections 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdd840a2a90 0x7fdd840a4f50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdd9c107660 0x7fdd9c19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd9c102010 0x7fdd9c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 --2- 172.21.15.119:0/4180115246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd9c101650 0x7fdd9c197ee0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.098+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 >> 172.21.15.119:0/4180115246 conn(0x7fdd9c0fd160 msgr2=0x7fdd9c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.099+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 shutdown_connections 2024-03-28T18:54:06.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.099+0000 7fdda4600640 1 -- 172.21.15.119:0/4180115246 wait complete. 2024-03-28T18:54:06.119 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-28T18:54:06.131 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-28T18:54:06.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-28T18:54:06.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 -- 172.21.15.119:0/1064926482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4103840 msgr2=0x7fbfe4103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 --2- 172.21.15.119:0/1064926482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4103840 0x7fbfe4103cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009890 tx=0x7fbfd002bb70 comp rx=0 tx=0).stop 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 -- 172.21.15.119:0/1064926482 shutdown_connections 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 --2- 172.21.15.119:0/1064926482 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe4110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 --2- 172.21.15.119:0/1064926482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4103840 0x7fbfe4103cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 --2- 172.21.15.119:0/1064926482 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4109850 0x7fbfe4109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 -- 172.21.15.119:0/1064926482 >> 172.21.15.119:0/1064926482 conn(0x7fbfe40ff2f0 msgr2=0x7fbfe4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 -- 172.21.15.119:0/1064926482 shutdown_connections 2024-03-28T18:54:06.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.738+0000 7fbfe9800640 1 -- 172.21.15.119:0/1064926482 wait complete. 2024-03-28T18:54:06.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 Processor -- start 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 -- start start 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4109850 0x7fbfe419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4113c20 con 0x7fbfe4109850 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4113aa0 con 0x7fbfe4103840 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.739+0000 7fbfe9800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4113da0 con 0x7fbfe4104200 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48246/0 (socket says 172.21.15.119:48246) 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 -- 172.21.15.119:0/73921991 learned_addr learned my addr 172.21.15.119:0/73921991 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe3400640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4109850 0x7fbfe419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 -- 172.21.15.119:0/73921991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 msgr2=0x7fbfe419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 -- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4109850 msgr2=0x7fbfe419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4109850 0x7fbfe419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 -- 172.21.15.119:0/73921991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe419f190 con 0x7fbfe4103840 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2000640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400ea60 tx=0x7fbfd400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.740+0000 7fbfdb400640 1 -- 172.21.15.119:0/73921991 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400cde0 con 0x7fbfe4103840 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.741+0000 7fbfdb400640 1 -- 172.21.15.119:0/73921991 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd40035f0 con 0x7fbfe4103840 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.741+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe419f480 con 0x7fbfe4103840 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.741+0000 7fbfdb400640 1 -- 172.21.15.119:0/73921991 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4010670 con 0x7fbfe4103840 2024-03-28T18:54:06.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.741+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe41a6d10 con 0x7fbfe4103840 2024-03-28T18:54:06.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.743+0000 7fbfdb400640 1 -- 172.21.15.119:0/73921991 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fbfd40037a0 con 0x7fbfe4103840 2024-03-28T18:54:06.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.744+0000 7fbfec477740 1 Processor -- start 2024-03-28T18:54:06.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.744+0000 7fbfec477740 1 -- start start 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.744+0000 7fbfec477740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6fc00 0x5573bff6e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.744+0000 7fbfec477740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff6f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.744+0000 7fbfec477740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff7a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfec477740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfed3250 con 0x5573bff6fc00 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfec477740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfe7ba30 con 0x5573bff78290 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfec477740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfecf2f0 con 0x5573bff6ef20 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff6f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff6f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41688/0 (socket says 172.21.15.119:41688) 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff7a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2a00640 1 -- 172.21.15.119:0/279792780 learned_addr learned my addr 172.21.15.119:0/279792780 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe3400640 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6fc00 0x5573bff6e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfdb400640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 0x7fbfc80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 -- 172.21.15.119:0/279792780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 msgr2=0x5573bff6f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff6f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 -- 172.21.15.119:0/279792780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6fc00 msgr2=0x5573bff6e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6fc00 0x5573bff6e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe2000640 1 -- 172.21.15.119:0/279792780 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bfe66690 con 0x5573bff78290 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfdb400640 1 -- 172.21.15.119:0/73921991 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbfd40cf170 con 0x7fbfe4103840 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.745+0000 7fbfe3400640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 0x7fbfc80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfe2000640 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff7a680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00057c0 tx=0x7fbfd00056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec63fadffe4ae95d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfcfe00640 1 -- 172.21.15.119:0/279792780 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0044020 con 0x5573bff78290 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfcfe00640 1 -- 172.21.15.119:0/279792780 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd0034440 con 0x5573bff78290 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfcfe00640 1 -- 172.21.15.119:0/279792780 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0039040 con 0x5573bff78290 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfec477740 1 -- 172.21.15.119:0/279792780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 msgr2=0x5573bff7a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfe3400640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 0x7fbfc80a4eb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc005fd0 tx=0x7fbfdc004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.746+0000 7fbfec477740 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff7a680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00057c0 tx=0x7fbfd00056d0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 -- 172.21.15.119:0/279792780 shutdown_connections 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff7a680 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff6f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 --2- 172.21.15.119:0/279792780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6fc00 0x5573bff6e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 -- 172.21.15.119:0/279792780 >> 172.21.15.119:0/279792780 conn(0x5573bff36200 msgr2=0x5573bff69970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 -- 172.21.15.119:0/279792780 shutdown_connections 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 -- 172.21.15.119:0/279792780 wait complete. 2024-03-28T18:54:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 -- start start 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.747+0000 7fbfec477740 1 Processor -- start 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6ef20 0x5573bff77910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff751e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bff7b0a0 con 0x5573bff6ef20 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bfe8ee80 con 0x5573bff78290 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573bff7b220 con 0x5573bff6fc00 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfec477740 10 client.0 _ll_register_callbacks cb 0x5573bfea1130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41700/0 (socket says 172.21.15.119:41700) 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 -- 172.21.15.119:0/3987608411 learned_addr learned my addr 172.21.15.119:0/3987608411 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6ef20 0x5573bff77910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 -- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 msgr2=0x5573bff751e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff751e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 -- 172.21.15.119:0/3987608411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6ef20 msgr2=0x5573bff77910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6ef20 0x5573bff77910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.748+0000 7fbfe2000640 1 -- 172.21.15.119:0/3987608411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573bfcb17d0 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfe2000640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbfd002e010 tx=0x7fbfd002c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b077e84fde8cd954 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfccc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0044040 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbfd002c9f0 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfd0034440 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfd0039040 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfec477740 10 client.28252 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.749+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5573bfbe73f0 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd0034440 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbfd0032ad0 con 0x5573bff6fc00 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 10 client.28252 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:06.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 10 client.28252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:54:06.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 20 client.28252 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:06.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bff87f60 0x5573bff8a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 4 client.28252 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:06.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573bfee0790 con 0x5573bff87f60 2024-03-28T18:54:06.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bff87f60 0x5573bff8a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bff87f60 0x5573bff8a350 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8689270892a0f780 server_cookie=f990fb01545e2510 in_seq=0 out_seq=0 2024-03-28T18:54:06.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.750+0000 7fbfccc00640 10 client.28252 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:06.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfccc00640 1 -- 172.21.15.119:0/3987608411 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5573bfee0790 con 0x5573bff87f60 2024-03-28T18:54:06.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfccc00640 10 client.28252 handle_command_reply: tid=0 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 client.28252 shutdown 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 msgr2=0x5573bff74ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbfd002e010 tx=0x7fbfd002c510 comp rx=0 tx=0).stop 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 shutdown_connections 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bff87f60 0x5573bff8a350 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff78290 0x5573bff751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff74ca0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 --2- 172.21.15.119:0/3987608411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff6ef20 0x5573bff77910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 >> 172.21.15.119:0/3987608411 conn(0x5573bff36200 msgr2=0x5573bff6b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.751+0000 7fbfccc00640 0 client.28252 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.752+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 shutdown_connections 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.752+0000 7fbfec477740 1 -- 172.21.15.119:0/3987608411 wait complete. 2024-03-28T18:54:06.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.752+0000 7fbfec477740 20 client.28252 trim_cache size 0 max 16384 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 Processor -- start 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 -- start start 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff773d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff79680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41af680 con 0x5573bff78290 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41b1000 con 0x5573bff6fc00 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.769+0000 7fbfec477740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41afce0 con 0x5573bff6ef20 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff79680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48286/0 (socket says 172.21.15.119:48286) 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 -- 172.21.15.119:0/1294513822 learned_addr learned my addr 172.21.15.119:0/1294513822 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 -- 172.21.15.119:0/1294513822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 msgr2=0x5573bff773d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff773d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 -- 172.21.15.119:0/1294513822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 msgr2=0x5573bff79680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff79680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 -- 172.21.15.119:0/1294513822 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573bfba70f0 con 0x5573bff6fc00 2024-03-28T18:54:06.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfe2000640 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00057c0 tx=0x7fbfd0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d113cbb710faeed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.770+0000 7fbfcf400640 1 -- 172.21.15.119:0/1294513822 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0044040 con 0x5573bff6fc00 2024-03-28T18:54:06.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfcf400640 1 -- 172.21.15.119:0/1294513822 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd0034440 con 0x5573bff6fc00 2024-03-28T18:54:06.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfcf400640 1 -- 172.21.15.119:0/1294513822 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0032640 con 0x5573bff6fc00 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 -- 172.21.15.119:0/1294513822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 msgr2=0x5573bff77910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00057c0 tx=0x7fbfd0005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 -- 172.21.15.119:0/1294513822 shutdown_connections 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff79680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6fc00 0x5573bff77910 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 --2- 172.21.15.119:0/1294513822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6ef20 0x5573bff773d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 -- 172.21.15.119:0/1294513822 >> 172.21.15.119:0/1294513822 conn(0x5573bff36200 msgr2=0x5573bff68a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 -- 172.21.15.119:0/1294513822 shutdown_connections 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.771+0000 7fbfec477740 1 -- 172.21.15.119:0/1294513822 wait complete. 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 -- start start 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 Processor -- start 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 0x5573bff89f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff8a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41b09a0 con 0x5573bff78290 2024-03-28T18:54:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41b0340 con 0x5573bff6ef20 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41ad860 con 0x5573bff6fc00 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.772+0000 7fbfec477740 10 client.0 _ll_register_callbacks cb 0x7fbfd4097980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 0x5573bff89f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48892/0 (socket says 172.21.15.119:48892) 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 -- 172.21.15.119:0/4682964 learned_addr learned my addr 172.21.15.119:0/4682964 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff8a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 -- 172.21.15.119:0/4682964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 msgr2=0x5573bff8a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff8a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 -- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 msgr2=0x5573bff89f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 0x5573bff89f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 -- 172.21.15.119:0/4682964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573bffebf90 con 0x5573bff78290 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 0x5573bff89f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfe3400640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc01add0 tx=0x7fbfdc010040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f4f60b32ba6dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.773+0000 7fbfccc00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc02c070 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbfd0034440 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fbfdc01ae00 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbfdc037050 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfec477740 10 client.18042 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5573bffed2c0 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfd0034440 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.774+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbfdc00b040 con 0x5573bff78290 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 10 client.18042 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 10 client.18042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 20 client.18042 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bffed9d0 0x5573bffefdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 4 client.18042 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-28T18:54:06.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5573bfedaae0 con 0x5573bffed9d0 2024-03-28T18:54:06.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bffed9d0 0x5573bffefdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfe2a00640 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bffed9d0 0x5573bffefdf0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d82c690338f8a6 server_cookie=c3215777f7690955 in_seq=0 out_seq=0 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfccc00640 10 client.18042 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfccc00640 1 -- 172.21.15.119:0/4682964 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5573bfedaae0 con 0x5573bffed9d0 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfccc00640 10 client.18042 handle_command_reply: tid=0 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.775+0000 7fbfec477740 1 client.18042 shutdown 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 msgr2=0x5573bff8aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc01add0 tx=0x7fbfdc010040 comp rx=0 tx=0).stop 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 shutdown_connections 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x5573bffed9d0 0x5573bffefdf0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5573bff78290 0x5573bff8aa30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5573bff6fc00 0x5573bff8a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 --2- 172.21.15.119:0/4682964 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x5573bff6ef20 0x5573bff89f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfccc00640 0 client.18042 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 >> 172.21.15.119:0/4682964 conn(0x5573bff36200 msgr2=0x5573bff6ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 shutdown_connections 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 1 -- 172.21.15.119:0/4682964 wait complete. 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.776+0000 7fbfec477740 20 client.18042 trim_cache size 0 max 16384 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-03-28T18:54:06.781 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:06.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 msgr2=0x7fbfc80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 0x7fbfc80a4eb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc005fd0 tx=0x7fbfdc004a40 comp rx=0 tx=0).stop 2024-03-28T18:54:06.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 msgr2=0x7fbfe419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400ea60 tx=0x7fbfd400ef20 comp rx=0 tx=0).stop 2024-03-28T18:54:06.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 shutdown_connections 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fbfc80a29f0 0x7fbfc80a4eb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfe4109850 0x7fbfe419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbfe4104200 0x7fbfe419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 --2- 172.21.15.119:0/73921991 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbfe4103840 0x7fbfe419a1a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 >> 172.21.15.119:0/73921991 conn(0x7fbfe40ff2f0 msgr2=0x7fbfe40ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 shutdown_connections 2024-03-28T18:54:06.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.777+0000 7fbfe9800640 1 -- 172.21.15.119:0/73921991 wait complete. 2024-03-28T18:54:06.792 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:06 smithi119 ceph-mon[31813]: pgmap v310: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:54:06.795 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-28T18:54:06.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:54:06.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:06 smithi196 ceph-mon[36300]: pgmap v310: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:54:06.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:06 smithi131 ceph-mon[36065]: pgmap v310: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T18:54:06.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 -- 172.21.15.119:0/3356061540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 msgr2=0x7faae8109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 --2- 172.21.15.119:0/3356061540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae8109c40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faadc009900 tx=0x7faadc02bc40 comp rx=0 tx=0).stop 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 -- 172.21.15.119:0/3356061540 shutdown_connections 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 --2- 172.21.15.119:0/3356061540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8104210 0x7faae8110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 --2- 172.21.15.119:0/3356061540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8103850 0x7faae8103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 --2- 172.21.15.119:0/3356061540 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae8109c40 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.996+0000 7faaf0000640 1 -- 172.21.15.119:0/3356061540 >> 172.21.15.119:0/3356061540 conn(0x7faae80ff2e0 msgr2=0x7faae8101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.997+0000 7faaf0000640 1 -- 172.21.15.119:0/3356061540 shutdown_connections 2024-03-28T18:54:06.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.997+0000 7faaf0000640 1 -- 172.21.15.119:0/3356061540 wait complete. 2024-03-28T18:54:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 Processor -- start 2024-03-28T18:54:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 -- start start 2024-03-28T18:54:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:06.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8104210 0x7faae819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8113c80 con 0x7faae8104210 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8113b00 con 0x7faae8109860 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaf0000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8113e00 con 0x7faae8103850 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41742/0 (socket says 172.21.15.119:41742) 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaed600640 1 -- 172.21.15.119:0/4088764972 learned_addr learned my addr 172.21.15.119:0/4088764972 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaee000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.998+0000 7faaecc00640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8104210 0x7faae819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 -- 172.21.15.119:0/4088764972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 msgr2=0x7faae819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 -- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8104210 msgr2=0x7faae819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8104210 0x7faae819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 -- 172.21.15.119:0/4088764972 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae819f130 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaed600640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7faadc005ab0 tx=0x7faadc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc041070 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faadc0054e0 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc003bf0 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae819f3c0 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:06.999+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae81a6cb0 con 0x7faae8103850 2024-03-28T18:54:07.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.000+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8104f90 con 0x7faae8103850 2024-03-28T18:54:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.003+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7faadc044070 con 0x7faae8103850 2024-03-28T18:54:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.004+0000 7faae6000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 0x7faad00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.004+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7faadc0f0d30 con 0x7faae8103850 2024-03-28T18:54:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.004+0000 7faaecc00640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 0x7faad00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.005+0000 7faaecc00640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 0x7faad00a4e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faae8102990 tx=0x7faad4005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.006+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faadc0bdab0 con 0x7faae8103850 2024-03-28T18:54:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.245+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faae8068490 con 0x7faae8103850 2024-03-28T18:54:07.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.246+0000 7faae6000640 1 -- 172.21.15.119:0/4088764972 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7faae8068490 con 0x7faae8103850 2024-03-28T18:54:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:07.248 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:54:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.249+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 msgr2=0x7faad00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.249+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 0x7faad00a4e30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faae8102990 tx=0x7faad4005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.249+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 msgr2=0x7faae819a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.249+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7faadc005ab0 tx=0x7faadc004270 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 shutdown_connections 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7faad00a2970 0x7faad00a4e30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7faae8109860 0x7faae819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae8104210 0x7faae819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 --2- 172.21.15.119:0/4088764972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faae8103850 0x7faae819a140 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 >> 172.21.15.119:0/4088764972 conn(0x7faae80ff2e0 msgr2=0x7faae80ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 shutdown_connections 2024-03-28T18:54:07.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.250+0000 7faaf0000640 1 -- 172.21.15.119:0/4088764972 wait complete. 2024-03-28T18:54:07.270 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:54:07.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 -- 172.21.15.119:0/2525403728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0108860 msgr2=0x7f61d010c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 --2- 172.21.15.119:0/2525403728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0108860 0x7f61d010c8a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009ca0 tx=0x7f61b802c100 comp rx=0 tx=0).stop 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 -- 172.21.15.119:0/2525403728 shutdown_connections 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 --2- 172.21.15.119:0/2525403728 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d010cf80 0x7f61d0114b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 --2- 172.21.15.119:0/2525403728 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0108860 0x7f61d010c8a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 --2- 172.21.15.119:0/2525403728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d0107eb0 0x7f61d0108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 -- 172.21.15.119:0/2525403728 >> 172.21.15.119:0/2525403728 conn(0x7f61d0075be0 msgr2=0x7f61d0078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 -- 172.21.15.119:0/2525403728 shutdown_connections 2024-03-28T18:54:07.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.484+0000 7f61d5a00640 1 -- 172.21.15.119:0/2525403728 wait complete. 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.485+0000 7f61d5a00640 1 Processor -- start 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.485+0000 7f61d5a00640 1 -- start start 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d010cf80 0x7f61d019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d01a2e40 0x7f61d01a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0117ff0 con 0x7f61d010cf80 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0117e70 con 0x7f61d01a2e40 2024-03-28T18:54:07.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61d5a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0118170 con 0x7f61d0107eb0 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61ce000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d010cf80 0x7f61d019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41758/0 (socket says 172.21.15.119:41758) 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 -- 172.21.15.119:0/25608385 learned_addr learned my addr 172.21.15.119:0/25608385 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cf400640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d01a2e40 0x7f61d01a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 -- 172.21.15.119:0/25608385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d01a2e40 msgr2=0x7f61d01a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d01a2e40 0x7f61d01a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 -- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d010cf80 msgr2=0x7f61d019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d010cf80 0x7f61d019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.486+0000 7f61cea00640 1 -- 172.21.15.119:0/25608385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d01a3930 con 0x7f61d0107eb0 2024-03-28T18:54:07.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61cea00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f61c800e9f0 tx=0x7f61c800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61c7400640 1 -- 172.21.15.119:0/25608385 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c800ccc0 con 0x7f61d0107eb0 2024-03-28T18:54:07.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61c7400640 1 -- 172.21.15.119:0/25608385 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c8010910 con 0x7f61d0107eb0 2024-03-28T18:54:07.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d01a3c20 con 0x7f61d0107eb0 2024-03-28T18:54:07.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61c7400640 1 -- 172.21.15.119:0/25608385 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c80188a0 con 0x7f61d0107eb0 2024-03-28T18:54:07.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.487+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d01ab430 con 0x7f61d0107eb0 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.490+0000 7f61d87dd740 1 Processor -- start 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.490+0000 7f61d87dd740 1 -- start start 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960912630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960915050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f96086d2c0 con 0x55f960912c00 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f960815c90 con 0x55f96090a080 2024-03-28T18:54:07.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61d87dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f960869360 con 0x55f9609096d0 2024-03-28T18:54:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61cf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61cf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41774/0 (socket says 172.21.15.119:41774) 2024-03-28T18:54:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61ce000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960915050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.491+0000 7f61cf400640 1 -- 172.21.15.119:0/1293660923 learned_addr learned my addr 172.21.15.119:0/1293660923 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 -- 172.21.15.119:0/1293660923 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 msgr2=0x55f960912630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960912630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 -- 172.21.15.119:0/1293660923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 msgr2=0x55f960915050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960915050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 -- 172.21.15.119:0/1293660923 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f96083a080 con 0x55f9609096d0 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61c7400640 1 -- 172.21.15.119:0/25608385 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f61c8010ac0 con 0x7f61d0107eb0 2024-03-28T18:54:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61cf400640 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009c10 tx=0x7f61c0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bac8319fef460508 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61bfe00640 1 -- 172.21.15.119:0/1293660923 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0004230 con 0x55f9609096d0 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61bfe00640 1 -- 172.21.15.119:0/1293660923 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c00043d0 con 0x55f9609096d0 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.492+0000 7f61bfe00640 1 -- 172.21.15.119:0/1293660923 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c00197a0 con 0x55f9609096d0 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 -- 172.21.15.119:0/1293660923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 msgr2=0x55f960909ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f61c0009c10 tx=0x7f61c0011810 comp rx=0 tx=0).stop 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 -- 172.21.15.119:0/1293660923 shutdown_connections 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960915050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960912630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1293660923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f960909ab0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 -- 172.21.15.119:0/1293660923 >> 172.21.15.119:0/1293660923 conn(0x55f960903da0 msgr2=0x55f960904180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 -- 172.21.15.119:0/1293660923 shutdown_connections 2024-03-28T18:54:07.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.493+0000 7f61d87dd740 1 -- 172.21.15.119:0/1293660923 wait complete. 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61c7400640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 0x7f61b40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61ce000640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 0x7f61b40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61c7400640 1 -- 172.21.15.119:0/25608385 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f61c80cf300 con 0x7f61d0107eb0 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61d87dd740 1 -- start start 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61d87dd740 1 Processor -- start 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.494+0000 7f61ce000640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 0x7f61b40a4dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f61b802c710 tx=0x7f61b80093a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f96090a080 0x55f96090f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f960912c00 0x55f96090ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f960915b70 con 0x55f96090a080 2024-03-28T18:54:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9608290d0 con 0x55f9609096d0 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f960915cf0 con 0x55f960912c00 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61d87dd740 10 client.0 _ll_register_callbacks cb 0x55f96082de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48356/0 (socket says 172.21.15.119:48356) 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61ce000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f96090a080 0x55f96090f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f960912c00 0x55f96090ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 -- 172.21.15.119:0/1528299445 learned_addr learned my addr 172.21.15.119:0/1528299445 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 -- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f960912c00 msgr2=0x55f96090ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f960912c00 0x55f96090ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 -- 172.21.15.119:0/1528299445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f96090a080 msgr2=0x55f96090f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f96090a080 0x55f96090f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.495+0000 7f61cea00640 1 -- 172.21.15.119:0/1528299445 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9605813f0 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61cea00640 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cc9c0 tx=0x7f61c809b4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d86b2e8567f0d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8086270 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f61c809cba0 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61c00043d0 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c809b870 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61d87dd740 10 client.28295 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f96064b7d0 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c00043d0 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.496+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f61c8097650 con 0x55f9609096d0 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 10 client.28295 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 10 client.28295 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:54:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 20 client.28295 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f960922a50 0x55f960924e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 4 client.28295 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f96087a800 con 0x55f960922a50 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61ce000640 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f960922a50 0x55f960924e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61ce000640 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f960922a50 0x55f960924e40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99a56d5410c6412 server_cookie=43effde2724b9742 in_seq=0 out_seq=0 2024-03-28T18:54:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.497+0000 7f61bcc00640 10 client.28295 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:54:07.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.498+0000 7f61bcc00640 1 -- 172.21.15.119:0/1528299445 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f96087a800 con 0x55f960922a50 2024-03-28T18:54:07.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.498+0000 7f61bcc00640 10 client.28295 handle_command_reply: tid=0 2024-03-28T18:54:07.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.498+0000 7f61d87dd740 1 client.28295 shutdown 2024-03-28T18:54:07.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 msgr2=0x55f960912590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f61c80cc9c0 tx=0x7f61c809b4c0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 shutdown_connections 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f960922a50 0x55f960924e40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f960912c00 0x55f96090ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f96090a080 0x55f96090f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1528299445 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912590 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61bcc00640 0 client.28295 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 >> 172.21.15.119:0/1528299445 conn(0x55f960903da0 msgr2=0x55f960905af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 shutdown_connections 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 1 -- 172.21.15.119:0/1528299445 wait complete. 2024-03-28T18:54:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.499+0000 7f61d87dd740 20 client.28295 trim_cache size 0 max 16384 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 Processor -- start 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 -- start start 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 0x55f960912590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f96090cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b3a10 con 0x55f960912c00 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b5390 con 0x55f9609096d0 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61d87dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b4070 con 0x55f96090a080 2024-03-28T18:54:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61cf400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f96090cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61ce000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 0x55f960912590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61cea00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48364/0 (socket says 172.21.15.119:48364) 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61cea00640 1 -- 172.21.15.119:0/839154682 learned_addr learned my addr 172.21.15.119:0/839154682 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.513+0000 7f61ce000640 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 0x55f960912590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41790/0 (socket says 172.21.15.119:41790) 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 -- 172.21.15.119:0/839154682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 msgr2=0x55f960912590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 0x55f960912590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 -- 172.21.15.119:0/839154682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 msgr2=0x55f96090cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f96090cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 -- 172.21.15.119:0/839154682 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f96090ded0 con 0x55f9609096d0 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61cea00640 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f61c8097c80 tx=0x7f61c800ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a61dcbb5c3dd7a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61bf400640 1 -- 172.21.15.119:0/839154682 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c809aab0 con 0x55f9609096d0 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61bf400640 1 -- 172.21.15.119:0/839154682 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c809bf70 con 0x55f9609096d0 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.514+0000 7f61bf400640 1 -- 172.21.15.119:0/839154682 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c809cba0 con 0x55f9609096d0 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- 172.21.15.119:0/839154682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 msgr2=0x55f960912050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f61c8097c80 tx=0x7f61c800ce60 comp rx=0 tx=0).stop 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- 172.21.15.119:0/839154682 shutdown_connections 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f96090cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f96090a080 0x55f960912590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 --2- 172.21.15.119:0/839154682 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f9609096d0 0x55f960912050 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- 172.21.15.119:0/839154682 >> 172.21.15.119:0/839154682 conn(0x55f960903da0 msgr2=0x55f960902eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- 172.21.15.119:0/839154682 shutdown_connections 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- 172.21.15.119:0/839154682 wait complete. 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 -- start start 2024-03-28T18:54:07.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.515+0000 7f61d87dd740 1 Processor -- start 2024-03-28T18:54:07.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960919230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960919970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b4d30 con 0x55f960912c00 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b46d0 con 0x55f96090a080 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01b1bf0 con 0x55f9609096d0 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61d87dd740 10 client.0 _ll_register_callbacks cb 0x7f61c8010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61ce000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960919230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41792/0 (socket says 172.21.15.119:41792) 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 -- 172.21.15.119:0/1186317453 learned_addr learned my addr 172.21.15.119:0/1186317453 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cf400640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960919970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 -- 172.21.15.119:0/1186317453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 msgr2=0x55f960919230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960919230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 -- 172.21.15.119:0/1186317453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 msgr2=0x55f960919970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960919970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61cea00640 1 -- 172.21.15.119:0/1186317453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f96097e250 con 0x55f9609096d0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.516+0000 7f61ce000640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960919230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61cea00640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f61c8098240 tx=0x7f61c8098160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b7b89cfa93ba3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c80a4010 con 0x55f9609096d0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61c809bf70 con 0x55f9609096d0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f61c809cba0 con 0x55f9609096d0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c809f9c0 con 0x55f9609096d0 2024-03-28T18:54:07.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61d87dd740 10 client.28282 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f96097e250 con 0x55f9609096d0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c809bf70 con 0x55f9609096d0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.517+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f61c80a0ae0 con 0x55f9609096d0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 10 client.28282 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 10 client.28282 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 20 client.28282 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f96097f6b0 0x55f960981aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 4 client.28282 mds_command: new command op to 24505 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 --> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f960874b50 con 0x55f96097f6b0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61ce000640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f96097f6b0 0x55f960981aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61ce000640 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f96097f6b0 0x55f960981aa0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85845d4053fd08ff server_cookie=b6b0b7f13d3888e5 in_seq=0 out_seq=0 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.518+0000 7f61bcc00640 10 client.28282 ms_handle_connect on v2:172.21.15.196:6834/480911404 2024-03-28T18:54:07.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61bcc00640 1 -- 172.21.15.119:0/1186317453 <== mds.4 v2:172.21.15.196:6834/480911404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f960874b50 con 0x55f96097f6b0 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61bcc00640 10 client.28282 handle_command_reply: tid=0 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 client.28282 shutdown 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 msgr2=0x55f96091ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f61c8098240 tx=0x7f61c8098160 comp rx=0 tx=0).stop 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 shutdown_connections 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:6834/480911404,v1:172.21.15.196:6835/480911404] conn(0x55f96097f6b0 0x55f960981aa0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f960912c00 0x55f960919970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55f96090a080 0x55f960919230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61bcc00640 0 client.28282 ms_handle_reset on v2:172.21.15.196:6834/480911404 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 --2- 172.21.15.119:0/1186317453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f9609096d0 0x55f96091ae90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 >> 172.21.15.119:0/1186317453 conn(0x55f960903da0 msgr2=0x55f960902620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 shutdown_connections 2024-03-28T18:54:07.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 1 -- 172.21.15.119:0/1186317453 wait complete. 2024-03-28T18:54:07.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.519+0000 7f61d87dd740 20 client.28282 trim_cache size 0 max 16384 2024-03-28T18:54:07.527 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 msgr2=0x7f61b40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 0x7f61b40a4dc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f61b802c710 tx=0x7f61b80093a0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 msgr2=0x7f61d019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f61c800e9f0 tx=0x7f61c800edf0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 shutdown_connections 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f61b40a2900 0x7f61b40a4dc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f61d01a2e40 0x7f61d01a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61d010cf80 0x7f61d019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 --2- 172.21.15.119:0/25608385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61d0107eb0 0x7f61d019e5d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 >> 172.21.15.119:0/25608385 conn(0x7f61d0075be0 msgr2=0x7f61d0076f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 shutdown_connections 2024-03-28T18:54:07.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.520+0000 7f61d5a00640 1 -- 172.21.15.119:0/25608385 wait complete. 2024-03-28T18:54:07.538 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:54:07.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-28T18:54:07.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 -- 172.21.15.119:0/1749419785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 msgr2=0x7f0f0c114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 --2- 172.21.15.119:0/1749419785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c114b50 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009ca0 tx=0x7f0f0402c100 comp rx=0 tx=0).stop 2024-03-28T18:54:07.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 -- 172.21.15.119:0/1749419785 shutdown_connections 2024-03-28T18:54:07.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 --2- 172.21.15.119:0/1749419785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c114b50 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 --2- 172.21.15.119:0/1749419785 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c108860 0x7f0f0c10c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 --2- 172.21.15.119:0/1749419785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 -- 172.21.15.119:0/1749419785 >> 172.21.15.119:0/1749419785 conn(0x7f0f0c075be0 msgr2=0x7f0f0c078000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 -- 172.21.15.119:0/1749419785 shutdown_connections 2024-03-28T18:54:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.744+0000 7f0f13e00640 1 -- 172.21.15.119:0/1749419785 wait complete. 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.745+0000 7f0f13e00640 1 Processor -- start 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.745+0000 7f0f13e00640 1 -- start start 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 0x7f0f0c1a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c118030 con 0x7f0f0c1a2da0 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c117eb0 con 0x7f0f0c107eb0 2024-03-28T18:54:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f13e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1181b0 con 0x7f0f0c10cf80 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 0x7f0f0c1a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48402/0 (socket says 172.21.15.119:48402) 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11400640 1 -- 172.21.15.119:0/3695424758 learned_addr learned my addr 172.21.15.119:0/3695424758 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 -- 172.21.15.119:0/3695424758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 msgr2=0x7f0f0c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 -- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 msgr2=0x7f0f0c1a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 0x7f0f0c1a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f10a00640 1 -- 172.21.15.119:0/3695424758 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c1a3890 con 0x7f0f0c10cf80 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11400640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.746+0000 7f0f11e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 0x7f0f0c1a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f10a00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c19ea70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00e9c0 tx=0x7f0efc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f0a000640 1 -- 172.21.15.119:0/3695424758 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc00cd80 con 0x7f0f0c10cf80 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f0a000640 1 -- 172.21.15.119:0/3695424758 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0efc0035f0 con 0x7f0f0c10cf80 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c1a3b80 con 0x7f0f0c10cf80 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f0a000640 1 -- 172.21.15.119:0/3695424758 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc010690 con 0x7f0f0c10cf80 2024-03-28T18:54:07.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.747+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c1ab3c0 con 0x7f0f0c10cf80 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.750+0000 7f0f0a000640 1 -- 172.21.15.119:0/3695424758 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f0efc0037a0 con 0x7f0f0c10cf80 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.750+0000 7f0f16c17740 1 Processor -- start 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 -- start start 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39573050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d394cb2c0 con 0x563d39570c00 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d39473c90 con 0x563d39568080 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f16c17740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d394c7360 con 0x563d395676d0 2024-03-28T18:54:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f11e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f11e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41820/0 (socket says 172.21.15.119:41820) 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f11e00640 1 -- 172.21.15.119:0/124973197 learned_addr learned my addr 172.21.15.119:0/124973197 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f11400640 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.751+0000 7f0f10a00640 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39573050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 -- 172.21.15.119:0/124973197 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 msgr2=0x563d39570630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 -- 172.21.15.119:0/124973197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 msgr2=0x563d39573050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39573050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 -- 172.21.15.119:0/124973197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d39498080 con 0x563d395676d0 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11e00640 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0f0402c710 tx=0x7f0f04005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5249cd858db8b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f0a000640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 0x7f0ef40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0efa600640 1 -- 172.21.15.119:0/124973197 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04043070 con 0x563d395676d0 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0efa600640 1 -- 172.21.15.119:0/124973197 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f04032c90 con 0x563d395676d0 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0efa600640 1 -- 172.21.15.119:0/124973197 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04039040 con 0x563d395676d0 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f0a000640 1 -- 172.21.15.119:0/3695424758 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f0efc0cf140 con 0x7f0f0c10cf80 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.752+0000 7f0f11400640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 0x7f0ef40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 -- 172.21.15.119:0/124973197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 msgr2=0x563d39567ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0f0402c710 tx=0x7f0f04005f80 comp rx=0 tx=0).stop 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f11400640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 0x7f0ef40a4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x563d39562e10 tx=0x7f0f00005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 -- 172.21.15.119:0/124973197 shutdown_connections 2024-03-28T18:54:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39573050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 --2- 172.21.15.119:0/124973197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39567ab0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 -- 172.21.15.119:0/124973197 >> 172.21.15.119:0/124973197 conn(0x563d39561da0 msgr2=0x563d39562180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 -- 172.21.15.119:0/124973197 shutdown_connections 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.753+0000 7f0f16c17740 1 -- 172.21.15.119:0/124973197 wait complete. 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 -- start start 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 Processor -- start 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 0x563d39570560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d3956d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 0x563d3956df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d39573b10 con 0x563d39568080 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d394870d0 con 0x563d395676d0 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d39573c90 con 0x563d39570c00 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.754+0000 7f0f16c17740 10 client.0 _ll_register_callbacks cb 0x563d3948be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 0x563d39570560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 0x563d3956df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 0x563d39570560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48428/0 (socket says 172.21.15.119:48428) 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11400640 1 -- 172.21.15.119:0/2687138616 learned_addr learned my addr 172.21.15.119:0/2687138616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d3956d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 -- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 msgr2=0x563d39570560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 0x563d39570560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 -- 172.21.15.119:0/2687138616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 msgr2=0x563d3956d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d3956d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 -- 172.21.15.119:0/2687138616 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d391df3f0 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f11e00640 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 0x563d3956df00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0f04037040 tx=0x7f0f040095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfaceac88423dcf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0eeea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0404c030 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f0f04032590 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.755+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f04032c90 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f04039070 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 10 client.28294 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563d392a97d0 con 0x563d39570c00 2024-03-28T18:54:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f04032c90 con 0x563d39570c00 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0f04004330 con 0x563d39570c00 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 10 client.28294 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 10 client.28294 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 20 client.28294 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395808f0 0x563d39582ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 4 client.28294 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.756+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d394d8800 con 0x563d395808f0 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.757+0000 7f0f11400640 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395808f0 0x563d39582ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.757+0000 7f0f11400640 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395808f0 0x563d39582ce0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=741d1e93f7125221 server_cookie=b329cf922f5181d in_seq=0 out_seq=0 2024-03-28T18:54:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.757+0000 7f0eeea00640 10 client.28294 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0eeea00640 1 -- 172.21.15.119:0/2687138616 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d394d8800 con 0x563d395808f0 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0eeea00640 10 client.28294 handle_command_reply: tid=0 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 client.28294 shutdown 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 msgr2=0x563d3956df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 0x563d3956df00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0f04037040 tx=0x7f0f040095c0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 shutdown_connections 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d39570c00 0x563d3956df00 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395808f0 0x563d39582ce0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d3956d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 --2- 172.21.15.119:0/2687138616 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d395676d0 0x563d39570560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0eeea00640 0 client.28294 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:54:07.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 >> 172.21.15.119:0/2687138616 conn(0x563d39561da0 msgr2=0x563d39563af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 shutdown_connections 2024-03-28T18:54:07.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 1 -- 172.21.15.119:0/2687138616 wait complete. 2024-03-28T18:54:07.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.758+0000 7f0f16c17740 20 client.28294 trim_cache size 0 max 16384 2024-03-28T18:54:07.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 Processor -- start 2024-03-28T18:54:07.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 -- start start 2024-03-28T18:54:07.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39570560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b39b0 con 0x563d39570c00 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b5330 con 0x563d39568080 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f16c17740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b4010 con 0x563d395676d0 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f11400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.772+0000 7f0f11400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41836/0 (socket says 172.21.15.119:41836) 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48440/0 (socket says 172.21.15.119:48440) 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 -- 172.21.15.119:0/459958839 learned_addr learned my addr 172.21.15.119:0/459958839 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11e00640 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39570560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 -- 172.21.15.119:0/459958839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 msgr2=0x563d39570020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 -- 172.21.15.119:0/459958839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 msgr2=0x563d39570560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39570560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 -- 172.21.15.119:0/459958839 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d39566990 con 0x563d395676d0 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0f11400640 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x563d39573c80 tx=0x7f0f0001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89bfa514e2e1cfe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0ef9c00640 1 -- 172.21.15.119:0/459958839 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00004020 con 0x563d395676d0 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0ef9c00640 1 -- 172.21.15.119:0/459958839 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f00005500 con 0x563d395676d0 2024-03-28T18:54:07.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.773+0000 7f0ef9c00640 1 -- 172.21.15.119:0/459958839 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000049d0 con 0x563d395676d0 2024-03-28T18:54:07.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- 172.21.15.119:0/459958839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 msgr2=0x563d3957cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x563d39573c80 tx=0x7f0f0001a000 comp rx=0 tx=0).stop 2024-03-28T18:54:07.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- 172.21.15.119:0/459958839 shutdown_connections 2024-03-28T18:54:07.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39570c00 0x563d39570560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39568080 0x563d39570020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 --2- 172.21.15.119:0/459958839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d3957cef0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- 172.21.15.119:0/459958839 >> 172.21.15.119:0/459958839 conn(0x563d39561da0 msgr2=0x563d39560eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- 172.21.15.119:0/459958839 shutdown_connections 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- 172.21.15.119:0/459958839 wait complete. 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.774+0000 7f0f16c17740 1 -- start start 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 Processor -- start 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39585700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 0x563d39583cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b4cd0 con 0x563d39568080 2024-03-28T18:54:07.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b4670 con 0x563d39570c00 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c1b1b90 con 0x563d395676d0 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f16c17740 10 client.0 _ll_register_callbacks cb 0x7f0f00009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f11400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39585700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f10a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49004/0 (socket says 172.21.15.119:49004) 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f11e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 0x563d39583cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f10a00640 1 -- 172.21.15.119:0/813975775 learned_addr learned my addr 172.21.15.119:0/813975775 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.775+0000 7f0f11400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39585700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41852/0 (socket says 172.21.15.119:41852) 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 -- 172.21.15.119:0/813975775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 msgr2=0x563d39585700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39585700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 -- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 msgr2=0x563d39583cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 0x563d39583cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 -- 172.21.15.119:0/813975775 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d395844c0 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f11e00640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 0x563d39583cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f10a00640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0efc098080 tx=0x7f0efc09b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa5c76c88f98e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0eeea00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc09b6b0 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f0efc097d20 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f00005500 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.776+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0efc09f800 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 10 client.18084 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563d395dcae0 con 0x563d39568080 2024-03-28T18:54:07.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f00005500 con 0x563d39568080 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0efc0b4390 con 0x563d39568080 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 10 client.18084 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 10 client.18084 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 20 client.18084 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395dd170 0x563d395df590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 4 client.18084 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 --> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563d394d2b50 con 0x563d395dd170 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.777+0000 7f0f11400640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395dd170 0x563d395df590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f11400640 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395dd170 0x563d395df590 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0616954d511b3af server_cookie=4e2f38b1c3aa8e9a in_seq=0 out_seq=0 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0eeea00640 10 client.18084 ms_handle_connect on v2:172.21.15.131:6836/2241845025 2024-03-28T18:54:07.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0eeea00640 1 -- 172.21.15.119:0/813975775 <== mds.3 v2:172.21.15.131:6836/2241845025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563d394d2b50 con 0x563d395dd170 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0eeea00640 10 client.18084 handle_command_reply: tid=0 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 client.18084 shutdown 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 msgr2=0x563d39585c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0efc098080 tx=0x7f0efc09b2b0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 shutdown_connections 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x563d39570c00 0x563d39583cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.131:6836/2241845025,v1:172.21.15.131:6837/2241845025] conn(0x563d395dd170 0x563d395df590 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563d39568080 0x563d39585c40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 --2- 172.21.15.119:0/813975775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563d395676d0 0x563d39585700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0eeea00640 0 client.18084 ms_handle_reset on v2:172.21.15.131:6836/2241845025 2024-03-28T18:54:07.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 >> 172.21.15.119:0/813975775 conn(0x563d39561da0 msgr2=0x563d395630a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 shutdown_connections 2024-03-28T18:54:07.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.778+0000 7f0f16c17740 1 -- 172.21.15.119:0/813975775 wait complete. 2024-03-28T18:54:07.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f16c17740 20 client.18084 trim_cache size 0 max 16384 2024-03-28T18:54:07.787 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 msgr2=0x7f0ef40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 0x7f0ef40a4eb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x563d39562e10 tx=0x7f0f00005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 msgr2=0x7f0f0c19ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c19ea70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00e9c0 tx=0x7f0efc00ee80 comp rx=0 tx=0).stop 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 shutdown_connections 2024-03-28T18:54:07.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f0ef40a29f0 0x7f0ef40a4eb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f0c1a2da0 0x7f0f0c1a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f0c10cf80 0x7f0f0c19ea70 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 --2- 172.21.15.119:0/3695424758 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0f0c107eb0 0x7f0f0c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 >> 172.21.15.119:0/3695424758 conn(0x7f0f0c075be0 msgr2=0x7f0f0c077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 shutdown_connections 2024-03-28T18:54:07.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:07.779+0000 7f0f13e00640 1 -- 172.21.15.119:0/3695424758 wait complete. 2024-03-28T18:54:07.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:07 smithi119 ceph-mon[31813]: from='client.28234 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:07 smithi119 ceph-mon[31813]: from='client.28240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:07 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/4088764972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:07.799 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:54:07.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-28T18:54:07.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:07 smithi196 ceph-mon[36300]: from='client.28234 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:07 smithi196 ceph-mon[36300]: from='client.28240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:07 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/4088764972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:07.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:07 smithi131 ceph-mon[36065]: from='client.28234 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:07 smithi131 ceph-mon[36065]: from='client.28240 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T18:54:07.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:07 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/4088764972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 -- 172.21.15.119:0/4266427411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768109850 msgr2=0x7f9768109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 --2- 172.21.15.119:0/4266427411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768109850 0x7f9768109c30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f975c009900 tx=0x7f975c02bc50 comp rx=0 tx=0).stop 2024-03-28T18:54:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 -- 172.21.15.119:0/4266427411 shutdown_connections 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 --2- 172.21.15.119:0/4266427411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768104200 0x7f9768110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 --2- 172.21.15.119:0/4266427411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f9768103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 --2- 172.21.15.119:0/4266427411 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768109850 0x7f9768109c30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 -- 172.21.15.119:0/4266427411 >> 172.21.15.119:0/4266427411 conn(0x7f97680ff2f0 msgr2=0x7f9768101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 -- 172.21.15.119:0/4266427411 shutdown_connections 2024-03-28T18:54:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.012+0000 7f976f400640 1 -- 172.21.15.119:0/4266427411 wait complete. 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 Processor -- start 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 -- start start 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 0x7f976819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 0x7f976819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768113c70 con 0x7f9768109850 2024-03-28T18:54:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768113af0 con 0x7f9768104200 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.013+0000 7f976f400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768113df0 con 0x7f9768103840 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f9767e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 0x7f976819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41870/0 (socket says 172.21.15.119:41870) 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 -- 172.21.15.119:0/3195359376 learned_addr learned my addr 172.21.15.119:0/3195359376 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976d400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 0x7f976819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 -- 172.21.15.119:0/3195359376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 msgr2=0x7f976819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 0x7f976819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 -- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 msgr2=0x7f976819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 0x7f976819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 -- 172.21.15.119:0/3195359376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976819f140 con 0x7f9768103840 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976d400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 0x7f976819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f9767e00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 0x7f976819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.014+0000 7f976ca00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f975c000c00 tx=0x7f975c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.015+0000 7f9765600640 1 -- 172.21.15.119:0/3195359376 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c041070 con 0x7f9768103840 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.015+0000 7f9765600640 1 -- 172.21.15.119:0/3195359376 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f975c02cce0 con 0x7f9768103840 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.015+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976819f3d0 con 0x7f9768103840 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.015+0000 7f9765600640 1 -- 172.21.15.119:0/3195359376 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c003d10 con 0x7f9768103840 2024-03-28T18:54:08.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.015+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97681a6cc0 con 0x7f9768103840 2024-03-28T18:54:08.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9765600640 1 -- 172.21.15.119:0/3195359376 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f975c044070 con 0x7f9768103840 2024-03-28T18:54:08.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 Processor -- start 2024-03-28T18:54:08.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 -- start start 2024-03-28T18:54:08.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd357c00 0x55dacd35a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd2b22c0 con 0x55dacd357c00 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd25ac90 con 0x55dacd34f080 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.018+0000 7f9772164740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd2ae360 con 0x55dacd34e6d0 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41882/0 (socket says 172.21.15.119:41882) 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 -- 172.21.15.119:0/1950012771 learned_addr learned my addr 172.21.15.119:0/1950012771 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976ca00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48484/0 (socket says 172.21.15.119:48484) 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f9767e00640 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd357c00 0x55dacd35a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 -- 172.21.15.119:0/1950012771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 msgr2=0x55dacd357630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 -- 172.21.15.119:0/1950012771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd357c00 msgr2=0x55dacd35a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd357c00 0x55dacd35a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 -- 172.21.15.119:0/1950012771 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dacd27f080 con 0x55dacd34e6d0 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976ca00640 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f976d400640 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9760009fd0 tx=0x7f9760010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=551392f09bdae8e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.019+0000 7f9765600640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 0x7f97500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9759c00640 1 -- 172.21.15.119:0/1950012771 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760011700 con 0x55dacd34e6d0 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9759c00640 1 -- 172.21.15.119:0/1950012771 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9760011d80 con 0x55dacd34e6d0 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9759c00640 1 -- 172.21.15.119:0/1950012771 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976000e3c0 con 0x55dacd34e6d0 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9765600640 1 -- 172.21.15.119:0/3195359376 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f975c0f08d0 con 0x7f9768103840 2024-03-28T18:54:08.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f976ca00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 0x7f97500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9772164740 1 -- 172.21.15.119:0/1950012771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 msgr2=0x55dacd34eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f9772164740 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9760009fd0 tx=0x7f9760010860 comp rx=0 tx=0).stop 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.020+0000 7f976ca00640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 0x7f97500a4e30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55dacd34e1a0 tx=0x7f975c0f6530 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 -- 172.21.15.119:0/1950012771 shutdown_connections 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd357c00 0x55dacd35a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34f080 0x55dacd357630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 --2- 172.21.15.119:0/1950012771 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34e6d0 0x55dacd34eab0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 -- 172.21.15.119:0/1950012771 >> 172.21.15.119:0/1950012771 conn(0x55dacd348da0 msgr2=0x55dacd349180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 -- 172.21.15.119:0/1950012771 shutdown_connections 2024-03-28T18:54:08.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 -- 172.21.15.119:0/1950012771 wait complete. 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 -- start start 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.021+0000 7f9772164740 1 Processor -- start 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd354880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd354e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd35ab70 con 0x55dacd34f080 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd26e0d0 con 0x55dacd34e6d0 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacd35acf0 con 0x55dacd357c00 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9772164740 10 client.0 _ll_register_callbacks cb 0x55dacd272e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48494/0 (socket says 172.21.15.119:48494) 2024-03-28T18:54:08.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 -- 172.21.15.119:0/3874545850 learned_addr learned my addr 172.21.15.119:0/3874545850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd354e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9767e00640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd354880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 -- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 msgr2=0x55dacd354e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd354e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 -- 172.21.15.119:0/3874545850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 msgr2=0x55dacd354880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd354880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f976d400640 1 -- 172.21.15.119:0/3874545850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55daccfc63f0 con 0x55dacd34e6d0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.022+0000 7f9767e00640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd354880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f976d400640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f976001bc90 tx=0x7f9760013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed6bf042dd39a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f974e000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976002f060 con 0x55dacd34e6d0 2024-03-28T18:54:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f9760011730 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9760011d80 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9760036050 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f9772164740 10 client.28331 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dacd0907d0 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.023+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9760011d80 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9760029040 con 0x55dacd34e6d0 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 10 client.28331 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 10 client.28331 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 20 client.28331 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd367950 0x55dacd369d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 4 client.28331 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dacd2bf800 con 0x55dacd367950 2024-03-28T18:54:08.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9767e00640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd367950 0x55dacd369d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f9767e00640 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd367950 0x55dacd369d40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae1e99e6fd9c7191 server_cookie=8385342c2e35eb2b in_seq=0 out_seq=0 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.024+0000 7f974e000640 10 client.28331 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f974e000640 1 -- 172.21.15.119:0/3874545850 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dacd2bf800 con 0x55dacd367950 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f974e000640 10 client.28331 handle_command_reply: tid=0 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 client.28331 shutdown 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 msgr2=0x55dacd3574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f976001bc90 tx=0x7f9760013040 comp rx=0 tx=0).stop 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 shutdown_connections 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd367950 0x55dacd369d40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd354e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd354880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 --2- 172.21.15.119:0/3874545850 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd3574f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f974e000640 0 client.28331 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:54:08.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 >> 172.21.15.119:0/3874545850 conn(0x55dacd348da0 msgr2=0x55dacd34aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 shutdown_connections 2024-03-28T18:54:08.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 1 -- 172.21.15.119:0/3874545850 wait complete. 2024-03-28T18:54:08.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.026+0000 7f9772164740 20 client.28331 trim_cache size 0 max 16384 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.039+0000 7f9772164740 1 Processor -- start 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 -- start start 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 0x55dacd356fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd357c00 0x55dacd351b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681af680 con 0x55dacd34e6d0 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681b1000 con 0x55dacd357c00 2024-03-28T18:54:08.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9772164740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681afce0 con 0x55dacd34f080 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9767e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9767e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41892/0 (socket says 172.21.15.119:41892) 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f976d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 0x55dacd356fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f9767e00640 1 -- 172.21.15.119:0/556184805 learned_addr learned my addr 172.21.15.119:0/556184805 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.040+0000 7f976d400640 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd357c00 0x55dacd351b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 -- 172.21.15.119:0/556184805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd357c00 msgr2=0x55dacd351b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd357c00 0x55dacd351b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 -- 172.21.15.119:0/556184805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 msgr2=0x55dacd356fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 0x55dacd356fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 -- 172.21.15.119:0/556184805 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dacd349970 con 0x55dacd34f080 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f976d400640 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 0x55dacd356fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9767e00640 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55dacd355e50 tx=0x7f9754005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c624eab82e4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9759200640 1 -- 172.21.15.119:0/556184805 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754026070 con 0x55dacd34f080 2024-03-28T18:54:08.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9759200640 1 -- 172.21.15.119:0/556184805 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9754011b50 con 0x55dacd34f080 2024-03-28T18:54:08.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.041+0000 7f9759200640 1 -- 172.21.15.119:0/556184805 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754003b50 con 0x55dacd34f080 2024-03-28T18:54:08.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- 172.21.15.119:0/556184805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 msgr2=0x55dacd3574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55dacd355e50 tx=0x7f9754005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:08.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- 172.21.15.119:0/556184805 shutdown_connections 2024-03-28T18:54:08.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd357c00 0x55dacd351b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd34f080 0x55dacd3574f0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 --2- 172.21.15.119:0/556184805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34e6d0 0x55dacd356fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- 172.21.15.119:0/556184805 >> 172.21.15.119:0/556184805 conn(0x55dacd348da0 msgr2=0x55dacd347eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- 172.21.15.119:0/556184805 shutdown_connections 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- 172.21.15.119:0/556184805 wait complete. 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 -- start start 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.042+0000 7f9772164740 1 Processor -- start 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd35dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd3c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681b09a0 con 0x55dacd34f080 2024-03-28T18:54:08.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681b0340 con 0x55dacd34e6d0 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97681ad860 con 0x55dacd357c00 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9772164740 10 client.0 _ll_register_callbacks cb 0x7f975c0f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f9767e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd35dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48506/0 (socket says 172.21.15.119:48506) 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 -- 172.21.15.119:0/437082109 learned_addr learned my addr 172.21.15.119:0/437082109 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd3c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 -- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 msgr2=0x55dacd3c1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd3c1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 -- 172.21.15.119:0/437082109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 msgr2=0x55dacd35dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd35dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.043+0000 7f976d400640 1 -- 172.21.15.119:0/437082109 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dacd3c2510 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f976d400640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9760002230 tx=0x7f976001bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54af5f06b8536a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f974e000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976002f040 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9754011b50 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f976000e3c0 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97600099e0 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f9772164740 10 client.28337 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dacd3c3710 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.044+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9754011b50 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9760029480 con 0x55dacd34e6d0 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 10 client.28337 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 10 client.28337 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-28T18:54:08.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 20 client.28337 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd3c3c70 0x55dacd3c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 4 client.28337 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 --> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dacd2b9b50 con 0x55dacd3c3c70 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9767e00640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd3c3c70 0x55dacd3c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f9767e00640 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd3c3c70 0x55dacd3c6060 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a80f61a67377278 server_cookie=fc4e476afd303ade in_seq=0 out_seq=0 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.045+0000 7f974e000640 10 client.28337 ms_handle_connect on v2:172.21.15.196:6836/4226502296 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f974e000640 1 -- 172.21.15.119:0/437082109 <== mds.2 v2:172.21.15.196:6836/4226502296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dacd2b9b50 con 0x55dacd3c3c70 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f974e000640 10 client.28337 handle_command_reply: tid=0 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 client.28337 shutdown 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 msgr2=0x55dacd35d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9760002230 tx=0x7f976001bcc0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 shutdown_connections 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:6836/4226502296,v1:172.21.15.196:6837/4226502296] conn(0x55dacd3c3c70 0x55dacd3c6060 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55dacd357c00 0x55dacd3c1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dacd34f080 0x55dacd35dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 --2- 172.21.15.119:0/437082109 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55dacd34e6d0 0x55dacd35d980 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f974e000640 0 client.28337 ms_handle_reset on v2:172.21.15.196:6836/4226502296 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 >> 172.21.15.119:0/437082109 conn(0x55dacd348da0 msgr2=0x55dacd34a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 shutdown_connections 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 1 -- 172.21.15.119:0/437082109 wait complete. 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.046+0000 7f9772164740 20 client.28337 trim_cache size 0 max 16384 2024-03-28T18:54:08.053 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:54:08.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 msgr2=0x7f97500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 0x7f97500a4e30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55dacd34e1a0 tx=0x7f975c0f6530 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 msgr2=0x7f976819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f975c000c00 tx=0x7f975c0042f0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 shutdown_connections 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f97500a2970 0x7f97500a4e30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9768109850 0x7f976819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9768104200 0x7f976819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 --2- 172.21.15.119:0/3195359376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9768103840 0x7f976819a150 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 >> 172.21.15.119:0/3195359376 conn(0x7f97680ff2f0 msgr2=0x7f97680ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 shutdown_connections 2024-03-28T18:54:08.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.047+0000 7f976f400640 1 -- 172.21.15.119:0/3195359376 wait complete. 2024-03-28T18:54:08.065 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:54:08.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-28T18:54:08.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 -- 172.21.15.119:0/1924773554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d4075bd0 msgr2=0x7f36d4075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 --2- 172.21.15.119:0/1924773554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d4075bd0 0x7f36d4075fb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009930 tx=0x7f36c402bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 -- 172.21.15.119:0/1924773554 shutdown_connections 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 --2- 172.21.15.119:0/1924773554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d41124e0 0x7f36d41148d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 --2- 172.21.15.119:0/1924773554 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d40764f0 0x7f36d4111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 --2- 172.21.15.119:0/1924773554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d4075bd0 0x7f36d4075fb0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 -- 172.21.15.119:0/1924773554 >> 172.21.15.119:0/1924773554 conn(0x7f36d40ff2f0 msgr2=0x7f36d4101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 -- 172.21.15.119:0/1924773554 shutdown_connections 2024-03-28T18:54:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.280+0000 7f36dac00640 1 -- 172.21.15.119:0/1924773554 wait complete. 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.281+0000 7f36dac00640 1 Processor -- start 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 -- start start 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 0x7f36d419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4118090 con 0x7f36d4075bd0 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4117f10 con 0x7f36d41124e0 2024-03-28T18:54:08.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36dac00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4118210 con 0x7f36d40764f0 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 0x7f36d419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36d8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41918/0 (socket says 172.21.15.119:41918) 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 0x7f36d419e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49078/0 (socket says 172.21.15.119:49078) 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36cbe00640 1 -- 172.21.15.119:0/2433983601 learned_addr learned my addr 172.21.15.119:0/2433983601 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.282+0000 7f36d8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48528/0 (socket says 172.21.15.119:48528) 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 -- 172.21.15.119:0/2433983601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 msgr2=0x7f36d419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 -- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 msgr2=0x7f36d419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 0x7f36d419e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 -- 172.21.15.119:0/2433983601 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d41a3620 con 0x7f36d41124e0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36cbe00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d8c00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f36cc005fd0 tx=0x7f36cc00edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d1600640 1 -- 172.21.15.119:0/2433983601 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc004d20 con 0x7f36d41124e0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d1600640 1 -- 172.21.15.119:0/2433983601 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36cc005020 con 0x7f36d41124e0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36d1600640 1 -- 172.21.15.119:0/2433983601 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0116d0 con 0x7f36d41124e0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.283+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d41a3910 con 0x7f36d41124e0 2024-03-28T18:54:08.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.284+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d41ab1a0 con 0x7f36d41124e0 2024-03-28T18:54:08.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 Processor -- start 2024-03-28T18:54:08.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36d1600640 1 -- 172.21.15.119:0/2433983601 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f36cc005540 con 0x7f36d41124e0 2024-03-28T18:54:08.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 -- start start 2024-03-28T18:54:08.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab4099dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab409012c0 con 0x55ab4099e080 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab408a9c90 con 0x55ab409a6c00 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.287+0000 7f36dd998740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab408fd360 con 0x55ab4099d6d0 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab4099dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49086/0 (socket says 172.21.15.119:49086) 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 -- 172.21.15.119:0/88925646 learned_addr learned my addr 172.21.15.119:0/88925646 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36cbe00640 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 -- 172.21.15.119:0/88925646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 msgr2=0x55ab4099dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab4099dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 -- 172.21.15.119:0/88925646 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 msgr2=0x55ab409a9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d3e00640 1 -- 172.21.15.119:0/88925646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab408ce080 con 0x55ab4099e080 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d1600640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 0x7f36b80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36cbe00640 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.288+0000 7f36d1600640 1 -- 172.21.15.119:0/2433983601 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f36cc0ceeb0 con 0x7f36d41124e0 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.289+0000 7f36d3e00640 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005ab0 tx=0x7f36c4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f27a775ac29e6d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.289+0000 7f36cbe00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 0x7f36b80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.289+0000 7f36c9200640 1 -- 172.21.15.119:0/88925646 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4045070 con 0x55ab4099e080 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.289+0000 7f36c9200640 1 -- 172.21.15.119:0/88925646 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36c402c640 con 0x55ab4099e080 2024-03-28T18:54:08.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.289+0000 7f36c9200640 1 -- 172.21.15.119:0/88925646 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4004b20 con 0x55ab4099e080 2024-03-28T18:54:08.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- 172.21.15.119:0/88925646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 msgr2=0x55ab409a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36cbe00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 0x7f36b80a4eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009fd0 tx=0x7f36bc004730 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f36c4005ab0 tx=0x7f36c4009560 comp rx=0 tx=0).stop 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- 172.21.15.119:0/88925646 shutdown_connections 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409a6630 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 --2- 172.21.15.119:0/88925646 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab4099dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- 172.21.15.119:0/88925646 >> 172.21.15.119:0/88925646 conn(0x55ab40997da0 msgr2=0x55ab40998180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- 172.21.15.119:0/88925646 shutdown_connections 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- 172.21.15.119:0/88925646 wait complete. 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.290+0000 7f36dd998740 1 -- start start 2024-03-28T18:54:08.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 Processor -- start 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab409a9b10 con 0x55ab4099d6d0 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab408bd0d0 con 0x55ab409a6c00 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab409a9c90 con 0x55ab4099e080 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36dd998740 10 client.0 _ll_register_callbacks cb 0x55ab408c1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36d8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41930/0 (socket says 172.21.15.119:41930) 2024-03-28T18:54:08.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49098/0 (socket says 172.21.15.119:49098) 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.291+0000 7f36cbe00640 1 -- 172.21.15.119:0/1377881668 learned_addr learned my addr 172.21.15.119:0/1377881668 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 -- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 msgr2=0x55ab409a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 -- 172.21.15.119:0/1377881668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 msgr2=0x55ab409a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 -- 172.21.15.119:0/1377881668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab406153f0 con 0x55ab409a6c00 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36cbe00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36d8c00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a3e10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0cc510 tx=0x7f36cc09b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4a4f399e070b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36c1600640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc01f040 con 0x55ab409a6c00 2024-03-28T18:54:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f36cc09a920 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36c402c640 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36cc09b4d0 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.292+0000 7f36dd998740 10 client.28349 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ab406df7d0 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36c402c640 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f36cc00f040 con 0x55ab409a6c00 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 10 client.28349 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 10 client.28349 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 20 client.28349 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab409b6af0 0x55ab409b8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 4 client.28349 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab4090e800 con 0x55ab409b6af0 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.293+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab409b6af0 0x55ab409b8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.294+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab409b6af0 0x55ab409b8ee0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23c637b4c5ce1fab server_cookie=89f9dffdbfdbd0d9 in_seq=0 out_seq=0 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.294+0000 7f36c1600640 10 client.28349 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.294+0000 7f36c1600640 1 -- 172.21.15.119:0/1377881668 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ab4090e800 con 0x55ab409b6af0 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.294+0000 7f36c1600640 10 client.28349 handle_command_reply: tid=0 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 client.28349 shutdown 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 msgr2=0x55ab409a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a3e10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0cc510 tx=0x7f36cc09b360 comp rx=0 tx=0).stop 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 shutdown_connections 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab409b6af0 0x55ab409b8ee0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409a3e10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 --2- 172.21.15.119:0/1377881668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099d6d0 0x55ab409a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36c1600640 0 client.28349 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 >> 172.21.15.119:0/1377881668 conn(0x55ab40997da0 msgr2=0x55ab40999af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 shutdown_connections 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 1 -- 172.21.15.119:0/1377881668 wait complete. 2024-03-28T18:54:08.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.295+0000 7f36dd998740 20 client.28349 trim_cache size 0 max 16384 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 Processor -- start 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 -- start start 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab409a6c00 0x55ab409a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b39f0 con 0x55ab409a6c00 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b5370 con 0x55ab4099d6d0 2024-03-28T18:54:08.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.309+0000 7f36dd998740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b4050 con 0x55ab4099e080 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48568/0 (socket says 172.21.15.119:48568) 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 -- 172.21.15.119:0/1874539803 learned_addr learned my addr 172.21.15.119:0/1874539803 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d8c00640 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab409a6c00 0x55ab409a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36cbe00640 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 -- 172.21.15.119:0/1874539803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 msgr2=0x55ab409a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 -- 172.21.15.119:0/1874539803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab409a6c00 msgr2=0x55ab409a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab409a6c00 0x55ab409a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 -- 172.21.15.119:0/1874539803 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab40998e40 con 0x55ab4099d6d0 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.310+0000 7f36d3e00640 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f36c402c810 tx=0x7f36c402bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e625b68bd369f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36c3e00640 1 -- 172.21.15.119:0/1874539803 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4051070 con 0x55ab4099d6d0 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36c3e00640 1 -- 172.21.15.119:0/1874539803 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36c402c5c0 con 0x55ab4099d6d0 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36c3e00640 1 -- 172.21.15.119:0/1874539803 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4045040 con 0x55ab4099d6d0 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 -- 172.21.15.119:0/1874539803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 msgr2=0x55ab409a5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f36c402c810 tx=0x7f36c402bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 -- 172.21.15.119:0/1874539803 shutdown_connections 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab409a6c00 0x55ab409a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099e080 0x55ab409a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 --2- 172.21.15.119:0/1874539803 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab4099d6d0 0x55ab409a5fc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 -- 172.21.15.119:0/1874539803 >> 172.21.15.119:0/1874539803 conn(0x55ab40997da0 msgr2=0x55ab40996eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 -- 172.21.15.119:0/1874539803 shutdown_connections 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.311+0000 7f36dd998740 1 -- 172.21.15.119:0/1874539803 wait complete. 2024-03-28T18:54:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 -- start start 2024-03-28T18:54:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 Processor -- start 2024-03-28T18:54:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409acad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b4d10 con 0x55ab4099e080 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b46b0 con 0x55ab409a6c00 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41b1bd0 con 0x55ab4099d6d0 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.312+0000 7f36dd998740 10 client.0 _ll_register_callbacks cb 0x7f36cc005b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36cbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d8c00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41960/0 (socket says 172.21.15.119:41960) 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 -- 172.21.15.119:0/2223080459 learned_addr learned my addr 172.21.15.119:0/2223080459 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 -- 172.21.15.119:0/2223080459 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 msgr2=0x55ab409ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 -- 172.21.15.119:0/2223080459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 msgr2=0x55ab409acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 -- 172.21.15.119:0/2223080459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ab40a11560 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d8c00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36d3e00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006010 tx=0x7f36c40095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5df6de467debb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.313+0000 7f36c1600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c405b060 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36c402c5c0 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f36c40050e0 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36c4045070 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 10 client.28336 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ab40a12860 con 0x55ab4099d6d0 2024-03-28T18:54:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36c402c5c0 con 0x55ab4099d6d0 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f36c40569a0 con 0x55ab4099d6d0 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 10 client.28336 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 10 client.28336 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 20 client.28336 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab40a12b50 0x55ab40a14f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 4 client.28336 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.314+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 --> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ab40908b50 con 0x55ab40a12b50 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36cbe00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab40a12b50 0x55ab40a14f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36cbe00640 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab40a12b50 0x55ab40a14f40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef13088358df87e1 server_cookie=8047f26665d8929c in_seq=0 out_seq=0 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36c1600640 10 client.28336 ms_handle_connect on v2:172.21.15.196:6838/2667663000 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36c1600640 1 -- 172.21.15.119:0/2223080459 <== mds.1 v2:172.21.15.196:6838/2667663000 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ab40908b50 con 0x55ab40a12b50 2024-03-28T18:54:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36c1600640 10 client.28336 handle_command_reply: tid=0 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.315+0000 7f36dd998740 1 client.28336 shutdown 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 msgr2=0x55ab409ac590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006010 tx=0x7f36c40095e0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 shutdown_connections 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:6838/2667663000,v1:172.21.15.196:6839/2667663000] conn(0x55ab40a12b50 0x55ab40a14f40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55ab409a6c00 0x55ab409ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ab4099e080 0x55ab409acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36c1600640 0 client.28336 ms_handle_reset on v2:172.21.15.196:6838/2667663000 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 --2- 172.21.15.119:0/2223080459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ab4099d6d0 0x55ab409ac590 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 >> 172.21.15.119:0/2223080459 conn(0x55ab40997da0 msgr2=0x55ab4099e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 shutdown_connections 2024-03-28T18:54:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 1 -- 172.21.15.119:0/2223080459 wait complete. 2024-03-28T18:54:08.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dd998740 20 client.28336 trim_cache size 0 max 16384 2024-03-28T18:54:08.323 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 msgr2=0x7f36b80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 0x7f36b80a4eb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009fd0 tx=0x7f36bc004730 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.316+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 msgr2=0x7f36d41a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f36cc005fd0 tx=0x7f36cc00edd0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 shutdown_connections 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f36b80a29f0 0x7f36b80a4eb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f36d41124e0 0x7f36d41a2ef0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36d40764f0 0x7f36d419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 --2- 172.21.15.119:0/2433983601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4075bd0 0x7f36d419e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 >> 172.21.15.119:0/2433983601 conn(0x7f36d40ff2f0 msgr2=0x7f36d4113150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 shutdown_connections 2024-03-28T18:54:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.317+0000 7f36dac00640 1 -- 172.21.15.119:0/2433983601 wait complete. 2024-03-28T18:54:08.334 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-28T18:54:08.335 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 -- 172.21.15.119:0/1346912233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc10aa50 msgr2=0x7f94bc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 --2- 172.21.15.119:0/1346912233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc10aa50 0x7f94bc10ea90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009900 tx=0x7f94a802bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 -- 172.21.15.119:0/1346912233 shutdown_connections 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 --2- 172.21.15.119:0/1346912233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10f170 0x7f94bc116d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 --2- 172.21.15.119:0/1346912233 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc10aa50 0x7f94bc10ea90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 --2- 172.21.15.119:0/1346912233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10a0a0 0x7f94bc10a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.526+0000 7f94c3200640 1 -- 172.21.15.119:0/1346912233 >> 172.21.15.119:0/1346912233 conn(0x7f94bc072c70 msgr2=0x7f94bc073080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.527+0000 7f94c3200640 1 -- 172.21.15.119:0/1346912233 shutdown_connections 2024-03-28T18:54:08.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.527+0000 7f94c3200640 1 -- 172.21.15.119:0/1346912233 wait complete. 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 Processor -- start 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 -- start start 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10a0a0 0x7f94bc1a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 0x7f94bc1a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc11a230 con 0x7f94bc10a0a0 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc11a0b0 con 0x7f94bc1a27d0 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.528+0000 7f94c3200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc11a3b0 con 0x7f94bc10f170 2024-03-28T18:54:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10a0a0 0x7f94bc1a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94c1200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 0x7f94bc1a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:41974/0 (socket says 172.21.15.119:41974) 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 -- 172.21.15.119:0/2589698114 learned_addr learned my addr 172.21.15.119:0/2589698114 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 -- 172.21.15.119:0/2589698114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 msgr2=0x7f94bc1a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 0x7f94bc1a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 -- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10a0a0 msgr2=0x7f94bc1a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10a0a0 0x7f94bc1a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 -- 172.21.15.119:0/2589698114 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc1a3360 con 0x7f94bc10f170 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94c1200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 0x7f94bc1a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.529+0000 7f94bb400640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94a802c240 tx=0x7f94a8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.532+0000 7f94b8c00640 1 -- 172.21.15.119:0/2589698114 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8043020 con 0x7f94bc10f170 2024-03-28T18:54:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.532+0000 7f94b8c00640 1 -- 172.21.15.119:0/2589698114 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94a80054e0 con 0x7f94bc10f170 2024-03-28T18:54:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.532+0000 7f94b8c00640 1 -- 172.21.15.119:0/2589698114 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8003bb0 con 0x7f94bc10f170 2024-03-28T18:54:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.532+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc1a35f0 con 0x7f94bc10f170 2024-03-28T18:54:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.533+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc1af7d0 con 0x7f94bc10f170 2024-03-28T18:54:08.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.535+0000 7f94b8c00640 1 -- 172.21.15.119:0/2589698114 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f94a8046020 con 0x7f94bc10f170 2024-03-28T18:54:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.536+0000 7f94c5f85740 1 Processor -- start 2024-03-28T18:54:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 -- start start 2024-03-28T18:54:08.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1527ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba1533050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba148b2c0 con 0x55fba15276d0 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba1433c90 con 0x55fba1528080 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c5f85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba1487360 con 0x55fba1530c00 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48600/0 (socket says 172.21.15.119:48600) 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94bbe00640 1 -- 172.21.15.119:0/557053442 learned_addr learned my addr 172.21.15.119:0/557053442 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba1533050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.537+0000 7f94c1200640 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1527ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94b8c00640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 0x7f94a00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 -- 172.21.15.119:0/557053442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 msgr2=0x55fba1533050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba1533050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 -- 172.21.15.119:0/557053442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 msgr2=0x55fba1527ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1527ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 -- 172.21.15.119:0/557053442 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba1458080 con 0x55fba1528080 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94b8c00640 1 -- 172.21.15.119:0/2589698114 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f94a80f0d50 con 0x7f94bc10f170 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94c1200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 0x7f94a00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.538+0000 7f94bbe00640 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009cf0 tx=0x7f94ac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ef42eb146a5b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.539+0000 7f94c1200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 0x7f94a00a4eb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94b40094d0 tx=0x7f94b4014040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.540+0000 7f94b1600640 1 -- 172.21.15.119:0/557053442 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac01e070 con 0x55fba1528080 2024-03-28T18:54:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.540+0000 7f94b1600640 1 -- 172.21.15.119:0/557053442 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94ac019b10 con 0x55fba1528080 2024-03-28T18:54:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.540+0000 7f94b1600640 1 -- 172.21.15.119:0/557053442 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00ea60 con 0x55fba1528080 2024-03-28T18:54:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 -- 172.21.15.119:0/557053442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 msgr2=0x55fba1530630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009cf0 tx=0x7f94ac011810 comp rx=0 tx=0).stop 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 -- 172.21.15.119:0/557053442 shutdown_connections 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba1533050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba1530630 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 --2- 172.21.15.119:0/557053442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1527ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.541+0000 7f94c5f85740 1 -- 172.21.15.119:0/557053442 >> 172.21.15.119:0/557053442 conn(0x55fba1521da0 msgr2=0x55fba1522180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.542+0000 7f94c5f85740 1 -- 172.21.15.119:0/557053442 shutdown_connections 2024-03-28T18:54:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.542+0000 7f94c5f85740 1 -- 172.21.15.119:0/557053442 wait complete. 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.543+0000 7f94c5f85740 1 -- start start 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.543+0000 7f94c5f85740 1 Processor -- start 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1530510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba152de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba1533b70 con 0x55fba15276d0 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba14470d0 con 0x55fba1528080 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba1533cf0 con 0x55fba1530c00 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c5f85740 10 client.0 _ll_register_callbacks cb 0x55fba144be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94bb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94bb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48602/0 (socket says 172.21.15.119:48602) 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94bb400640 1 -- 172.21.15.119:0/1550084533 learned_addr learned my addr 172.21.15.119:0/1550084533 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94bbe00640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1530510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.544+0000 7f94c1200640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba152de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 -- 172.21.15.119:0/1550084533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 msgr2=0x55fba152de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba152de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 -- 172.21.15.119:0/1550084533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 msgr2=0x55fba1530510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1530510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 -- 172.21.15.119:0/1550084533 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba119f3f0 con 0x55fba1528080 2024-03-28T18:54:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94bb400640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f94a80b9830 tx=0x7f94a8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18325a7042d6107e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.545+0000 7f94a6000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004c80 con 0x55fba1528080 2024-03-28T18:54:08.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8710+0+0 (secure 0 0 0) 0x7f94a80b9a40 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94ac019b10 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a8005690 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94c5f85740 10 client.28367 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.546+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fba12697d0 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.547+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94ac019b10 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.547+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f94a80ee630 con 0x55fba1528080 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.547+0000 7f94c5f85740 10 client.28367 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.547+0000 7f94c5f85740 10 client.28367 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94c5f85740 20 client.28367 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba1549280 0x55fba154b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94c5f85740 4 client.28367 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T18:54:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fba1498800 con 0x55fba1549280 2024-03-28T18:54:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94bbe00640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba1549280 0x55fba154b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94bbe00640 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba1549280 0x55fba154b6a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1643a0ff0aef5f33 server_cookie=bd727be058804b8 in_seq=0 out_seq=0 2024-03-28T18:54:08.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.548+0000 7f94a6000640 10 client.28367 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:08.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.549+0000 7f94a6000640 1 -- 172.21.15.119:0/1550084533 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fba1498800 con 0x55fba1549280 2024-03-28T18:54:08.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.549+0000 7f94a6000640 10 client.28367 handle_command_reply: tid=0 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.549+0000 7f94c5f85740 1 client.28367 shutdown 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 msgr2=0x55fba152d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f94a80b9830 tx=0x7f94a8004610 comp rx=0 tx=0).stop 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 shutdown_connections 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba1549280 0x55fba154b6a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94a6000640 0 client.28367 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba152de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba152d8a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1550084533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba1530510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.550+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 >> 172.21.15.119:0/1550084533 conn(0x55fba1521da0 msgr2=0x55fba1523af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.551+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 shutdown_connections 2024-03-28T18:54:08.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.551+0000 7f94c5f85740 1 -- 172.21.15.119:0/1550084533 wait complete. 2024-03-28T18:54:08.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.551+0000 7f94c5f85740 20 client.28367 trim_cache size 0 max 16384 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.564+0000 7f94c5f85740 1 Processor -- start 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.564+0000 7f94c5f85740 1 -- start start 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba1530c00 0x55fba1530510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b3a40 con 0x55fba1530c00 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b53c0 con 0x55fba15276d0 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c5f85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b40a0 con 0x55fba1528080 2024-03-28T18:54:08.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48616/0 (socket says 172.21.15.119:48616) 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:42006/0 (socket says 172.21.15.119:42006) 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 -- 172.21.15.119:0/925456776 learned_addr learned my addr 172.21.15.119:0/925456776 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94c1200640 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba1530c00 0x55fba1530510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 -- 172.21.15.119:0/925456776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 msgr2=0x55fba152ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 -- 172.21.15.119:0/925456776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba1530c00 msgr2=0x55fba1530510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba1530c00 0x55fba1530510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bbe00640 1 -- 172.21.15.119:0/925456776 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba152e1b0 con 0x55fba15276d0 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.565+0000 7f94bb400640 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:08.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.566+0000 7f94bbe00640 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009c40 tx=0x7f94ac009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8cc5707fb9d0b0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.566+0000 7f94b0c00640 1 -- 172.21.15.119:0/925456776 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac01e040 con 0x55fba15276d0 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.566+0000 7f94b0c00640 1 -- 172.21.15.119:0/925456776 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94ac019430 con 0x55fba15276d0 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.566+0000 7f94b0c00640 1 -- 172.21.15.119:0/925456776 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00e9b0 con 0x55fba15276d0 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- 172.21.15.119:0/925456776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 msgr2=0x55fba1523820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009c40 tx=0x7f94ac009370 comp rx=0 tx=0).stop 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- 172.21.15.119:0/925456776 shutdown_connections 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba1530c00 0x55fba1530510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1528080 0x55fba152ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 --2- 172.21.15.119:0/925456776 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba15276d0 0x55fba1523820 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- 172.21.15.119:0/925456776 >> 172.21.15.119:0/925456776 conn(0x55fba1521da0 msgr2=0x55fba1523110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- 172.21.15.119:0/925456776 shutdown_connections 2024-03-28T18:54:08.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- 172.21.15.119:0/925456776 wait complete. 2024-03-28T18:54:08.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.567+0000 7f94c5f85740 1 -- start start 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 Processor -- start 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba153cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba153d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b4d60 con 0x55fba15276d0 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b4700 con 0x55fba1528080 2024-03-28T18:54:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1b1c80 con 0x55fba1530c00 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.568+0000 7f94c5f85740 10 client.0 _ll_register_callbacks cb 0x7f94a80b9670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94c1200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba153d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba153cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:48632/0 (socket says 172.21.15.119:48632) 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 -- 172.21.15.119:0/1923814751 learned_addr learned my addr 172.21.15.119:0/1923814751 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 -- 172.21.15.119:0/1923814751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 msgr2=0x55fba153d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba153d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 -- 172.21.15.119:0/1923814751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 msgr2=0x55fba153cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba153cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 -- 172.21.15.119:0/1923814751 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba15a3d00 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94bb400640 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f94a80c1520 tx=0x7f94a80035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3965731d8168fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94a6000640 10 client.0 ms_handle_connect on v2:172.21.15.131:3300/0 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a8004c80 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94ac019430 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.569+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mon.2 v2:172.21.15.131:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8710+0+0 (secure 0 0 0) 0x7f94a80c5730 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94a80be700 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 10 client.28373 fetch_fsmap learned FSMap version 28 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fba15a4fc0 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mon.2 v2:172.21.15.131:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94ac019430 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mon.2 v2:172.21.15.131:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f94a80dd0b0 con 0x55fba1528080 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 10 client.28373 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T18:54:08.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 10 client.28373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 20 client.28373 populate_metadata read hostname 'smithi119' 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba15a5500 0x55fba15a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 4 client.28373 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 --> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fba1492b50 con 0x55fba15a5500 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.570+0000 7f94bbe00640 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba15a5500 0x55fba15a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94a6000640 10 client.28373 ms_handle_connect on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94bbe00640 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba15a5500 0x55fba15a7920 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a482d1d67f342171 server_cookie=3a3489dfb5be39c1 in_seq=0 out_seq=0 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94a6000640 1 -- 172.21.15.119:0/1923814751 <== mds.0 v2:172.21.15.131:6838/2855046426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fba1492b50 con 0x55fba15a5500 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94a6000640 10 client.28373 handle_command_reply: tid=0 2024-03-28T18:54:08.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 client.28373 shutdown 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 msgr2=0x55fba153d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f94a80c1520 tx=0x7f94a80035f0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 shutdown_connections 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:6838/2855046426,v1:172.21.15.131:6839/2855046426] conn(0x55fba15a5500 0x55fba15a7920 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fba1530c00 0x55fba153d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x55fba1528080 0x55fba153d2d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 --2- 172.21.15.119:0/1923814751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fba15276d0 0x55fba153cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94a6000640 0 client.28373 ms_handle_reset on v2:172.21.15.131:6838/2855046426 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 >> 172.21.15.119:0/1923814751 conn(0x55fba1521da0 msgr2=0x55fba1528cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 shutdown_connections 2024-03-28T18:54:08.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.571+0000 7f94c5f85740 1 -- 172.21.15.119:0/1923814751 wait complete. 2024-03-28T18:54:08.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c5f85740 20 client.28373 trim_cache size 0 max 16384 2024-03-28T18:54:08.580 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 msgr2=0x7f94a00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 0x7f94a00a4eb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94b40094d0 tx=0x7f94b4014040 comp rx=0 tx=0).stop 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 msgr2=0x7f94bc1a8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94a802c240 tx=0x7f94a8004360 comp rx=0 tx=0).stop 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 shutdown_connections 2024-03-28T18:54:08.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f94a00a29f0 0x7f94a00a4eb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f94bc1a27d0 0x7f94bc1a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94bc10f170 0x7f94bc1a8b50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 --2- 172.21.15.119:0/2589698114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94bc10a0a0 0x7f94bc1a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 >> 172.21.15.119:0/2589698114 conn(0x7f94bc072c70 msgr2=0x7f94bc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 shutdown_connections 2024-03-28T18:54:08.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:08.572+0000 7f94c3200640 1 -- 172.21.15.119:0/2589698114 wait complete. 2024-03-28T18:54:08.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:08 smithi119 ceph-mon[31813]: pgmap v311: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:54:08.835 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:08 smithi196 ceph-mon[36300]: pgmap v311: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:54:08.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:08 smithi131 ceph-mon[36065]: pgmap v311: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T18:54:09.591 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-28T18:54:09.591 INFO:tasks.fwd_scrub:done joining 2024-03-28T18:54:09.591 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T18:54:09.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:54:09.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:09 smithi119 ceph-mon[31813]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 -- 172.21.15.119:0/999711945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 msgr2=0x7f4274107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 --2- 172.21.15.119:0/999711945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f4274107ee0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4260009900 tx=0x7f426002bc50 comp rx=0 tx=0).stop 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 -- 172.21.15.119:0/999711945 shutdown_connections 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 --2- 172.21.15.119:0/999711945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4274108420 0x7f4274114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 --2- 172.21.15.119:0/999711945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f4274107ee0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:09.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 --2- 172.21.15.119:0/999711945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f427410da70 0x7f427410de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:09.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 -- 172.21.15.119:0/999711945 >> 172.21.15.119:0/999711945 conn(0x7f42740771c0 msgr2=0x7f42740795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:09.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 -- 172.21.15.119:0/999711945 shutdown_connections 2024-03-28T18:54:09.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.826+0000 7f4273200640 1 -- 172.21.15.119:0/999711945 wait complete. 2024-03-28T18:54:09.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.827+0000 7f4273200640 1 Processor -- start 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.827+0000 7f4273200640 1 -- start start 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4274108420 0x7f427419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274117ed0 con 0x7f4274108420 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274117d50 con 0x7f4274107a60 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4273200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274118050 con 0x7f427410da70 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4271e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52878/0 (socket says 172.21.15.119:52878) 2024-03-28T18:54:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 -- 172.21.15.119:0/1033109606 learned_addr learned my addr 172.21.15.119:0/1033109606 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4271e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52426/0 (socket says 172.21.15.119:52426) 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4271400640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4274108420 0x7f427419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 -- 172.21.15.119:0/1033109606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 msgr2=0x7f427419e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 -- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4274108420 msgr2=0x7f427419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.828+0000 7f4272800640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4274108420 0x7f427419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f4272800640 1 -- 172.21.15.119:0/1033109606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42741a3300 con 0x7f427410da70 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f4271e00640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f4272800640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f426c00eb10 tx=0x7f426c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c004b20 con 0x7f427410da70 2024-03-28T18:54:09.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f426c004cc0 con 0x7f427410da70 2024-03-28T18:54:09.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c010760 con 0x7f427410da70 2024-03-28T18:54:09.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42741a35f0 con 0x7f427410da70 2024-03-28T18:54:09.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.829+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42741aae80 con 0x7f427410da70 2024-03-28T18:54:09.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.830+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4274068800 con 0x7f427410da70 2024-03-28T18:54:09.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.832+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f426c005020 con 0x7f427410da70 2024-03-28T18:54:09.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.833+0000 7f426aa00640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 0x7f42580a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:09.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.834+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f426c0cf590 con 0x7f427410da70 2024-03-28T18:54:09.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.834+0000 7f4271e00640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 0x7f42580a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:09.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.834+0000 7f4271e00640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 0x7f42580a4eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4264005d00 tx=0x7f426400b410 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:09.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:09.837+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f426c09c200 con 0x7f427410da70 2024-03-28T18:54:09.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:09 smithi196 ceph-mon[36300]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:54:09.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:09 smithi131 ceph-mon[36065]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T18:54:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.045+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f427402d070 con 0x7f427410da70 2024-03-28T18:54:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.048+0000 7f426aa00640 1 -- 172.21.15.119:0/1033109606 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+23434 (secure 0 0 0) 0x7f426c09bb00 con 0x7f427410da70 2024-03-28T18:54:10.050 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:10.050 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:54:09.747892+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/393583864":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6801/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/1763050303":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2790763787":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/748679560":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 msgr2=0x7f42580a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 0x7f42580a4eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4264005d00 tx=0x7f426400b410 comp rx=0 tx=0).stop 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 msgr2=0x7f42741a2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f426c00eb10 tx=0x7f426c00c2d0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 shutdown_connections 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f42580a29f0 0x7f42580a4eb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f427410da70 0x7f42741a2bd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4274108420 0x7f427419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 --2- 172.21.15.119:0/1033109606 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4274107a60 0x7f427419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 >> 172.21.15.119:0/1033109606 conn(0x7f42740771c0 msgr2=0x7f4274077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 shutdown_connections 2024-03-28T18:54:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.051+0000 7f4273200640 1 -- 172.21.15.119:0/1033109606 wait complete. 2024-03-28T18:54:10.071 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:54:10.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 -- 172.21.15.119:0/857188695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 msgr2=0x7fd158109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 --2- 172.21.15.119:0/857188695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd158109b10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009930 tx=0x7fd14c02bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 -- 172.21.15.119:0/857188695 shutdown_connections 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 --2- 172.21.15.119:0/857188695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 0x7fd1581105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 --2- 172.21.15.119:0/857188695 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd158103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 --2- 172.21.15.119:0/857188695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd158109b10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.289+0000 7fd15f000640 1 -- 172.21.15.119:0/857188695 >> 172.21.15.119:0/857188695 conn(0x7fd1580ff1d0 msgr2=0x7fd1581015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:10.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.290+0000 7fd15f000640 1 -- 172.21.15.119:0/857188695 shutdown_connections 2024-03-28T18:54:10.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.290+0000 7fd15f000640 1 -- 172.21.15.119:0/857188695 wait complete. 2024-03-28T18:54:10.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.290+0000 7fd15f000640 1 Processor -- start 2024-03-28T18:54:10.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 -- start start 2024-03-28T18:54:10.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 0x7fd15819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd15819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158113b90 con 0x7fd1581040e0 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158113a10 con 0x7fd158103720 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15f000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158113d10 con 0x7fd158109730 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15dc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52442/0 (socket says 172.21.15.119:52442) 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15dc00640 1 -- 172.21.15.119:0/3245624855 learned_addr learned my addr 172.21.15.119:0/3245624855 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15d200640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 0x7fd15819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.291+0000 7fd15e600640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd15819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 -- 172.21.15.119:0/3245624855 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 msgr2=0x7fd15819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd15819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 -- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 msgr2=0x7fd15819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 0x7fd15819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 -- 172.21.15.119:0/3245624855 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15819f040 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15dc00640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009900 tx=0x7fd14c004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c043020 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15819f2d0 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd14c0054e0 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.292+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c003bb0 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.293+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1581a6bc0 con 0x7fd158103720 2024-03-28T18:54:10.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.293+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd158068800 con 0x7fd158103720 2024-03-28T18:54:10.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.296+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fd14c046020 con 0x7fd158103720 2024-03-28T18:54:10.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.297+0000 7fd156000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 0x7fd1400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.297+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fd14c0f11f0 con 0x7fd158103720 2024-03-28T18:54:10.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.297+0000 7fd15d200640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 0x7fd1400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.298+0000 7fd15d200640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 0x7fd1400a4eb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd1581017e0 tx=0x7fd144005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:10.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.305+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd14c04d020 con 0x7fd158103720 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.502+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd15810bc90 con 0x7fd158103720 2024-03-28T18:54:10.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.503+0000 7fd156000640 1 -- 172.21.15.119:0/3245624855 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd14c0bdf70 con 0x7fd158103720 2024-03-28T18:54:10.504 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:54:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 msgr2=0x7fd1400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 0x7fd1400a4eb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd1581017e0 tx=0x7fd144005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 msgr2=0x7fd15819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009900 tx=0x7fd14c004360 comp rx=0 tx=0).stop 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 shutdown_connections 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fd1400a29f0 0x7fd1400a4eb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd158109730 0x7fd15819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1581040e0 0x7fd15819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 --2- 172.21.15.119:0/3245624855 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd158103720 0x7fd15819a050 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 >> 172.21.15.119:0/3245624855 conn(0x7fd1580ff1d0 msgr2=0x7fd1580ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 shutdown_connections 2024-03-28T18:54:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.506+0000 7fd15f000640 1 -- 172.21.15.119:0/3245624855 wait complete. 2024-03-28T18:54:10.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 -- 172.21.15.119:0/4232252930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64104200 msgr2=0x7efc64110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 --2- 172.21.15.119:0/4232252930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64104200 0x7efc64110710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009ca0 tx=0x7efc5c02c0f0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 -- 172.21.15.119:0/4232252930 shutdown_connections 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 --2- 172.21.15.119:0/4232252930 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64104200 0x7efc64110710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 --2- 172.21.15.119:0/4232252930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64103840 0x7efc64103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 --2- 172.21.15.119:0/4232252930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64109850 0x7efc64109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 -- 172.21.15.119:0/4232252930 >> 172.21.15.119:0/4232252930 conn(0x7efc640ff2f0 msgr2=0x7efc64101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 -- 172.21.15.119:0/4232252930 shutdown_connections 2024-03-28T18:54:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.745+0000 7efc69400640 1 -- 172.21.15.119:0/4232252930 wait complete. 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.746+0000 7efc69400640 1 Processor -- start 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.746+0000 7efc69400640 1 -- start start 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.746+0000 7efc69400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc69400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64104200 0x7efc6419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc69400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 0x7efc6419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc69400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc64113c20 con 0x7efc64104200 2024-03-28T18:54:10.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc69400640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc64113aa0 con 0x7efc64103840 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc69400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc64113da0 con 0x7efc64109850 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc5be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64104200 0x7efc6419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc62800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 0x7efc6419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc62800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52464/0 (socket says 172.21.15.119:52464) 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc62800640 1 -- 172.21.15.119:0/1013322091 learned_addr learned my addr 172.21.15.119:0/1013322091 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 -- 172.21.15.119:0/1013322091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 msgr2=0x7efc6419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 -- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64104200 msgr2=0x7efc6419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64104200 0x7efc6419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc63200640 1 -- 172.21.15.119:0/1013322091 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6419f100 con 0x7efc64109850 2024-03-28T18:54:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.747+0000 7efc62800640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc63200640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 0x7efc6419e9d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009bf0 tx=0x7efc5c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c042020 con 0x7efc64109850 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6419f390 con 0x7efc64109850 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc5c034640 con 0x7efc64109850 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c039040 con 0x7efc64109850 2024-03-28T18:54:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.748+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc641a6c80 con 0x7efc64109850 2024-03-28T18:54:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.749+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc64104f80 con 0x7efc64109850 2024-03-28T18:54:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.751+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7efc5c045020 con 0x7efc64109850 2024-03-28T18:54:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.752+0000 7efc5b400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 0x7efc480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.752+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7efc5c0f09e0 con 0x7efc64109850 2024-03-28T18:54:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.752+0000 7efc62800640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 0x7efc480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.753+0000 7efc62800640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 0x7efc480a4e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efc54005fd0 tx=0x7efc5400b4a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.757+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efc5c04c020 con 0x7efc64109850 2024-03-28T18:54:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:10 smithi119 ceph-mon[31813]: pgmap v313: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-28T18:54:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:10 smithi119 ceph-mon[31813]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:54:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:10 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1033109606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:10 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3245624855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:10.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:10 smithi196 ceph-mon[36300]: pgmap v313: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-28T18:54:10.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:10 smithi196 ceph-mon[36300]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:54:10.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:10 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1033109606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:10.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:10 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3245624855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:10 smithi131 ceph-mon[36065]: pgmap v313: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-28T18:54:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:10 smithi131 ceph-mon[36065]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T18:54:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:10 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1033109606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:10.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:10 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3245624855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:10.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.955+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc6419b460 con 0x7efc64109850 2024-03-28T18:54:10.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.956+0000 7efc5b400640 1 -- 172.21.15.119:0/1013322091 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc5c0bd760 con 0x7efc64109850 2024-03-28T18:54:10.957 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.958+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 msgr2=0x7efc480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.958+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 0x7efc480a4e30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efc54005fd0 tx=0x7efc5400b4a0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.958+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 msgr2=0x7efc6419e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.958+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 0x7efc6419e9d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009bf0 tx=0x7efc5c032ec0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 shutdown_connections 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7efc480a2970 0x7efc480a4e30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc64109850 0x7efc6419e9d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc64104200 0x7efc6419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 --2- 172.21.15.119:0/1013322091 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7efc64103840 0x7efc6419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 >> 172.21.15.119:0/1013322091 conn(0x7efc640ff2f0 msgr2=0x7efc640ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 shutdown_connections 2024-03-28T18:54:10.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:10.959+0000 7efc69400640 1 -- 172.21.15.119:0/1013322091 wait complete. 2024-03-28T18:54:10.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:54:11.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 -- 172.21.15.119:0/3462350753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 msgr2=0x7f0378109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 --2- 172.21.15.119:0/3462350753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f0378109c30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f0368009900 tx=0x7f036802bc40 comp rx=0 tx=0).stop 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 -- 172.21.15.119:0/3462350753 shutdown_connections 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 --2- 172.21.15.119:0/3462350753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 0x7f0378110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 --2- 172.21.15.119:0/3462350753 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f0378103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 --2- 172.21.15.119:0/3462350753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f0378109c30 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 -- 172.21.15.119:0/3462350753 >> 172.21.15.119:0/3462350753 conn(0x7f03780ff2f0 msgr2=0x7f0378101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.188+0000 7f037ee00640 1 -- 172.21.15.119:0/3462350753 shutdown_connections 2024-03-28T18:54:11.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.189+0000 7f037ee00640 1 -- 172.21.15.119:0/3462350753 wait complete. 2024-03-28T18:54:11.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 Processor -- start 2024-03-28T18:54:11.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 -- start start 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 0x7f037819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f037819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378113cb0 con 0x7f0378104200 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378113b30 con 0x7f0378109850 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.190+0000 7f037ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378113e30 con 0x7f0378103840 2024-03-28T18:54:11.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52936/0 (socket says 172.21.15.119:52936) 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 -- 172.21.15.119:0/3795278481 learned_addr learned my addr 172.21.15.119:0/3795278481 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f037ce00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f037819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377400640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 0x7f037819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 -- 172.21.15.119:0/3795278481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 msgr2=0x7f037819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f037819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 -- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 msgr2=0x7f037819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 0x7f037819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 -- 172.21.15.119:0/3795278481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037819f160 con 0x7f0378103840 2024-03-28T18:54:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.191+0000 7f0377e00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0368005ab0 tx=0x7f0368004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.192+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368041070 con 0x7f0378103840 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.192+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03680054e0 con 0x7f0378103840 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.192+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368003cc0 con 0x7f0378103840 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.192+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037819f3f0 con 0x7f0378103840 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.192+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03781a6ce0 con 0x7f0378103840 2024-03-28T18:54:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.193+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0378068490 con 0x7f0378103840 2024-03-28T18:54:11.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.195+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f0368044070 con 0x7f0378103840 2024-03-28T18:54:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.196+0000 7f0374c00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 0x7f035c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.196+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f03680f0ae0 con 0x7f0378103840 2024-03-28T18:54:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.196+0000 7f0377400640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 0x7f035c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.197+0000 7f0377400640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 0x7f035c0a4eb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0378108f40 tx=0x7f0360005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:11.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.201+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f03680bd860 con 0x7f0378103840 2024-03-28T18:54:11.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.399+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f037810bdb0 con 0x7f0378103840 2024-03-28T18:54:11.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.399+0000 7f0374c00640 1 -- 172.21.15.119:0/3795278481 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03680bd160 con 0x7f0378103840 2024-03-28T18:54:11.399 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 msgr2=0x7f035c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 0x7f035c0a4eb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0378108f40 tx=0x7f0360005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 msgr2=0x7f037819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0368005ab0 tx=0x7f0368004270 comp rx=0 tx=0).stop 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 shutdown_connections 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f035c0a29f0 0x7f035c0a4eb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0378109850 0x7f037819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0378104200 0x7f037819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 --2- 172.21.15.119:0/3795278481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0378103840 0x7f037819a170 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 >> 172.21.15.119:0/3795278481 conn(0x7f03780ff2f0 msgr2=0x7f03780ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:11.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.402+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 shutdown_connections 2024-03-28T18:54:11.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.403+0000 7f037ee00640 1 -- 172.21.15.119:0/3795278481 wait complete. 2024-03-28T18:54:11.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 -- 172.21.15.119:0/2039645342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2990107ec0 msgr2=0x7f29901082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 --2- 172.21.15.119:0/2039645342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2990107ec0 0x7f29901082a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2984009860 tx=0x7f298402bb70 comp rx=0 tx=0).stop 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 -- 172.21.15.119:0/2039645342 shutdown_connections 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 --2- 172.21.15.119:0/2039645342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f299010cf90 0x7f2990114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 --2- 172.21.15.119:0/2039645342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990108870 0x7f299010c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 --2- 172.21.15.119:0/2039645342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f2990107ec0 0x7f29901082a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 -- 172.21.15.119:0/2039645342 >> 172.21.15.119:0/2039645342 conn(0x7f2990075bf0 msgr2=0x7f2990078010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 -- 172.21.15.119:0/2039645342 shutdown_connections 2024-03-28T18:54:11.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.638+0000 7f2997e00640 1 -- 172.21.15.119:0/2039645342 wait complete. 2024-03-28T18:54:11.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 Processor -- start 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 -- start start 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 0x7f29901a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2990118040 con 0x7f29901a2e60 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2990117ec0 con 0x7f299010cf90 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.639+0000 7f2997e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29901181c0 con 0x7f2990107ec0 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:52952/0 (socket says 172.21.15.119:52952) 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995400640 1 -- 172.21.15.119:0/632000347 learned_addr learned my addr 172.21.15.119:0/632000347 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:52504/0 (socket says 172.21.15.119:52504) 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 0x7f29901a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 -- 172.21.15.119:0/632000347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 msgr2=0x7f299019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 -- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 msgr2=0x7f29901a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 0x7f29901a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 -- 172.21.15.119:0/632000347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29901a3950 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995400640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2995e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 0x7f29901a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.640+0000 7f2994a00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f299006adc0 tx=0x7f297c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.641+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00ccd0 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.641+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f297c0035f0 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.641+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29901a3c40 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.641+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c010690 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.641+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29901ab480 con 0x7f299010cf90 2024-03-28T18:54:11.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.642+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29900688e0 con 0x7f299010cf90 2024-03-28T18:54:11.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.644+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f297c0037a0 con 0x7f299010cf90 2024-03-28T18:54:11.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.645+0000 7f298e000640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 0x7f29780a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:11.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.646+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f297c0cf140 con 0x7f299010cf90 2024-03-28T18:54:11.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.646+0000 7f2995400640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 0x7f29780a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:11.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.646+0000 7f2995400640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 0x7f29780a4eb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f29840095f0 tx=0x7f298402e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:11.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.652+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f297c009510 con 0x7f299010cf90 2024-03-28T18:54:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:11 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1013322091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:11 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3795278481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:11.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.903+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f2990110220 con 0x7f299010cf90 2024-03-28T18:54:11.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.919+0000 7f298e000640 1 -- 172.21.15.119:0/632000347 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+740366 (secure 0 0 0) 0x7f297c09be10 con 0x7f299010cf90 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: "epoch": 27, 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: "active_gid": 25418, 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: "active_name": "x", 2024-03-28T18:54:11.931 INFO:teuthology.orchestra.run.smithi119.stdout: "active_addrs": { 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:6800", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 212877697 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:6801", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 212877697 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "active_addr": "172.21.15.119:6801/212877697", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "active_change": "2024-03-28T18:44:03.152574+0000", 2024-03-28T18:54:11.932 INFO:teuthology.orchestra.run.smithi119.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "available": true, 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "standbys": [ 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "gid": 25409, 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "y", 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_features": 4540701547738038271, 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "available_modules": [ 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "alerts", 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:11.933 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.937 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.938 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_destination": { 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_destination", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_from_name": { 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_from_name", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.939 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "Ceph", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email From: name", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_host": { 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_host", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.940 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP server", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_password": { 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_password", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.941 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Password to authenticate with", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_port": { 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_port", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "465", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP port", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.942 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_sender": { 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_sender", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_ssl": { 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_ssl", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_user": { 2024-03-28T18:54:11.944 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_user", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "User to authenticate as", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.945 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "balancer", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "active": { 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "active", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.946 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_time": { 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_time", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0000", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_weekday": { 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_weekday", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_max_iterations": { 2024-03-28T18:54:11.948 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "25", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "250", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_metrics": { 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_metrics", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.949 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_step": { 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_step", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.5", 2024-03-28T18:54:11.950 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0.001", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "0.999", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "end_time": { 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_time", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2359", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.951 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "end_weekday": { 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_weekday", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.952 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.953 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.954 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.955 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "min_score": { 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_score", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.957 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "upmap", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.958 INFO:teuthology.orchestra.run.smithi119.stdout: "crush-compat", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "none", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "read", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap-read" 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Balancer mode", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_ids": { 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_ids", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.959 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.960 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_deviation": { 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_deviation", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:54:11.961 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_optimizations": { 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.962 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_down_multiplier": { 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_down_multiplier", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.963 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_refresh_rate": { 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_refresh_rate", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "20", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.964 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_starting_port": { 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_starting_port", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4721", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:54:11.965 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_ptrace": { 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_ptrace", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_interval": { 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_interval", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:54:11.967 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.7", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm_log_destination": { 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm_log_destination", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.968 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "file", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "file,syslog", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "syslog" 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "cgroups_split": { 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cgroups_split", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.969 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "config_checks_enabled": { 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_checks_enabled", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.970 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "config_dashboard": { 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_dashboard", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.971 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_alertmanager": { 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_alertmanager", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.972 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_base": { 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_base", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_elasticsearch": { 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "elasticsearch container image", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.974 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_grafana": { 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_grafana", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_haproxy": { 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_haproxy", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.975 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "HAproxy container image", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_agent": { 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.976 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_collector": { 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:54:11.977 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_query": { 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.978 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger query container image", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_keepalived": { 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_keepalived", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Keepalived container image", 2024-03-28T18:54:11.979 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_loki": { 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_loki", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Loki container image", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_node_exporter": { 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_node_exporter", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_nvmeof": { 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_nvmeof", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.981 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Nvme-of container image", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_prometheus": { 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_prometheus", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.982 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_promtail": { 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_promtail", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:54:11.983 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Promtail container image", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_snmp_gateway": { 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.984 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "container_init": { 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_init", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.985 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "daemon_cache_timeout": { 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.986 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "900", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "default_registry": { 2024-03-28T18:54:11.987 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_registry", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "device_cache_timeout": { 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_cache_timeout", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.988 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1800", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "device_enhanced_scan": { 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_enhanced_scan", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.989 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "facts_cache_timeout": { 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "facts_cache_timeout", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.990 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "host_check_interval": { 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "host_check_interval", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:54:11.991 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "hw_monitoring": { 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hw_monitoring", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "inventory_list_all": { 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "inventory_list_all", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:11.993 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.994 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "log_refresh_metadata": { 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_refresh_metadata", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:11.996 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.997 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.998 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:11.999 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "*", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "max_count_per_host": { 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_count_per_host", 2024-03-28T18:54:12.000 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "max_osd_draining_count": { 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_osd_draining_count", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.001 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "migration_current": { 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "migration_current", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.002 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "internal - do not modify", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "root", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm-package", 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: "root" 2024-03-28T18:54:12.003 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "oob_default_addr": { 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "oob_default_addr", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "169.254.1.1", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:54:12.004 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "prometheus_alerts_path": { 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_insecure": { 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_insecure", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_password": { 2024-03-28T18:54:12.006 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_password", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_url": { 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_url", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.007 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_username": { 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_username", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.008 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "secure_monitoring_stack": { 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.009 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "service_discovery_port": { 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "service_discovery_port", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8765", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.010 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "ssh_config_file": { 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssh_config_file", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.011 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "use_agent": { 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_agent", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.012 INFO:teuthology.orchestra.run.smithi119.stdout: "use_repo_digest": { 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_repo_digest", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_failed_host_check": { 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.013 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_daemons": { 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.014 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_hosts": { 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.015 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crash", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.016 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.017 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.018 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.019 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "retain_interval": { 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retain_interval", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "31536000", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_recent_interval": { 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_recent_interval", 2024-03-28T18:54:12.021 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1209600", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dashboard", 2024-03-28T18:54:12.022 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.023 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.024 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.025 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.026 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.027 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.028 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:54:12.029 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.030 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.031 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.032 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.033 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.034 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.035 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_URL": { 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.036 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.037 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.038 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:54:12.039 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.040 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "{}", 2024-03-28T18:54:12.041 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.042 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.043 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.044 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:54:12.045 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.046 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.047 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.048 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:54:12.051 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.052 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "45", 2024-03-28T18:54:12.053 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.054 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.055 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:54:12.057 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.058 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.059 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.060 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "cross_origin_url": { 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cross_origin_url", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.061 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "crt_file": { 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crt_file", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "debug": { 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "debug", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.063 INFO:teuthology.orchestra.run.smithi119.stdout: "jwt_token_ttl": { 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "jwt_token_ttl", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "28800", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.064 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.065 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.066 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.067 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.068 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "motd": { 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "motd", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.069 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "The message of the day", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.070 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8080", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.072 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.073 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl_server_port": { 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl_server_port", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8443", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.074 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "redirect", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect" 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.075 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "url_prefix": { 2024-03-28T18:54:12.076 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url_prefix", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "devicehealth", 2024-03-28T18:54:12.077 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_monitoring": { 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_monitoring", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "monitor device health metrics", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.078 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.079 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.080 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.081 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.082 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "mark_out_threshold": { 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mark_out_threshold", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2419200", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:54:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_name": { 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_name", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "device_health_metrics", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "retention_period": { 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retention_period", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15552000", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_frequency": { 2024-03-28T18:54:12.085 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_frequency", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "self_heal": { 2024-03-28T18:54:12.086 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "self_heal", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.087 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_threshold": { 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_threshold", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7257600", 2024-03-28T18:54:12.088 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "diskprediction_local", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.089 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.090 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.092 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.093 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "predict_interval": { 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predict_interval", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "predictor_model": { 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predictor_model", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "prophetstor", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.095 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "influx", 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": false, 2024-03-28T18:54:12.096 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "batch_size": { 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "batch_size", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5000", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "database": { 2024-03-28T18:54:12.097 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "database", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "ceph", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": { 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hostname", 2024-03-28T18:54:12.098 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.099 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "30", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "5", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.100 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.102 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "password": { 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "password", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "port": { 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "port", 2024-03-28T18:54:12.105 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8086", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server port", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.106 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "false", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "threads": { 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threads", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "32", 2024-03-28T18:54:12.107 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "username": { 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "username", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.108 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "verify_ssl": { 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "verify_ssl", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "true", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.109 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "insights", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.110 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.111 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.112 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.113 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "iostat", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.115 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.116 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.117 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.118 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "k8sevents", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_event_retention_days": { 2024-03-28T18:54:12.119 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "config_check_secs": { 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_check_secs", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.120 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "10", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.121 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.122 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.123 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.124 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.125 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.126 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "localpool", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "failure_domain": { 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "failure_domain", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "host", 2024-03-28T18:54:12.127 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.128 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.129 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.130 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.131 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "min_size": { 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_size", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.132 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "num_rep": { 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "num_rep", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.133 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_num": { 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_num", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "128", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.134 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "prefix": { 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prefix", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:54:12.135 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "subtree": { 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "subtree", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "rack", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mds_autoscaler", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.137 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.138 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.139 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.140 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirroring", 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.141 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.142 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.143 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.144 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.145 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "nfs", 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.146 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.147 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.149 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.150 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "fail_fs": { 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "fail_fs", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.151 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.152 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.153 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.155 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator": { 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "rook", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "test_orchestrator" 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Orchestrator backend", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_perf_query", 2024-03-28T18:54:12.157 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.158 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.159 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.160 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.161 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_support", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.163 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.164 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.165 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.166 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_autoscaler", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.168 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.169 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.170 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.171 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "threshold": { 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threshold", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.173 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1.0", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "scaling threshold", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "progress", 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.174 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_pg_recovery_event": { 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.176 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.177 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.178 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.179 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.180 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "max_completed_events": { 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_completed_events", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "50", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:54:12.181 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.182 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus", 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "cache": { 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cache", 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.183 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "exclude_perf_counters": { 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "exclude_perf_counters", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.184 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.185 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.186 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.187 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.188 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools": { 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools", 2024-03-28T18:54:12.189 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.190 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "300", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_interval": { 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_interval", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.191 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15.0", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.192 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "9283", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.193 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "stale_cache_strategy": { 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stale_cache_strategy", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "log", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.194 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "default", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.195 INFO:teuthology.orchestra.run.smithi119.stdout: "default", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "error" 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.196 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_support", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.197 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.199 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.200 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.201 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_snap_create": { 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.202 INFO:teuthology.orchestra.run.smithi119.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "trash_purge_schedule": { 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "trash_purge_schedule", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.203 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "restful", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.204 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_auth": { 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_auth", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.205 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.206 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.207 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.209 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.210 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.211 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.212 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rgw", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.213 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.214 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.215 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.216 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rook", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.217 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.218 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.219 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.220 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.221 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "storage_class": { 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "storage_class", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "local", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.222 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "selftest", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.223 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "roption1": { 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption1", 2024-03-28T18:54:12.227 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "roption2": { 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption2", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.228 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "xyz", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption1": { 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption1", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.229 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption2": { 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption2", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.230 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption3": { 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption3", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.231 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption4": { 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption4", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption5": { 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption5", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption6": { 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption6", 2024-03-28T18:54:12.233 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption7": { 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption7", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.234 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "42", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "testkey": { 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testkey", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.235 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "testlkey": { 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testlkey", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.236 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "testnewline": { 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testnewline", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.237 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snap_schedule", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_m_granularity": { 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_m_granularity", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.238 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "dump_on_update": { 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dump_on_update", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:54:12.239 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.240 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.241 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.243 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stats", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.244 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.245 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.246 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.247 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.248 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "status", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.249 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.251 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.252 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telegraf", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "address": { 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "address", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.254 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15", 2024-03-28T18:54:12.255 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.257 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.258 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.259 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telemetry", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_basic": { 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_basic", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.262 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_crash": { 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_crash", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.263 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_device": { 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_device", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.264 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_ident": { 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_ident", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_perf": { 2024-03-28T18:54:12.265 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_perf", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "contact": { 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "contact", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.266 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "description": { 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "description", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.267 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "device_url": { 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_url", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.268 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.269 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "24", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "8", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.270 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "last_opt_revision": { 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "last_opt_revision", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard": { 2024-03-28T18:54:12.271 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard_description": { 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard_description", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.272 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.273 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.274 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.275 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.276 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "organization": { 2024-03-28T18:54:12.277 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "organization", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "proxy": { 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "proxy", 2024-03-28T18:54:12.278 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "url": { 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.279 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "test_orchestrator", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.280 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.281 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.282 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.283 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.284 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "volumes", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.285 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.287 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.288 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.289 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_clones": { 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_clones", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.290 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "periodic_async_work": { 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "periodic_async_work", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Periodically check for async work", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.291 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_delay": { 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:54:12.292 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix", 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.293 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "discovery_interval": { 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "discovery_interval", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "100", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.294 INFO:teuthology.orchestra.run.smithi119.stdout: "identifier": { 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "identifier", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.295 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.296 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.297 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.298 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_host": { 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_host", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.300 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_port": { 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_port", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.301 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10051", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_sender": { 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_sender", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:54:12.302 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "gid": 25441, 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "z", 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_features": 4540701547738038271, 2024-03-28T18:54:12.303 INFO:teuthology.orchestra.run.smithi119.stdout: "available_modules": [ 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "alerts", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.304 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.305 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.306 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.307 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.308 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_destination": { 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_destination", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:54:12.309 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_from_name": { 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_from_name", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "Ceph", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email From: name", 2024-03-28T18:54:12.310 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_host": { 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_host", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP server", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.311 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_password": { 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_password", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Password to authenticate with", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.312 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_port": { 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_port", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "465", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP port", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_sender": { 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_sender", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.313 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_ssl": { 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_ssl", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.314 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_user": { 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_user", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.315 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "User to authenticate as", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "balancer", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "active": { 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "active", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.316 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_time": { 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_time", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0000", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.317 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_weekday": { 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_weekday", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.318 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_max_iterations": { 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "25", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "250", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.319 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_metrics": { 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_metrics", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_step": { 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_step", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.5", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0.001", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "0.999", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "end_time": { 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_time", 2024-03-28T18:54:12.321 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2359", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "end_weekday": { 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_weekday", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.322 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.323 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.324 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.325 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.326 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "min_score": { 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_score", 2024-03-28T18:54:12.327 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.328 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "upmap", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "crush-compat", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "none", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "read", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap-read" 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Balancer mode", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_ids": { 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_ids", 2024-03-28T18:54:12.329 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.330 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_deviation": { 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_deviation", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.331 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_optimizations": { 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:54:12.332 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_down_multiplier": { 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_down_multiplier", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.333 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_refresh_rate": { 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_refresh_rate", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "20", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.334 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_starting_port": { 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_starting_port", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4721", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.335 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_ptrace": { 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_ptrace", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:54:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_interval": { 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_interval", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.7", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm_log_destination": { 2024-03-28T18:54:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm_log_destination", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "file", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "file,syslog", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "syslog" 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.339 INFO:teuthology.orchestra.run.smithi119.stdout: "cgroups_split": { 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cgroups_split", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "config_checks_enabled": { 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_checks_enabled", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.340 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "config_dashboard": { 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_dashboard", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.341 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_alertmanager": { 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_alertmanager", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.342 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_base": { 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_base", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.343 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_elasticsearch": { 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "elasticsearch container image", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_grafana": { 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_grafana", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_haproxy": { 2024-03-28T18:54:12.345 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_haproxy", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "HAproxy container image", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_agent": { 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.346 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_collector": { 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.347 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_query": { 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.348 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger query container image", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_keepalived": { 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_keepalived", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Keepalived container image", 2024-03-28T18:54:12.349 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_loki": { 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_loki", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Loki container image", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_node_exporter": { 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_node_exporter", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_nvmeof": { 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_nvmeof", 2024-03-28T18:54:12.351 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Nvme-of container image", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_prometheus": { 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_prometheus", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.352 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_promtail": { 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_promtail", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.353 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Promtail container image", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_snmp_gateway": { 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.354 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "container_init": { 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_init", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.355 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "daemon_cache_timeout": { 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:54:12.356 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "900", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "default_registry": { 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_registry", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.357 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "device_cache_timeout": { 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_cache_timeout", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.358 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1800", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "device_enhanced_scan": { 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_enhanced_scan", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.359 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "facts_cache_timeout": { 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "facts_cache_timeout", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.360 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "host_check_interval": { 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "host_check_interval", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.361 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "hw_monitoring": { 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hw_monitoring", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.362 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "inventory_list_all": { 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "inventory_list_all", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.363 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.364 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "log_refresh_metadata": { 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_refresh_metadata", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.366 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.367 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.369 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "*", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "max_count_per_host": { 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_count_per_host", 2024-03-28T18:54:12.370 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "max_osd_draining_count": { 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_osd_draining_count", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.371 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "migration_current": { 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "migration_current", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.372 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "internal - do not modify", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "root", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm-package", 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: "root" 2024-03-28T18:54:12.373 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "oob_default_addr": { 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "oob_default_addr", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "169.254.1.1", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.374 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "prometheus_alerts_path": { 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.375 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_insecure": { 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_insecure", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.376 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_password": { 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_password", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_url": { 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_url", 2024-03-28T18:54:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_username": { 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_username", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.378 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "secure_monitoring_stack": { 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.379 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "service_discovery_port": { 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "service_discovery_port", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8765", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.380 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "ssh_config_file": { 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssh_config_file", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.381 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "use_agent": { 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_agent", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "use_repo_digest": { 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_repo_digest", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_failed_host_check": { 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:54:12.383 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_daemons": { 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.384 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_hosts": { 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.385 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crash", 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.386 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.387 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.390 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "retain_interval": { 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retain_interval", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "31536000", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_recent_interval": { 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_recent_interval", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1209600", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dashboard", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.393 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:54:12.395 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.396 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.397 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.398 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.399 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:54:12.400 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:54:12.401 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.402 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.404 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.405 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_URL": { 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.407 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.408 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.409 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.410 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.411 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.412 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "{}", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.413 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.414 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.415 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.416 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:54:12.417 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.418 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.419 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.420 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.421 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.422 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.423 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.424 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:54:12.425 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "45", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.426 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.427 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.428 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.429 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.430 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.431 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "cross_origin_url": { 2024-03-28T18:54:12.433 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cross_origin_url", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "crt_file": { 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crt_file", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.434 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "debug": { 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "debug", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.435 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "jwt_token_ttl": { 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "jwt_token_ttl", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "28800", 2024-03-28T18:54:12.436 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.437 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.438 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.439 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.441 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.442 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.443 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "motd": { 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "motd", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "The message of the day", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.445 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8080", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.447 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl_server_port": { 2024-03-28T18:54:12.448 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl_server_port", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8443", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.449 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "redirect", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect" 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.450 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "url_prefix": { 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url_prefix", 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.452 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.453 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "devicehealth", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_monitoring": { 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_monitoring", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "monitor device health metrics", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.454 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.455 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.456 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.457 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.458 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "mark_out_threshold": { 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mark_out_threshold", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2419200", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:54:12.459 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_name": { 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_name", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "device_health_metrics", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.460 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "retention_period": { 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retention_period", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15552000", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.461 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_frequency": { 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_frequency", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "self_heal": { 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "self_heal", 2024-03-28T18:54:12.462 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.463 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_threshold": { 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_threshold", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7257600", 2024-03-28T18:54:12.464 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "diskprediction_local", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.465 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.466 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.467 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.469 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "predict_interval": { 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predict_interval", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.470 INFO:teuthology.orchestra.run.smithi119.stdout: "predictor_model": { 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predictor_model", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "prophetstor", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.471 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "influx", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": false, 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.472 INFO:teuthology.orchestra.run.smithi119.stdout: "batch_size": { 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "batch_size", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5000", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "database": { 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "database", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.473 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "ceph", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": { 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hostname", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.474 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "30", 2024-03-28T18:54:12.475 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "5", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.476 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.477 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.478 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "password": { 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "password", 2024-03-28T18:54:12.480 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "port": { 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "port", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.481 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8086", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server port", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "false", 2024-03-28T18:54:12.482 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "threads": { 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threads", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "32", 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.483 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "username": { 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "username", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.484 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "verify_ssl": { 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "verify_ssl", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "true", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "insights", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.486 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.487 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.488 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "iostat", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.491 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.492 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.493 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.494 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "k8sevents", 2024-03-28T18:54:12.495 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_event_retention_days": { 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.496 INFO:teuthology.orchestra.run.smithi119.stdout: "config_check_secs": { 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_check_secs", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "10", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.497 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.498 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "localpool", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "failure_domain": { 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "failure_domain", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "host", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.502 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.503 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.504 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.505 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.506 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "min_size": { 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_size", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.507 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "num_rep": { 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "num_rep", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.508 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_num": { 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_num", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "128", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.509 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "prefix": { 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prefix", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "subtree": { 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "subtree", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "rack", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:54:12.511 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mds_autoscaler", 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.512 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.513 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.514 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.515 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirroring", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.517 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.518 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.519 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.520 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "nfs", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.521 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.522 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.523 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.524 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.525 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "fail_fs": { 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "fail_fs", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.526 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.527 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.528 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.529 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.530 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.531 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.531 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.531 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.531 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.531 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.532 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.532 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator": { 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "rook", 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "test_orchestrator" 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.533 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Orchestrator backend", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_perf_query", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.534 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.535 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.536 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.537 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.538 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_support", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.539 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.540 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.541 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.542 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.543 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_autoscaler", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.544 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.545 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.546 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.547 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.548 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "threshold": { 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threshold", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1.0", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.549 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "scaling threshold", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "progress", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_pg_recovery_event": { 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.550 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.551 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.552 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.553 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.555 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "max_completed_events": { 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_completed_events", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.556 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "50", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.557 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "cache": { 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cache", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.558 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "exclude_perf_counters": { 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "exclude_perf_counters", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.559 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.560 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.561 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.562 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.563 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.564 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools": { 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.565 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "300", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_interval": { 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_interval", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15.0", 2024-03-28T18:54:12.566 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.567 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "9283", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:54:12.568 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "stale_cache_strategy": { 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stale_cache_strategy", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "log", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "default", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "default", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "error" 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.570 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.571 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_support", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.572 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.573 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.574 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.575 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_snap_create": { 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.576 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.577 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "trash_purge_schedule": { 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "trash_purge_schedule", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.578 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "restful", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_auth": { 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_auth", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.579 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.580 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.581 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.582 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.583 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.584 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.585 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.586 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rgw", 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.588 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.589 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.590 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.591 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rook", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.592 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.593 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.594 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.595 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "storage_class": { 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "storage_class", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "local", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "selftest", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.598 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.599 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.600 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.601 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "roption1": { 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption1", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.602 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "roption2": { 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption2", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "xyz", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.603 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption1": { 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption1", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.604 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption2": { 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption2", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.605 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption3": { 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption3", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption4": { 2024-03-28T18:54:12.606 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption4", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption5": { 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption5", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.607 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption6": { 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption6", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.608 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption7": { 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption7", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "42", 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.609 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "testkey": { 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testkey", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.610 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "testlkey": { 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testlkey", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.611 INFO:teuthology.orchestra.run.smithi119.stdout: "testnewline": { 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testnewline", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snap_schedule", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_m_granularity": { 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_m_granularity", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.613 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "dump_on_update": { 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dump_on_update", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.614 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.617 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.618 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stats", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.619 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.620 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.621 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.622 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.623 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "status", 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.624 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.625 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.626 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.627 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.628 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telegraf", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "address": { 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "address", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.629 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.630 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.631 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.632 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.633 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.634 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telemetry", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_basic": { 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_basic", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.635 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_crash": { 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_crash", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.636 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_device": { 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_device", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.637 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_ident": { 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_ident", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.638 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_perf": { 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_perf", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.639 INFO:teuthology.orchestra.run.smithi119.stdout: "contact": { 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "contact", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "description": { 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "description", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.640 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "device_url": { 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_url", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:54:12.641 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.642 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "24", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "8", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.643 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "last_opt_revision": { 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "last_opt_revision", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.644 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard": { 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard_description": { 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard_description", 2024-03-28T18:54:12.645 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.646 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.647 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.648 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.649 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.650 INFO:teuthology.orchestra.run.smithi119.stdout: "organization": { 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "organization", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "proxy": { 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "proxy", 2024-03-28T18:54:12.651 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "url": { 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.652 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "test_orchestrator", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.653 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.654 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.655 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.656 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.657 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "volumes", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.658 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.659 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.660 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.661 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.662 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_clones": { 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_clones", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "periodic_async_work": { 2024-03-28T18:54:12.663 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "periodic_async_work", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Periodically check for async work", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_delay": { 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.664 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.665 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.666 INFO:teuthology.orchestra.run.smithi119.stdout: "discovery_interval": { 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "discovery_interval", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "100", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "identifier": { 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "identifier", 2024-03-28T18:54:12.667 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.668 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.669 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.670 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.671 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.672 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.673 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_host": { 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_host", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_port": { 2024-03-28T18:54:12.674 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_port", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10051", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_sender": { 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_sender", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.675 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "modules": [ 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm", 2024-03-28T18:54:12.676 INFO:teuthology.orchestra.run.smithi119.stdout: "dashboard", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "iostat", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "nfs", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "restful", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "snap_schedule" 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "available_modules": [ 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "alerts", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.677 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.678 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.679 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.680 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.681 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_destination": { 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_destination", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email address to send alerts to", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_from_name": { 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_from_name", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "Ceph", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Email From: name", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_host": { 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_host", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.684 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP server", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_password": { 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_password", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.685 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Password to authenticate with", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_port": { 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_port", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "465", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.686 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP port", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_sender": { 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_sender", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SMTP envelope sender", 2024-03-28T18:54:12.687 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_ssl": { 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_ssl", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.688 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "smtp_user": { 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "smtp_user", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "User to authenticate as", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "balancer", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "active": { 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "active", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.690 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_time": { 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_time", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0000", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:12.691 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "begin_weekday": { 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "begin_weekday", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_max_iterations": { 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_max_iterations", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "25", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "250", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_metrics": { 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_metrics", 2024-03-28T18:54:12.693 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_compat_step": { 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crush_compat_step", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.694 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.5", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0.001", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "0.999", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "aggressiveness of optimization", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "end_time": { 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_time", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2359", 2024-03-28T18:54:12.695 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "end_weekday": { 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "end_weekday", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "0", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "6", 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.696 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.697 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.698 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.699 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.700 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "min_score": { 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_score", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.701 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "upmap", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.702 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "crush-compat", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "none", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "read", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap-read" 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Balancer mode", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_ids": { 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_ids", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.703 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.704 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_deviation": { 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_deviation", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.705 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "upmap_max_optimizations": { 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "upmap_max_optimizations", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T18:54:12.706 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_down_multiplier": { 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_down_multiplier", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.707 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_refresh_rate": { 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_refresh_rate", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "20", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.708 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "agent_starting_port": { 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "agent_starting_port", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4721", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T18:54:12.709 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_ptrace": { 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_ptrace", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_interval": { 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_interval", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "autotune_memory_target_ratio": { 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T18:54:12.711 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0.7", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm_log_destination": { 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cephadm_log_destination", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.712 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "file", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "file,syslog", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "syslog" 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "cgroups_split": { 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cgroups_split", 2024-03-28T18:54:12.713 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "config_checks_enabled": { 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_checks_enabled", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.714 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "config_dashboard": { 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_dashboard", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.715 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_alertmanager": { 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_alertmanager", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.716 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_base": { 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_base", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Container image name, without the tag", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.717 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_elasticsearch": { 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_elasticsearch", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "elasticsearch container image", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.718 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_grafana": { 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_grafana", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_haproxy": { 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_haproxy", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.719 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "HAproxy container image", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_agent": { 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_agent", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.720 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger agent container image", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_collector": { 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_collector", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.721 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger collector container image", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_jaeger_query": { 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_jaeger_query", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Jaeger query container image", 2024-03-28T18:54:12.722 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_keepalived": { 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_keepalived", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Keepalived container image", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.723 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_loki": { 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_loki", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Loki container image", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.724 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_node_exporter": { 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_node_exporter", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_nvmeof": { 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_nvmeof", 2024-03-28T18:54:12.725 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Nvme-of container image", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_prometheus": { 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_prometheus", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.726 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Prometheus container image", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_promtail": { 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_promtail", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T18:54:12.727 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Promtail container image", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image_snmp_gateway": { 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_image_snmp_gateway", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.728 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "SNMP Gateway container image", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "container_init": { 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "container_init", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.729 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "daemon_cache_timeout": { 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "daemon_cache_timeout", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.730 INFO:teuthology.orchestra.run.smithi119.stdout: "default_cephadm_command_timeout": { 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "900", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "default_registry": { 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "default_registry", 2024-03-28T18:54:12.731 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "docker.io", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "device_cache_timeout": { 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_cache_timeout", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.732 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1800", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache device inventory", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "device_enhanced_scan": { 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_enhanced_scan", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.733 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "facts_cache_timeout": { 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "facts_cache_timeout", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "seconds to cache host facts data", 2024-03-28T18:54:12.734 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "host_check_interval": { 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "host_check_interval", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to perform a host check", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.735 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "hw_monitoring": { 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hw_monitoring", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.736 INFO:teuthology.orchestra.run.smithi119.stdout: "inventory_list_all": { 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "inventory_list_all", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.738 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "log_refresh_metadata": { 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_refresh_metadata", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.739 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.740 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T18:54:12.742 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.743 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "*", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "max_count_per_host": { 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_count_per_host", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.744 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of daemons per service per host", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "max_osd_draining_count": { 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_osd_draining_count", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.745 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "migration_current": { 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "migration_current", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "internal - do not modify", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.746 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": { 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mode", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "root", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm-package", 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "root" 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.747 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "oob_default_addr": { 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "oob_default_addr", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "169.254.1.1", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "prometheus_alerts_path": { 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus_alerts_path", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_insecure": { 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_insecure", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.750 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_password": { 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_password", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_url": { 2024-03-28T18:54:12.751 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_url", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "registry_username": { 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "registry_username", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.752 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "secure_monitoring_stack": { 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "secure_monitoring_stack", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.753 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "service_discovery_port": { 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "service_discovery_port", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8765", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "cephadm service discovery port", 2024-03-28T18:54:12.754 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "ssh_config_file": { 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssh_config_file", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.755 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "use_agent": { 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_agent", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.756 INFO:teuthology.orchestra.run.smithi119.stdout: "use_repo_digest": { 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "use_repo_digest", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_failed_host_check": { 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_failed_host_check", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.757 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_daemons": { 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_daemons", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.758 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_on_stray_hosts": { 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_on_stray_hosts", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.759 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crash", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.760 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.761 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.762 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.763 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.764 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "retain_interval": { 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retain_interval", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "31536000", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.765 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_recent_interval": { 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_recent_interval", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1209600", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dashboard", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.768 INFO:teuthology.orchestra.run.smithi119.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_ENABLED": { 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.769 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.770 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.771 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.772 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T18:54:12.774 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.775 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.776 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.777 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.778 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.779 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_URL": { 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_URL", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.780 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.781 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.782 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.783 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.784 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.785 INFO:teuthology.orchestra.run.smithi119.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "{}", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.786 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.787 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.788 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.789 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.790 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T18:54:12.792 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.793 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.794 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.795 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.796 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.797 INFO:teuthology.orchestra.run.smithi119.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "45", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T18:54:12.798 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.799 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "admin", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.800 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.801 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T18:54:12.803 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.804 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "cross_origin_url": { 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cross_origin_url", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.805 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "crt_file": { 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "crt_file", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.806 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "debug": { 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "debug", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.807 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Enable/disable debug options", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "jwt_token_ttl": { 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "jwt_token_ttl", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "28800", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.808 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.810 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.811 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.812 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.813 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "motd": { 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "motd", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.814 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "The message of the day", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect_resolve_ip_addr": { 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.815 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.816 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8080", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.817 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.818 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl_server_port": { 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl_server_port", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8443", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.819 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "redirect", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "redirect" 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.820 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.821 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "url_prefix": { 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url_prefix", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.822 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "devicehealth", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_monitoring": { 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_monitoring", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.823 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "monitor device health metrics", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.824 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.825 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.826 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.827 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.828 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "mark_out_threshold": { 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mark_out_threshold", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "2419200", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.829 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "pool_name": { 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pool_name", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "device_health_metrics", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.830 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "retention_period": { 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "retention_period", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15552000", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long to retain device health metrics", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_frequency": { 2024-03-28T18:54:12.831 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_frequency", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "self_heal": { 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "self_heal", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.832 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.833 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "warn_threshold": { 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "warn_threshold", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7257600", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.834 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "diskprediction_local", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.835 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.838 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.839 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "predict_interval": { 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predict_interval", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "86400", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "predictor_model": { 2024-03-28T18:54:12.840 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "predictor_model", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "prophetstor", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "600", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.842 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "influx", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": false, 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "influxdb python module not found", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "batch_size": { 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "batch_size", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5000", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.843 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "database": { 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "database", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "ceph", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.844 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": { 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "hostname", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server hostname", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "30", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "5", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.847 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.848 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.849 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.850 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "password": { 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "password", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.851 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "password of InfluxDB server user", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "port": { 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "port", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "8086", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "InfluxDB server port", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.852 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "ssl": { 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ssl", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "false", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "threads": { 2024-03-28T18:54:12.853 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threads", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "32", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "username": { 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "username", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.854 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "username of InfluxDB server user", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "verify_ssl": { 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "verify_ssl", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.855 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "true", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "insights", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.856 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.857 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.858 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.859 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.860 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "iostat", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.861 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.862 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.863 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.864 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.865 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "k8sevents", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_event_retention_days": { 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "ceph_event_retention_days", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "7", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.866 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "config_check_secs": { 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "config_check_secs", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "10", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.867 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.868 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.870 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.871 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "localpool", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.872 INFO:teuthology.orchestra.run.smithi119.stdout: "failure_domain": { 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "failure_domain", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "host", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "failure domain for any created local pool", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.873 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.874 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.876 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.877 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "min_size": { 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "min_size", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default min_size for any created local pool", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "num_rep": { 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "num_rep", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default replica count for any created local pool", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_num": { 2024-03-28T18:54:12.879 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_num", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "128", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "prefix": { 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prefix", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.880 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "name prefix for any created local pool", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "subtree": { 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "subtree", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "rack", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.881 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mds_autoscaler", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.882 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.883 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.885 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.886 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirroring", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.887 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.888 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.889 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.890 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.891 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "nfs", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.892 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.893 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.894 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.895 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.896 INFO:teuthology.orchestra.run.smithi119.stdout: "fail_fs": { 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "fail_fs", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.897 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.898 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.899 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.900 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.901 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator": { 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "orchestrator", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "rook", 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: "test_orchestrator" 2024-03-28T18:54:12.902 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Orchestrator backend", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_perf_query", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.903 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.906 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.907 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "osd_support", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.908 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.909 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.910 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.911 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "pg_autoscaler", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.912 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.913 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.914 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.915 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.916 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.917 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "threshold": { 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "threshold", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "3.0", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1.0", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "scaling threshold", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.918 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "progress", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_pg_recovery_event": { 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_pg_recovery_event", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.919 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.921 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.922 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.923 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.924 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "max_completed_events": { 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_completed_events", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "50", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.925 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "number of past completed events to remember", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "sleep_interval": { 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "sleep_interval", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "5", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.926 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "how long the module is going to sleep", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "prometheus", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "cache": { 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "cache", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.927 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "exclude_perf_counters": { 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "exclude_perf_counters", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.928 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.929 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.930 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.931 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.932 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools": { 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.933 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "300", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.934 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "scrape_interval": { 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "scrape_interval", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15.0", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.935 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "::", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.936 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "9283", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "stale_cache_strategy": { 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stale_cache_strategy", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.937 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "log", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_behaviour": { 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_behaviour", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.938 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "default", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "default", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "error" 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.939 INFO:teuthology.orchestra.run.smithi119.stdout: "standby_error_status_code": { 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "standby_error_status_code", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "500", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "400", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "599", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_support", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.942 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.943 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.944 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.945 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_snap_create": { 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_snap_create", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.946 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "mirror_snapshot_schedule": { 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.947 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "trash_purge_schedule": { 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "trash_purge_schedule", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "restful", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_auth": { 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enable_auth", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.949 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "key_file": { 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "key_file", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.950 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.951 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.952 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.953 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.954 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "server_addr": { 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_addr", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.955 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "server_port": { 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "server_port", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rgw", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.957 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.958 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.960 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.961 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rook", 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.962 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.963 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.964 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.965 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "storage_class": { 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "storage_class", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "local", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "selftest", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.968 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.969 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.970 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.971 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "roption1": { 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption1", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.972 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "roption2": { 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "roption2", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "xyz", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.973 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption1": { 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption1", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.974 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption2": { 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption2", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption3": { 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption3", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "float", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption4": { 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption4", 2024-03-28T18:54:12.976 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption5": { 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption5", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption6": { 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption6", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:12.978 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "rwoption7": { 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rwoption7", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "1", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "42", 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.979 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "testkey": { 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testkey", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.980 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "testlkey": { 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testlkey", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "testnewline": { 2024-03-28T18:54:12.981 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "testnewline", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.982 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snap_schedule", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "allow_m_granularity": { 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "allow_m_granularity", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "dump_on_update": { 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "dump_on_update", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "dump database to debug log on update", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.984 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.985 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.987 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.988 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "stats", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.989 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.991 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.992 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.993 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "status", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.994 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.995 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.996 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:12.997 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telegraf", 2024-03-28T18:54:12.998 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "address": { 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "address", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:12.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "15", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:13.000 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.001 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.003 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.004 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "telemetry", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_basic": { 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_basic", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:13.005 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_crash": { 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_crash", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.006 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_device": { 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_device", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T18:54:13.007 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_ident": { 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_ident", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.008 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "channel_perf": { 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "channel_perf", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.009 INFO:teuthology.orchestra.run.smithi119.stdout: "contact": { 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "contact", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "description": { 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "description", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.010 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "device_url": { 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "device_url", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T18:54:13.011 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "enabled": { 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "enabled", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.012 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "24", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "8", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.013 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "last_opt_revision": { 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "last_opt_revision", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "1", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.014 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard": { 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "leaderboard_description": { 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "leaderboard_description", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.015 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.016 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:13.017 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.018 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.019 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.020 INFO:teuthology.orchestra.run.smithi119.stdout: "organization": { 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "organization", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "proxy": { 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "proxy", 2024-03-28T18:54:13.021 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "url": { 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "url", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.022 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "test_orchestrator", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:13.023 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.024 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.025 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.026 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.027 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "volumes", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.028 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:13.030 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.031 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.032 INFO:teuthology.orchestra.run.smithi119.stdout: "max_concurrent_clones": { 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "max_concurrent_clones", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "4", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "periodic_async_work": { 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "periodic_async_work", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.033 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Periodically check for async work", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_delay": { 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_delay", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.034 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "0", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "snapshot_clone_no_wait": { 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "True", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.035 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "can_run": true, 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "error_string": "", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "module_options": { 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "discovery_interval": { 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "discovery_interval", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "uint", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.036 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "100", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "identifier": { 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "identifier", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.037 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "interval": { 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "interval", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "secs", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "60", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.038 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "log_level": { 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_level", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.039 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster": { 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.040 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_cluster_level": { 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_cluster_level", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "info", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [ 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "critical", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "debug", 2024-03-28T18:54:13.041 INFO:teuthology.orchestra.run.smithi119.stdout: "error", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "info", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "warning" 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": { 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "log_to_file", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "bool", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 1, 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "False", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.042 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_host": { 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_host", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.043 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_port": { 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_port", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "int", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "10051", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.044 INFO:teuthology.orchestra.run.smithi119.stdout: "zabbix_sender": { 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "zabbix_sender", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "str", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "level": "advanced", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "flags": 0, 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "min": "", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "max": "", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "enum_allowed": [], 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "desc": "", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "long_desc": "", 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "tags": [], 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "see_also": [] 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.045 INFO:teuthology.orchestra.run.smithi119.stdout: "services": { 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "dashboard": "https://172.21.15.119:8443/" 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "always_on_modules": { 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "octopus": [ 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "balancer", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "crash", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "devicehealth", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_autoscaler", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "progress", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_support", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "status", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "telemetry", 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: "volumes" 2024-03-28T18:54:13.046 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pacific": [ 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "balancer", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "crash", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "devicehealth", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_autoscaler", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "progress", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_support", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "status", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "telemetry", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "volumes" 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "quincy": [ 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "balancer", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "crash", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "devicehealth", 2024-03-28T18:54:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_autoscaler", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "progress", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_support", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "status", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "telemetry", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "volumes" 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "reef": [ 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "balancer", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "crash", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "devicehealth", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_autoscaler", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "progress", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_support", 2024-03-28T18:54:13.048 INFO:teuthology.orchestra.run.smithi119.stdout: "status", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "telemetry", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "volumes" 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "squid": [ 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "balancer", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "crash", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "devicehealth", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "orchestrator", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "pg_autoscaler", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "progress", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_support", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "status", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "telemetry", 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: "volumes" 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "last_failure_osd_epoch": 91, 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "active_clients": [ 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "devicehealth", 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1018513408 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "libcephsqlite", 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 755667471 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "rbd_support", 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 848629601 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snap_schedule", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 405426484 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "snap_schedule", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 252423194 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.052 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "volumes", 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: "addrvec": [ 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v2", 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1044481762 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-28T18:54:13.053 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:13.054 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:11 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1013322091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:11 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3795278481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:12 smithi131 ceph-mon[36065]: pgmap v315: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 42 op/s 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:12 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/632000347' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:11 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1013322091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:11 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3795278481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:12 smithi196 ceph-mon[36300]: pgmap v315: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 42 op/s 2024-03-28T18:54:13.055 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:12 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/632000347' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.933+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 msgr2=0x7f29780a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.933+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 0x7f29780a4eb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f29840095f0 tx=0x7f298402e040 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.933+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 msgr2=0x7f299019eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.933+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f299006adc0 tx=0x7f297c00ee50 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 shutdown_connections 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f29780a29f0 0x7f29780a4eb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29901a2e60 0x7f29901a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f299010cf90 0x7f299019eae0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 --2- 172.21.15.119:0/632000347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2990107ec0 0x7f299019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 >> 172.21.15.119:0/632000347 conn(0x7f2990075bf0 msgr2=0x7f2990076f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 shutdown_connections 2024-03-28T18:54:13.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:11.934+0000 7f2997e00640 1 -- 172.21.15.119:0/632000347 wait complete. 2024-03-28T18:54:13.061 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T18:54:13.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T18:54:13.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:12 smithi119 ceph-mon[31813]: pgmap v315: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 42 op/s 2024-03-28T18:54:13.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:12 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/632000347' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T18:54:13.516 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:54:14.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:14 smithi119 ceph-mon[31813]: pgmap v316: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T18:54:14.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:14 smithi196 ceph-mon[36300]: pgmap v316: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T18:54:14.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:14 smithi131 ceph-mon[36065]: pgmap v316: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1539, 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1541, 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 461442, 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 489403, 2024-03-28T18:54:15.946 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 6, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.300233721, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.105116110, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.073116298, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.090375659, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1539, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.131351674, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000085348 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.947 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 856, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 843, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1893424, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 325096, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.229329420, 2024-03-28T18:54:15.948 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.074626830, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.114655336, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.054234078, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 842, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.085516863, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000101563 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1741728, 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 129776 2024-03-28T18:54:15.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 7, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 164, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 3612, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 445726, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 6, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.032187937, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.021771487, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.005140353, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000043922, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 162, 2024-03-28T18:54:15.950 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.029247809, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000180542 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 445408 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-28T18:54:15.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3410, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.187273903, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000054919 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.952 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-03-28T18:54:15.953 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-03-28T18:54:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 1, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:15.955 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:15.956 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-28T18:54:15.957 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:15.958 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 5, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-03-28T18:54:15.959 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 2, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 4195815, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 4195674, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 5 2024-03-28T18:54:15.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 3, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 3, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 1, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-03-28T18:54:15.961 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 48768, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215420 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.962 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.963 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.968 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.969 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-28T18:54:15.970 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-28T18:54:15.971 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.972 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-28T18:54:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 3168, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 189393, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 51, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 178, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-28T18:54:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 8320, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 13, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 2154, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 21, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 3, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 2154, 2024-03-28T18:54:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2154, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.250970655, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000580766 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2154, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2155 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 2154, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 2154, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 1378, 2024-03-28T18:54:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 774, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-28T18:54:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 99, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 13, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 5, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 5, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-28T18:54:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 2398, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 2127311, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 2398, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2398, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2127311, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2154, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 1516, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2154, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1516, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2154, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 2154, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2154, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2154, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55a3a560a460": { 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-28T18:54:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:15.990 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:16.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:16 smithi119 ceph-mon[31813]: pgmap v317: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:16.913 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:16.913 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T18:54:16.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T18:54:16.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:16 smithi196 ceph-mon[36300]: pgmap v317: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:54:16.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:16 smithi131 ceph-mon[36065]: pgmap v317: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 255 B/s wr, 16 op/s 2024-03-28T18:54:17.356 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:54:18.696 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:18 smithi119 ceph-mon[31813]: pgmap v318: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 15 op/s 2024-03-28T18:54:18.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:18 smithi196 ceph-mon[36300]: pgmap v318: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 15 op/s 2024-03-28T18:54:18.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:18 smithi131 ceph-mon[36065]: pgmap v318: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 15 op/s 2024-03-28T18:54:19.580 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:19.580 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:19.580 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 0, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 2, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 0, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 0, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 0, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.003702399, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.001020256, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:19.581 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 2, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 167, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 320, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 184608, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.031998232, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.022752338, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.003402711, 2024-03-28T18:54:19.582 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 164, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.031349934, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000191158 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 184608 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 233, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 219, 2024-03-28T18:54:19.583 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1716864, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 130848, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 2, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 1, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.095418310, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.031550301, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.076694939, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.009508075, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 219, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.024968976, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000114013 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:19.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1716864, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 130848 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:19.585 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:19.586 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 148714, 2024-03-28T18:54:19.587 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 30, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 928, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 2, 2024-03-28T18:54:19.588 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 231, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1681593, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 231, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 231, 2024-03-28T18:54:19.589 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1681593, 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:19.590 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-28T18:54:20.448 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-28T18:54:20.448 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T18:54:20.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T18:54:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:20 smithi119 ceph-mon[31813]: pgmap v319: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:54:20.905 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:54:20.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:20 smithi196 ceph-mon[36300]: pgmap v319: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:54:20.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:20 smithi131 ceph-mon[36065]: pgmap v319: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T18:54:21.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:21 smithi119 ceph-mon[31813]: pgmap v320: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:54:21.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:21 smithi196 ceph-mon[36300]: pgmap v320: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:54:21.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:21 smithi131 ceph-mon[36065]: pgmap v320: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-28T18:54:23.180 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:23.180 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:23.180 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1019, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1006, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 2002608, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 377780, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 1, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.290331034, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.117228803, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.119454775, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.056617211, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1005, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.114464745, 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000113895 2024-03-28T18:54:23.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1742544, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 131648 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1404, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1563, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 361126, 2024-03-28T18:54:23.182 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 891981, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 7, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.238393108, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.093575889, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.053926262, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.067503786, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1561, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.162850749, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000104324 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 447360 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 4, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 6, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1629, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 1284, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-28T18:54:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.003041339, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.000089639, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.002694803, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000077522, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.003646187, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000911546 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.185 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-28T18:54:23.186 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3667, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.173578048, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000047335 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-03-28T18:54:23.188 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 1, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-03-28T18:54:23.189 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:23.190 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:23.191 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-03-28T18:54:23.192 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:23.193 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 5, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-28T18:54:23.194 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 2, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 4195730, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 4195589, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.195 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 5 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 3, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 3, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 0, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 1, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 0, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 0, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 0, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 0, 2024-03-28T18:54:23.196 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 48768, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215420 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.197 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-28T18:54:23.198 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.199 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.204 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.205 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:23.206 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.207 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.208 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 3173, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-28T18:54:23.209 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:23.210 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:23.211 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 198504, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 56, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-28T18:54:23.212 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 178, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 8320, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 13, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-28T18:54:23.213 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 2176, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 21, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 3, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 2176, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2176, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.798957848, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000826726 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.214 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2176, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2177 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 2176, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 2176, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 1393, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 781, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-03-28T18:54:23.215 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-28T18:54:23.216 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-28T18:54:23.217 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 99, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 13, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 6, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 6, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-28T18:54:23.218 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-28T18:54:23.219 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 2423, 2024-03-28T18:54:23.220 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 2131790, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 2423, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2423, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2131790, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:23.221 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2176, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 1431, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2176, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1431, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2176, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 2176, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2176, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2176, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.223 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-28T18:54:23.224 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x5635962ec780": { 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:23.225 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:23.226 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:24.141 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:24.142 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T18:54:24.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T18:54:24.600 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T18:54:24.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:24 smithi119 ceph-mon[31813]: pgmap v321: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:54:24.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:24 smithi196 ceph-mon[36300]: pgmap v321: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:54:24.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:24 smithi131 ceph-mon[36065]: pgmap v321: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T18:54:25.934 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:25 smithi119 ceph-mon[31813]: pgmap v322: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:54:26.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:25 smithi196 ceph-mon[36300]: pgmap v322: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:54:26.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:25 smithi131 ceph-mon[36065]: pgmap v322: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T18:54:26.795 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-28T18:54:26.795 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:26.795 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 236, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 222, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1717680, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 132720, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 0, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.117480026, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.035659557, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.100570706, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.011123563, 2024-03-28T18:54:26.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 222, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.026746188, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000120478 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1717680, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 132720 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.797 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 2, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 168, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 320, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 184784, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.033972221, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.023211904, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.004748314, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:26.798 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 165, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.028905521, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000175184 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 184784 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.799 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 0, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 2, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 0, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 0, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 2, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 1, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.001068031, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.000890315, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:26.800 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-28T18:54:26.801 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:26.802 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:26.803 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:26.804 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 148731, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 31, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 0, 2024-03-28T18:54:26.805 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 928, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 2, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-28T18:54:26.806 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 234, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1681983, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 234, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 234, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1681983, 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-28T18:54:26.807 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-28T18:54:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-28T18:54:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-28T18:54:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-28T18:54:26.808 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-28T18:54:27.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:26 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:27.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:26 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:27.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:26 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T18:54:27.610 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-28T18:54:27.611 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-28T18:54:27.611 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T18:54:27.612 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T18:54:27.895 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:54:28.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: pgmap v323: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:28.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:28 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: pgmap v323: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:28.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:28 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: pgmap v323: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:28.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:28 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: pgmap v324: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:30.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.336 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:30 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: pgmap v324: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:30 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: pgmap v324: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:30 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 3038, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 3034, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 859179, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 2291253, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 14, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 3, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.203034367, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.060647413, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.056715412, 2024-03-28T18:54:30.581 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.066219782, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3032, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.993226459, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000327581 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1038, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000185454, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000178 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 313520 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 4598, 2024-03-28T18:54:30.582 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 3301, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 3193830, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 1352450, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 27, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 7, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.332793620, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.083763109, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.146385354, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.109349258, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3300, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.289864325, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000087837 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:30.583 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2261, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000459209, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000203 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 2150560, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 284160 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 2755, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 2343, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 985384, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 2477236, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 156, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.218292572, 2024-03-28T18:54:30.584 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.060495037, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.092735342, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.055031964, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2341, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.267310481, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000114186 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2403, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000352607, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000146 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 89024, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 223968 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-03-28T18:54:30.585 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 664, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.196371119, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000295739 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:30.586 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 4195, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.136276178, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000032485 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 179, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 179, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 179, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.158182112, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000883698 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:30.587 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 12, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 36, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 2, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 34, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 33, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 2, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 3, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 2, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 275, 2024-03-28T18:54:30.588 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 217, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 232, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:30.589 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 4, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 234, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 6, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 232, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 149, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 20, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 129, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 83, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 21, 2024-03-28T18:54:30.590 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 109, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 153, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 121, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 5, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 26 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 8, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 0, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 20, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 2, 2024-03-28T18:54:30.591 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 5, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 3, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 20, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-03-28T18:54:30.592 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 258, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 258, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 2, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 2, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 2, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-03-28T18:54:30.593 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 5224869, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 0, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 226, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.415208299, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.001837204 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 0 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 27, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 50, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 23, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 20, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 20, 2024-03-28T18:54:30.594 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 0, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 36, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 39, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 3, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 3, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 21, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 18, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 54584, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 231804 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 239, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 179, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 34, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-03-28T18:54:30.595 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 7, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.011687888, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.001669698 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 98, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.017113999, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000174632 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-03-28T18:54:30.596 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 24, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.005535463, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000230644 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000157671, 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000157671 2024-03-28T18:54:30.597 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.598 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 10, 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.001696957, 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000169695 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 8, 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.015461635, 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.001932704 2024-03-28T18:54:30.599 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 10, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.045189485, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.004518948 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.600 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.001066591, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000533295 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.004034805, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002017402 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:30.601 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.010996053, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.005498026 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 4, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.016083265, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.004020816 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:30.602 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000889063, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000889063 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.603 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 7, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.018969083, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002709869 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.604 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.009300154, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.003100051 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.605 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 2, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 5, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 3, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 2, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 27, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 13, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 605, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:30.606 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:30.607 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:30.608 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 379215, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 153, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 792, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 9, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 178, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:30.609 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 133693, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 1116, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 5709, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 1050322, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 37, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 5709, 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-03-28T18:54:30.610 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 5709, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 55.238832307, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.009675745 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 5709, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 5760 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 5709, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 275, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 5434, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 0, 2024-03-28T18:54:30.611 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 11, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 23, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 229, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 33, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 5122, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 252, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 22, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:30.612 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-03-28T18:54:30.613 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 99, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 14, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 12, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 12, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-03-28T18:54:30.614 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 59, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 0, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 9 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 1, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 3584, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-03-28T18:54:30.615 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 1, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 4610, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 3, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 10387, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 4049392, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 10387, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 10387, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 4049392, 2024-03-28T18:54:30.616 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 5709, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 1050641, 2024-03-28T18:54:30.617 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 5709, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1050641, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:30.618 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 5709, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 5709, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 5709, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 5709, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 3, 2024-03-28T18:54:30.619 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 319, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 3, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 3, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 319, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x55e3bc0be500": { 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:30.620 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 258, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 1030565, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 258, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 226, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1030565, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:30.621 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=8 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-28T18:54:31.303 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-28T18:54:31.303 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T18:54:31.304 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T18:54:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:31 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:54:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:31 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:54:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:31 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:31 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:31 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:54:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:31 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:54:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:31 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:31 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:31 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-28T18:54:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:31 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T18:54:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:31 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:31 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:31.593 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: pgmap v325: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:32 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: pgmap v325: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:32.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:32 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: pgmap v325: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:32.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:32 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.369 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:33.370 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:33 smithi131 ceph-mon[36065]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:33.370 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-28T18:54:33.370 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:33.370 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 1674, 2024-03-28T18:54:33.370 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 1676, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 1526200, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 532342, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 6, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.106043643, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.037737790, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.026333869, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.032878247, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1674, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.035844405, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000021412 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.371 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 10, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 174, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 5000, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 452778, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 7, 2024-03-28T18:54:33.372 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.008478952, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.004944538, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.002006558, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.000089789, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 172, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.036388813, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000211562 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.373 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 451440 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 1027, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 1014, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 1935057, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 382003, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 3, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.091635313, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.023057875, 2024-03-28T18:54:33.374 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.049000153, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.023990932, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1013, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.025485131, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000025158 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1744336, 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 135552 2024-03-28T18:54:33.375 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 4096, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.081215212, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000019827 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:33.376 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 0, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 0, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.377 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 27, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 27, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 27, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:33.378 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 2, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-03-28T18:54:33.379 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 4, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 234, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 6, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:33.380 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 0 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 0, 2024-03-28T18:54:33.381 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:33.382 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 258, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-03-28T18:54:33.383 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 2, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 2, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 5224869, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 5224757, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.384 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 258 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 18, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 21, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 3, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 11, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 11, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 0, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 27, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 27, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 0, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 0, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 0, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 0, 2024-03-28T18:54:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 47776, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 215420 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.386 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.387 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.388 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.389 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.390 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.391 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.392 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.393 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.394 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.395 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.396 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 2, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 5, 2024-03-28T18:54:33.397 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 3, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 4, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 3814, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:33.398 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:33.399 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:33.400 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 190472, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 61, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 178, 2024-03-28T18:54:33.401 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 90485, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 648, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 2453, 2024-03-28T18:54:33.402 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 26, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 4, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 2453, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2453, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.863982021, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000352214 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2453, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 2454 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 2453, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 2453, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 0, 2024-03-28T18:54:33.403 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 1559, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 891, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 2, 2024-03-28T18:54:33.404 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-03-28T18:54:33.405 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 99, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-03-28T18:54:33.406 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 13, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 5, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 5, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 2, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 0 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-03-28T18:54:33.407 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:33.408 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 2707, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 3206766, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 2707, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2707, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 3206766, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:33.409 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2453, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 1030575, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2453, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1030575, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.410 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2453, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 2453, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2453, 2024-03-28T18:54:33.411 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2453, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:33.412 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x5588262306e0": { 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:33.413 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:33.414 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:33.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:33.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:33 smithi196 ceph-mon[36300]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi119 to 176.7M 2024-03-28T18:54:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi119 to 185372364: error parsing value: Value '185372364' is below minimum 939524096 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: Adjusting osd_memory_target on smithi131 to 174.1M 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: Unable to set osd_memory_target on smithi131 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T18:54:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:33 smithi119 ceph-mon[31813]: from='mgr.25418 ' entity='mgr.x' 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:34.224 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:34.224 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T18:54:34.225 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T18:54:34.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:34 smithi196 ceph-mon[36300]: pgmap v326: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:34.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:34 smithi131 ceph-mon[36065]: pgmap v326: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:34.479 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:54:34.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:34 smithi119 ceph-mon[31813]: pgmap v326: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 478, 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 734, 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 136141, 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 534859, 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 10, 2024-03-28T18:54:36.318 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.053777493, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.023906346, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.023615992, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.003194578, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 737, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.073445607, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000099654 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1023, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000209096, 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000204 2024-03-28T18:54:36.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 313152 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 352, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 349, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 1786930, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 213490, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 12, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.045845095, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.011043523, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.045957925, 2024-03-28T18:54:36.320 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.004208010, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 348, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.036687191, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000105422 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 97, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000018208, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000187 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1762880, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 143920 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:36.321 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 149, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 192, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 48832, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 627429, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 51, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 6, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.020302959, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.004414907, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.014750266, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.000107279, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 185, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.017588786, 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000095074 2024-03-28T18:54:36.322 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 175, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000029471, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000168 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 140688 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-03-28T18:54:36.323 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 57, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.001297214, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000022758 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000006621, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000003310 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.324 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 11, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-03-28T18:54:36.325 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 14, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 1, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 13, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 12, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 2, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 4, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 2, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 2, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:36.326 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 4, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 234, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 6, 2024-03-28T18:54:36.327 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 0, 2024-03-28T18:54:36.328 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 0 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 6, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 4, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 5, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 0, 2024-03-28T18:54:36.329 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-03-28T18:54:36.330 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 5, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 5, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 2, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 2, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 2, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-03-28T18:54:36.331 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 4195815, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 0, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 4, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.009814181, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.002453545 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 0 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 17, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 17, 2024-03-28T18:54:36.332 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 16, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 16, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 14, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 14, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 49736, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 223612 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:36.333 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 0, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 12, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:36.334 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.335 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.336 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-03-28T18:54:36.337 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.338 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.339 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.340 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.341 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.342 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.343 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.344 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 1, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 2, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 1, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 1, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 573, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:36.345 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:36.346 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:36.347 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 255159, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 86, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 178, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-03-28T18:54:36.348 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 67936, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 251, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 41, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 11903, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 28, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 41, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 41, 2024-03-28T18:54:36.349 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.307993430, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.007512034 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 41, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 63 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 41, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 0, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 41, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 0, 2024-03-28T18:54:36.350 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 10, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 4, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 8, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 6, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 20, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:36.351 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-03-28T18:54:36.352 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 99, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 13, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 8, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 8, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 14, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 0, 2024-03-28T18:54:36.353 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 1 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 1, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:36.354 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 975, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 1866546, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 975, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 975, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1866546, 2024-03-28T18:54:36.355 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 41, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 11903, 2024-03-28T18:54:36.356 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 41, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 11903, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:36.357 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 41, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 41, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 41, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 41, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:36.358 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x555fee56e500": { 2024-03-28T18:54:36.359 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 5, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 1511, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 5, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 4, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 1511, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:36.361 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-28T18:54:36.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:36 smithi131 ceph-mon[36065]: pgmap v327: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:36.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:36 smithi119 ceph-mon[31813]: pgmap v327: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:36.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:36 smithi196 ceph-mon[36300]: pgmap v327: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=6 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:37.146 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:37.146 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T18:54:37.147 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T18:54:37.403 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T18:54:38.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:38 smithi131 ceph-mon[36065]: pgmap v328: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:38.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:38 smithi119 ceph-mon[31813]: pgmap v328: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:38.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:38 smithi196 ceph-mon[36300]: pgmap v328: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:39.074 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 798, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 800, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 211678, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 253828, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 6, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.051260587, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.019266966, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.012642899, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.014326571, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 798, 2024-03-28T18:54:39.075 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.018191525, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000022796 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 703, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 872, 2024-03-28T18:54:39.076 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 172788, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 676043, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 7, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.052927721, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.017784339, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.011658682, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.018172252, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 869, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.035091563, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000040381 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.077 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 453120 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_messages": 1049, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages": 1034, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_bytes": 2013728, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_bytes": 389556, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_created_connections": 3, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_total_time": 0.095924557, 2024-03-28T18:54:39.078 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_send_time": 0.026398012, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_recv_time": 0.053752330, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_running_fast_dispatch_time": 0.023454706, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 1034, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.032590672, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000031519 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_recv_encrypted_bytes": 1748288, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "msgr_send_encrypted_bytes": 138336 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.079 INFO:teuthology.orchestra.run.smithi131.stdout: "cct": { 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "total_workers": 1, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "unhealthy_workers": 0 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-MDSRank": { 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 3869, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.077982756, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000020155 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "queue_len": 0, 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "complete_latency": { 2024-03-28T18:54:39.080 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "request": 0, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "reply": 0, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "reply_latency": { 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.081 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_reply": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "forward": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_commit": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_split": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_merge": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_top": 0, 2024-03-28T18:54:39.082 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_bottom": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_pinned": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_expired": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "caps": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "subtrees": 1, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_hit": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_forward": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_discover": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "traverse_lock": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "load_cent": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "q": 0, 2024-03-28T18:54:39.083 INFO:teuthology.orchestra.run.smithi131.stdout: "exported": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "exported_inodes": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "imported": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "imported_inodes": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rfiles": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rbytes": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "root_rsnaps": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:39.084 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "process_request_cap_release": 0 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_cache": { 2024-03-28T18:54:39.085 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_try_discover": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_send_discover": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_created": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_enqueued": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "strays_migrated": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:39.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_started": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "recovery_completed": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_flush": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ireq_inodestats": 0 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_log": { 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "evlrg": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "evadd": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "evex": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "evtrm": 0, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "ev": 7, 2024-03-28T18:54:39.087 INFO:teuthology.orchestra.run.smithi131.stdout: "evexg": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "evexd": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segadd": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segex": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segtrm": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "seg": 2, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segmjr": 2, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segexg": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "segexd": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "expos": 4194304, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "wrpos": 4196347, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "rdpos": 4196206, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "jlat": { 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "replayed": 7 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_mem": { 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "ino": 3, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "ino+": 3, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "ino-": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dir": 1, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dir+": 1, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dir-": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dn": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dn+": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "dn-": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "cap": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "cap+": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "cap-": 0, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "rss": 50132, 2024-03-28T18:54:39.089 INFO:teuthology.orchestra.run.smithi131.stdout: "heap": 215420 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_server": { 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_request": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_client_session": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "handle_peer_request": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "req_create_latency": { 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getattr_latency": { 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.090 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "req_link_latency": { 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookup_latency": { 2024-03-28T18:54:39.091 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupino_latency": { 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupname_latency": { 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.092 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "req_lssnap_latency": { 2024-03-28T18:54:39.093 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mkdir_latency": { 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.094 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mknod_latency": { 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "req_mksnap_latency": { 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "req_open_latency": { 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.095 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "req_readdir_latency": { 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rename_latency": { 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.096 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmdir_latency": { 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.097 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setattr_latency": { 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.098 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setlayout_latency": { 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "req_setxattr_latency": { 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.099 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "req_symlink_latency": { 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "req_unlink_latency": { 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:39.100 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_sessions": { 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "session_count": 1, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "session_add": 3, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "session_remove": 2, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_open": 2, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "sessions_stale": 0, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "total_load": 0, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "average_load": 0, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "avg_session_uptime": 3818, 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:39.101 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "mempool": { 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:39.102 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:39.103 INFO:teuthology.orchestra.run.smithi131.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_bytes": 190179, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_anon_items": 53, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:39.104 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_items": 178, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "pgmap_items": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_bytes": 8320, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "mds_co_items": 13, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_1_items": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "unittest_2_items": 0 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "objecter": { 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "op_active": 0, 2024-03-28T18:54:39.105 INFO:teuthology.orchestra.run.smithi131.stdout: "op_laggy": 0, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send": 2289, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_send_bytes": 21, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_resend": 3, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_reply": 2289, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_latency": { 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2289, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 1.886960780, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000824360 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_inflight": 0, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "oplen_avg": { 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 2289, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 2290 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op": 2289, 2024-03-28T18:54:39.106 INFO:teuthology.orchestra.run.smithi131.stdout: "op_r": 2289, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "op_w": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "op_rmw": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "op_pg": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_stat": 1489, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_create": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_read": 798, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_write": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writefull": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_writesame": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_append": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_zero": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_truncate": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_delete": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_mapext": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:39.107 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_getxattr": 1, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_setxattr": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_call": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_watch": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_notify": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "osdop_other": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_active": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_send": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_resend": 0, 2024-03-28T18:54:39.108 INFO:teuthology.orchestra.run.smithi131.stdout: "linger_ping": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_active": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_send": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolop_resend": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_active": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_send": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "poolstat_resend": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_active": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_send": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "statfs_resend": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "command_active": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "command_send": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "command_resend": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "map_epoch": 99, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "map_full": 0, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "map_inc": 13, 2024-03-28T18:54:39.109 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_sessions": 5, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_open": 5, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_session_close": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "osd_laggy": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_wr": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_rd": 2, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_del": 0 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "oft": { 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_objs": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_updates": 0, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "omap_total_removes": 0 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.110 INFO:teuthology.orchestra.run.smithi131.stdout: "osd-slow-ops": { 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "slow_ops_count": 0 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "purge_queue": { 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_executed": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 2546, 2024-03-28T18:54:39.111 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 2153031, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 2546, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2546, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2153031, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 104857600, 2024-03-28T18:54:39.112 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2289, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 2048, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2289, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2048, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:39.113 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 1024, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 2289, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 2289, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 2289, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 2289, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.114 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.115 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "throttle-write_buf_throttle-0x561f1ef366e0": { 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "max": 3758096384, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "get_started": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "get": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "get_sum": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "take": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "take_sum": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "put": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "put_sum": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "wait": { 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "avgcount": 0, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "sum": 0.000000000, 2024-03-28T18:54:39.116 INFO:teuthology.orchestra.run.smithi131.stdout: "avgtime": 0.000000000 2024-03-28T18:54:39.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.117 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-03-28T18:54:39.117 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:39.865 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:39.865 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T18:54:39.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T18:54:40.194 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:54:40.596 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:40 smithi196 ceph-mon[36300]: pgmap v329: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:40.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:40 smithi119 ceph-mon[31813]: pgmap v329: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:40.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:40 smithi131 ceph-mon[36065]: pgmap v329: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:42.039 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-28T18:54:42.039 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 809, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 919, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1919915, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 399086, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 8, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.086157823, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.027533802, 2024-03-28T18:54:42.040 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.062909123, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.007786655, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 924, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.048760747, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000052771 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1122, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000206546, 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000184 2024-03-28T18:54:42.041 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1766816, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 140016 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 119, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 172, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 35302, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 673718, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 50, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 8, 2024-03-28T18:54:42.042 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.023909823, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.004988679, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.018245568, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.000356786, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 169, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.024436936, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000144597 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:42.043 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 160, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000027693, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000173 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 5840, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 226112 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 76, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 212, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 26566, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 409039, 2024-03-28T18:54:42.044 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 17, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 7, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.012962430, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.004892581, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.007045895, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.000359941, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 204, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.045893817, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000224969 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:42.045 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 111, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000018436, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000166 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1744, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 315920 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.046 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 61, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.001586711, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000026011 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2, 2024-03-28T18:54:42.047 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000009009, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000004504 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 2, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 1, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000161840, 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000161840 2024-03-28T18:54:42.048 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 1, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 11, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 14, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 1, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 13, 2024-03-28T18:54:42.049 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 12, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 0, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 2, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 8, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 4, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 0, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 3, 2024-03-28T18:54:42.050 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:42.051 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 4, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 234, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 6, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:42.052 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 1 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 6, 2024-03-28T18:54:42.053 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 4, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 6, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:42.054 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-28T18:54:42.055 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 7, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 7, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 2, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-28T18:54:42.056 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 4196347, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 0, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 6, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.009935704, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.001655950 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 0 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.057 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 17, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 17, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 0, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 16, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 17, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 1, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 14, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 14, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 0, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 0, 2024-03-28T18:54:42.058 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 1, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 1, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 49356, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 223612 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 3, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 2, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 16, 2024-03-28T18:54:42.059 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000161840, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000161840 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.060 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.061 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.062 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.063 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.064 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-28T18:54:42.066 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.068 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.069 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.070 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.071 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.072 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 3, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 2, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 1, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 1, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 576, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:42.074 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:42.075 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 239767, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 84, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-28T18:54:42.076 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 178, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 67968, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 255, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:42.077 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 44, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 12525, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 28, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 44, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 44, 2024-03-28T18:54:42.078 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.362992146, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.008249821 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 44, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 66 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 44, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 0, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 44, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 0, 2024-03-28T18:54:42.079 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 10, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 6, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 9, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 6, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:42.080 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 20, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:42.081 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-28T18:54:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 99, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 13, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 8, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 8, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 14, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 0, 2024-03-28T18:54:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 1 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 1, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:42.084 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 1000, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1873672, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 1000, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1000, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1873672, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 44, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 12525, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 44, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 12525, 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:42.087 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:42.088 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 44, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 44, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 44, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 44, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.089 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-28T18:54:42.090 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x557cf2bf8500": { 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 7, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 2043, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 7, 2024-03-28T18:54:42.091 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 6, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2043, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:42.092 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-28T18:54:42.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:42 smithi196 ceph-mon[36300]: pgmap v330: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:42 smithi119 ceph-mon[31813]: pgmap v330: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:42.906 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T18:54:42.906 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T18:54:42.907 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-28T18:54:42.907 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=6 2024-03-28T18:54:42.907 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-28T18:54:42.907 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:42.907 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:42.907 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T18:54:42.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T18:54:42.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:42 smithi131 ceph-mon[36065]: pgmap v330: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:43.175 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:54:44.611 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:44 smithi196 ceph-mon[36300]: pgmap v331: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:44 smithi119 ceph-mon[31813]: pgmap v331: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 408, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 397, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1802653, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 299548, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 20, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.053330988, 2024-03-28T18:54:44.909 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.012615912, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.064088162, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.004397368, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 392, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.023606490, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000060220 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:44.910 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 191, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000055876, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000292 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1757072, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 142064 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 484, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 790, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 127775, 2024-03-28T18:54:44.911 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 1056681, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 45, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.056953502, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.021026425, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.030459857, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.003338465, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 791, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.068833679, 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000087021 2024-03-28T18:54:44.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1044, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000185451, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000177 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 459008 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.913 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 86, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 114, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 32393, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 38083, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 9, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 5, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.011341818, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.003779183, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.007046250, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.000470260, 2024-03-28T18:54:44.914 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 113, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.029712047, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000262938 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 111, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000027301, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000245 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-28T18:54:44.915 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 58, 2024-03-28T18:54:44.916 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.001580383, 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000027247 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2, 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000006046, 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000003023 2024-03-28T18:54:44.917 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 4, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 3, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000483384, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000161128 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-28T18:54:44.918 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 1, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 11, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 13, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 13, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 13, 2024-03-28T18:54:44.919 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 1, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 1, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 2, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 7, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 5, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 1, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 1, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:44.920 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 4, 2024-03-28T18:54:44.921 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 234, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 6, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:44.922 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 0 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-28T18:54:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 3, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 2, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 3, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-28T18:54:44.924 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.925 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 5, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 5, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 2, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-28T18:54:44.926 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 4195730, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 0, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 4, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.010037870, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.002509467 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.927 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 0 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 16, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 16, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 0, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 15, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 15, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 0, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 13, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 13, 2024-03-28T18:54:44.928 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 0, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 1, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 3, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 2, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 48868, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 223612 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 4, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 4, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 13, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-28T18:54:44.929 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000200546, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000200546 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.930 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-28T18:54:44.931 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000152971, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000152971 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-28T18:54:44.932 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000129867, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000129867 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.933 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.934 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-28T18:54:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.935 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.936 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-28T18:54:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.937 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-28T18:54:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-28T18:54:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-28T18:54:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-28T18:54:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.942 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-28T18:54:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.945 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.946 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-28T18:54:44.947 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.948 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.949 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 2, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 1, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-28T18:54:44.950 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 2, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 1, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 638, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:44.951 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:44.952 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:44.953 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 280261, 2024-03-28T18:54:44.954 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 100, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 178, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:44.955 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 64232, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 247, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-28T18:54:44.956 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-28T18:54:44.957 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 43, 2024-03-28T18:54:44.957 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 12079, 2024-03-28T18:54:44.957 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 33, 2024-03-28T18:54:44.957 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 43, 2024-03-28T18:54:44.957 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-28T18:54:44.958 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 43, 2024-03-28T18:54:44.958 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.332992799, 2024-03-28T18:54:44.958 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.007744018 2024-03-28T18:54:44.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.958 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 43, 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 68 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 43, 2024-03-28T18:54:44.959 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 43, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 10, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 4, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 7, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-28T18:54:44.960 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 6, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 20, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:44.961 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-28T18:54:44.962 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-28T18:54:44.963 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 99, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 13, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 9, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 9, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 20, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 0, 2024-03-28T18:54:44.964 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 1 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 1, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:44.965 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:44.966 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 974, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1857576, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 974, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 974, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1857576, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.967 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 43, 2024-03-28T18:54:44.968 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 12079, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 43, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 12079, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-28T18:54:44.969 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 43, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 43, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 43, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 43, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.970 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:44.971 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x55f0dfb9a500": { 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 5, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1426, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:44.972 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 5, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 4, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1426, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:44.973 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-28T18:54:44.974 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:44 smithi131 ceph-mon[36065]: pgmap v331: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:45.766 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T18:54:45.766 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T18:54:45.767 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-28T18:54:45.767 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=3 2024-03-28T18:54:45.767 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-28T18:54:45.767 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:45.767 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:45.767 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T18:54:45.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T18:54:46.032 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:54:46.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:46 smithi196 ceph-mon[36300]: pgmap v332: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:46 smithi119 ceph-mon[31813]: pgmap v332: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:46.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:46 smithi131 ceph-mon[36065]: pgmap v332: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 540, 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 812, 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 155555, 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 708611, 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 11, 2024-03-28T18:54:47.762 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 4, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.055198107, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.022131131, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.024954008, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.004132624, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 814, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.064539398, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000079286 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1137, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000203711, 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000179 2024-03-28T18:54:47.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 316256 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 101, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 186, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 32930, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 271389, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 22, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 7, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.017769936, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.005218292, 2024-03-28T18:54:47.764 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.011538734, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.000270981, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 179, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.019678515, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000109935 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 189, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000246690, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000001305 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 5504, 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 154080 2024-03-28T18:54:47.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 392, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 370, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1803258, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 565230, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 44, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 8, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.056931547, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.012840245, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.054198192, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.004320127, 2024-03-28T18:54:47.766 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 370, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.020537985, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000055508 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 124, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000019501, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000157 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1775360, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 142240 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-28T18:54:47.767 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 65, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.001652241, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000025419 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:47.768 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000007761, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000003880 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 7, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 3, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000641580, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000213860 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-28T18:54:47.769 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 4, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 11, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 14, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 1, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 13, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 12, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 1, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 1, 2024-03-28T18:54:47.770 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 2, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 13, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 9, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 1, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 2, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-28T18:54:47.771 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 4, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 234, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 6, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:47.772 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 0 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 8, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 5, 2024-03-28T18:54:47.773 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 6, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-28T18:54:47.774 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 10, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 10, 2024-03-28T18:54:47.775 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 2, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 4197185, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 0, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 9, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.014345816, 2024-03-28T18:54:47.776 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.001593979 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 0 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 17, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 17, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 0, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 16, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 17, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 1, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 14, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 14, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 0, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 1, 2024-03-28T18:54:47.777 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 1, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 0, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 49364, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 223612 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 7, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 7, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 29, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.778 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000641580, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000213860 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.779 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-28T18:54:47.780 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:47.781 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.782 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-28T18:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.786 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.787 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.788 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.789 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 2, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 5, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 3, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 2, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 2, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 0, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 625, 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:47.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:47.791 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:47.792 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 257139, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 103, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 176, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 2, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:47.793 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 178, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 68344, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 258, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-28T18:54:47.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 48, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 13453, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 33, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 48, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 48, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.286993774, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.005979036 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-28T18:54:47.795 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 48, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 70 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 48, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 48, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 10, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 9, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 10, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-28T18:54:47.796 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 6, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 20, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-28T18:54:47.797 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-28T18:54:47.798 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 99, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 13, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 9, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 9, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 14, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 0, 2024-03-28T18:54:47.799 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 1 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 1, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-28T18:54:47.800 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 1029, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1880534, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 1029, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:47.801 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1029, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1880534, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:47.802 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 48, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 13453, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 48, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 13453, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:47.803 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 48, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 48, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 48, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 48, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:47.804 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x558c32fd2500": { 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 10, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 2881, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 10, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 9, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2881, 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:47.806 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:47.807 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:47.807 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:47.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:47.807 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-28T18:54:48.588 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=8 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:48.589 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:48.589 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T18:54:48.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T18:54:48.684 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:48 smithi196 ceph-mon[36300]: pgmap v333: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:48 smithi119 ceph-mon[31813]: pgmap v333: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:48.873 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T18:54:48.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:48 smithi131 ceph-mon[36065]: pgmap v333: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:50.644 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:50 smithi196 ceph-mon[36300]: pgmap v334: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 2323, 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 2325, 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 659974, 2024-03-28T18:54:50.645 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 738721, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 6, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.138429903, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.041017173, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.034853869, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.053246153, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2323, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.044143585, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000019002 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.646 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 4, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 178, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 876, 2024-03-28T18:54:50.647 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 538970, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 7, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 4, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.008732228, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.005286011, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.001906200, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.000040606, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 175, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.019583480, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000111905 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.648 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 538272 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 264, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 249, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1749644, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 141150, 2024-03-28T18:54:50.649 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 3, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.039678463, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.009697838, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.038026658, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.004416920, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 249, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.007866000, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000031590 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-28T18:54:50.650 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1749376, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 140832 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:50.651 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3933, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.083647448, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000021268 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.652 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 0, 2024-03-28T18:54:50.653 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 1, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 0, 2024-03-28T18:54:50.654 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 0, 2024-03-28T18:54:50.655 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T18:54:50.656 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 0 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-28T18:54:50.657 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-28T18:54:50.658 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 10, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-28T18:54:50.659 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 4197185, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 4197044, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 10 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-28T18:54:50.660 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 3, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 3, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 1, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 1, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 0, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 47968, 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 215420 2024-03-28T18:54:50.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.662 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.663 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.664 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.665 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.667 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-28T18:54:50.668 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-28T18:54:50.669 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.670 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.671 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.672 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 2, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 5, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 3, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 4, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-28T18:54:50.673 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 3732, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T18:54:50.674 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T18:54:50.675 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 182059, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 48, 2024-03-28T18:54:50.676 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8520, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 178, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 8320, 2024-03-28T18:54:50.677 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 13, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 2326, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 21, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 3, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 2326, 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-28T18:54:50.678 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2326, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.942980096, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000405408 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2326, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 2327 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 2326, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 2326, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 0, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 1513, 2024-03-28T18:54:50.679 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 811, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 1, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-28T18:54:50.680 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-28T18:54:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 99, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 13, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 4, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 4, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-28T18:54:50.682 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 0, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 2, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 0 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 0, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-28T18:54:50.683 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 2587, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 2161433, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:50.684 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 2587, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2587, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2161433, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:50.685 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2326, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 2886, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2326, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2886, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-28T18:54:50.686 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2326, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 2326, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2326, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2326, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.687 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.688 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x5651e7fee6e0": { 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-28T18:54:50.689 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-28T18:54:50.690 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-28T18:54:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:50 smithi119 ceph-mon[31813]: pgmap v334: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:50.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:50 smithi131 ceph-mon[36065]: pgmap v334: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T18:54:51.358 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-28T18:54:51.358 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-28T18:54:51.358 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-28T18:54:51.358 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-28T18:54:51.358 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-28T18:54:51.359 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-28T18:54:51.359 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T18:54:51.359 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T18:54:51.370 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T18:54:51.370 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:54:51.370 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:54:51.430 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-28T18:54:51.430 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T18:54:51.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 -- 172.21.15.119:0/926026057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe318107ec0 msgr2=0x7fe3181082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 --2- 172.21.15.119:0/926026057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe318107ec0 0x7fe3181082a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe308009860 tx=0x7fe30802bb70 comp rx=0 tx=0).stop 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 -- 172.21.15.119:0/926026057 shutdown_connections 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 --2- 172.21.15.119:0/926026057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31810cf90 0x7fe318114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 --2- 172.21.15.119:0/926026057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe318108870 0x7fe31810c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 --2- 172.21.15.119:0/926026057 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe318107ec0 0x7fe3181082a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 -- 172.21.15.119:0/926026057 >> 172.21.15.119:0/926026057 conn(0x7fe318075bf0 msgr2=0x7fe318078010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 -- 172.21.15.119:0/926026057 shutdown_connections 2024-03-28T18:54:51.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.698+0000 7fe31e200640 1 -- 172.21.15.119:0/926026057 wait complete. 2024-03-28T18:54:51.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.699+0000 7fe31e200640 1 Processor -- start 2024-03-28T18:54:51.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.699+0000 7fe31e200640 1 -- start start 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe31810cf90 0x7fe31819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3181a2f60 0x7fe3181a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318118040 con 0x7fe3181a2f60 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318117ec0 con 0x7fe31810cf90 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe31e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3181181c0 con 0x7fe318107ec0 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37674/0 (socket says 172.21.15.119:37674) 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 -- 172.21.15.119:0/3445482865 learned_addr learned my addr 172.21.15.119:0/3445482865 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:51.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317e00640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3181a2f60 0x7fe3181a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe30ea00640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe31810cf90 0x7fe31819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 -- 172.21.15.119:0/3445482865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe31810cf90 msgr2=0x7fe31819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe31810cf90 0x7fe31819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 -- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3181a2f60 msgr2=0x7fe3181a3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3181a2f60 0x7fe3181a3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.700+0000 7fe317400640 1 -- 172.21.15.119:0/3445482865 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3181a3a50 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe317400640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe308037040 tx=0x7fe3080045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308041020 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe30802c9f0 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3181a3ce0 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308039040 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.701+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181ab4a0 con 0x7fe318107ec0 2024-03-28T18:54:51.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.702+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3180688e0 con 0x7fe318107ec0 2024-03-28T18:54:51.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.704+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe308044020 con 0x7fe318107ec0 2024-03-28T18:54:51.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.706+0000 7fe314c00640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 0x7fe2fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:51.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.706+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fe3080f0ad0 con 0x7fe318107ec0 2024-03-28T18:54:51.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.706+0000 7fe30ea00640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 0x7fe2fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:51.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.707+0000 7fe30ea00640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 0x7fe2fc0a4e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe318107030 tx=0x7fe300005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:51.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.713+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe30804b020 con 0x7fe318107ec0 2024-03-28T18:54:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.914+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe31819f9f0 con 0x7fe318107ec0 2024-03-28T18:54:51.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.916+0000 7fe314c00640 1 -- 172.21.15.119:0/3445482865 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+23434 (secure 0 0 0) 0x7fe31819f9f0 con 0x7fe318107ec0 2024-03-28T18:54:51.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:51.917 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:54:09.747892+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/393583864":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6801/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/1763050303":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2790763787":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/748679560":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:54:51.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.918+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 msgr2=0x7fe2fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:51.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 0x7fe2fc0a4e30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe318107030 tx=0x7fe300005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 msgr2=0x7fe31819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe308037040 tx=0x7fe3080045b0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 shutdown_connections 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe2fc0a2970 0x7fe2fc0a4e30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3181a2f60 0x7fe3181a3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe31810cf90 0x7fe31819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 --2- 172.21.15.119:0/3445482865 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe318107ec0 0x7fe31819e6a0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 >> 172.21.15.119:0/3445482865 conn(0x7fe318075bf0 msgr2=0x7fe318076f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 shutdown_connections 2024-03-28T18:54:51.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:51.919+0000 7fe31e200640 1 -- 172.21.15.119:0/3445482865 wait complete. 2024-03-28T18:54:51.939 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T18:36:13.518512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T18:41:18.605969+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T18:41:20.021661+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T18:54:51.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 -- 172.21.15.119:0/996543784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8103840 msgr2=0x7fddf8103cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 --2- 172.21.15.119:0/996543784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8103840 0x7fddf8103cc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009930 tx=0x7fdde402bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 -- 172.21.15.119:0/996543784 shutdown_connections 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 --2- 172.21.15.119:0/996543784 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8104200 0x7fddf8110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 --2- 172.21.15.119:0/996543784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8103840 0x7fddf8103cc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 --2- 172.21.15.119:0/996543784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 -- 172.21.15.119:0/996543784 >> 172.21.15.119:0/996543784 conn(0x7fddf80ff2f0 msgr2=0x7fddf8101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 -- 172.21.15.119:0/996543784 shutdown_connections 2024-03-28T18:54:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.154+0000 7fddfe800640 1 -- 172.21.15.119:0/996543784 wait complete. 2024-03-28T18:54:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.155+0000 7fddfe800640 1 Processor -- start 2024-03-28T18:54:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.155+0000 7fddfe800640 1 -- start start 2024-03-28T18:54:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8104200 0x7fddf819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8113c70 con 0x7fddf8109850 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8113af0 con 0x7fddf8103840 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddfe800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8113df0 con 0x7fddf8104200 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59362/0 (socket says 172.21.15.119:59362) 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 -- 172.21.15.119:0/1935977713 learned_addr learned my addr 172.21.15.119:0/1935977713 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8104200 0x7fddf819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7e00640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 -- 172.21.15.119:0/1935977713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8104200 msgr2=0x7fddf819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8104200 0x7fddf819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 -- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 msgr2=0x7fddf819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 -- 172.21.15.119:0/1935977713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf819f180 con 0x7fddf8103840 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7e00640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.156+0000 7fddf7400640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9f0 tx=0x7fdde800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.157+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde800cd80 con 0x7fddf8103840 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.157+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdde8010910 con 0x7fddf8103840 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.157+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf819f470 con 0x7fddf8103840 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.157+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8018990 con 0x7fddf8103840 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.157+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf81a6d00 con 0x7fddf8103840 2024-03-28T18:54:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.158+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf8068490 con 0x7fddf8103840 2024-03-28T18:54:52.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.160+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fdde8010ac0 con 0x7fddf8103840 2024-03-28T18:54:52.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.161+0000 7fddefe00640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 0x7fdddc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.161+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fdde80cf2c0 con 0x7fddf8103840 2024-03-28T18:54:52.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.161+0000 7fddf6a00640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 0x7fdddc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.162+0000 7fddf6a00640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 0x7fdddc0a4e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009900 tx=0x7fdde402df90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.165+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdde809bf30 con 0x7fddf8103840 2024-03-28T18:54:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.363+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fddf802d070 con 0x7fddf8103840 2024-03-28T18:54:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.364+0000 7fddefe00640 1 -- 172.21.15.119:0/1935977713 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdde809b830 con 0x7fddf8103840 2024-03-28T18:54:52.365 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-03-28T18:54:52.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 msgr2=0x7fdddc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 0x7fdddc0a4e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009900 tx=0x7fdde402df90 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 msgr2=0x7fddf819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9f0 tx=0x7fdde800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 shutdown_connections 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fdddc0a2970 0x7fdddc0a4e30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf8109850 0x7fddf819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddf8104200 0x7fddf819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 --2- 172.21.15.119:0/1935977713 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fddf8103840 0x7fddf819a190 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 >> 172.21.15.119:0/1935977713 conn(0x7fddf80ff2f0 msgr2=0x7fddf80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 shutdown_connections 2024-03-28T18:54:52.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.367+0000 7fddfe800640 1 -- 172.21.15.119:0/1935977713 wait complete. 2024-03-28T18:54:52.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 -- 172.21.15.119:0/3486738303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101520 msgr2=0x7f99e01019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 --2- 172.21.15.119:0/3486738303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101520 0x7f99e01019a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009900 tx=0x7f99c802bc50 comp rx=0 tx=0).stop 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 -- 172.21.15.119:0/3486738303 shutdown_connections 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 --2- 172.21.15.119:0/3486738303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e0101ee0 0x7f99e010e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 --2- 172.21.15.119:0/3486738303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101520 0x7f99e01019a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 --2- 172.21.15.119:0/3486738303 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e01075e0 0x7f99e01079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 -- 172.21.15.119:0/3486738303 >> 172.21.15.119:0/3486738303 conn(0x7f99e00fd010 msgr2=0x7f99e00ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 -- 172.21.15.119:0/3486738303 shutdown_connections 2024-03-28T18:54:52.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.607+0000 7f99df200640 1 -- 172.21.15.119:0/3486738303 wait complete. 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 Processor -- start 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 -- start start 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 0x7f99e0073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 0x7f99e0074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e01118f0 con 0x7f99e0101ee0 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0111770 con 0x7f99e0101520 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.608+0000 7f99df200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0111a70 con 0x7f99e01075e0 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59386/0 (socket says 172.21.15.119:59386) 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 -- 172.21.15.119:0/3347585847 learned_addr learned my addr 172.21.15.119:0/3347585847 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 0x7f99e0073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99de800640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 0x7f99e0074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 -- 172.21.15.119:0/3347585847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 msgr2=0x7f99e0074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 0x7f99e0074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 -- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 msgr2=0x7f99e0073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 0x7f99e0073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 -- 172.21.15.119:0/3347585847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e01a61f0 con 0x7f99e0101520 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99de800640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 0x7f99e0074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:52.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99d7e00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 0x7f99e0073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.609+0000 7f99dde00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f99d000e9f0 tx=0x7f99d000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.610+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000ccc0 con 0x7f99e0101520 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.610+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e01a64e0 con 0x7f99e0101520 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.610+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99d0010910 con 0x7f99e0101520 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.610+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d00188a0 con 0x7f99e0101520 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.610+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e01a6a50 con 0x7f99e0101520 2024-03-28T18:54:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.611+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e00683b0 con 0x7f99e0101520 2024-03-28T18:54:52.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.613+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f99d0010ac0 con 0x7f99e0101520 2024-03-28T18:54:52.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.614+0000 7f99d6a00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 0x7f99c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:52.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.615+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f99d00cf230 con 0x7f99e0101520 2024-03-28T18:54:52.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.615+0000 7f99d7e00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 0x7f99c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:52.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.615+0000 7f99d7e00640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 0x7f99c40a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f99c8005ab0 tx=0x7f99c8005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:52.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.624+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f99d009bf00 con 0x7f99e0101520 2024-03-28T18:54:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:52 smithi119 ceph-mon[31813]: pgmap v335: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3445482865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:52.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:52 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1935977713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:52.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.822+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99e0109930 con 0x7f99e0101520 2024-03-28T18:54:52.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.822+0000 7f99d6a00640 1 -- 172.21.15.119:0/3347585847 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f99d009b800 con 0x7f99e0101520 2024-03-28T18:54:52.823 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 msgr2=0x7f99c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 0x7f99c40a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f99c8005ab0 tx=0x7f99c8005a20 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 msgr2=0x7f99e00739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f99d000e9f0 tx=0x7f99d000edf0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 shutdown_connections 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99c40a29a0 0x7f99c40a4e60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99e01075e0 0x7f99e0074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99e0101ee0 0x7f99e0073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 --2- 172.21.15.119:0/3347585847 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f99e0101520 0x7f99e00739a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 >> 172.21.15.119:0/3347585847 conn(0x7f99e00fd010 msgr2=0x7f99e00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 shutdown_connections 2024-03-28T18:54:52.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:52.825+0000 7f99df200640 1 -- 172.21.15.119:0/3347585847 wait complete. 2024-03-28T18:54:52.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T18:54:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:52 smithi196 ceph-mon[36300]: pgmap v335: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3445482865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:52.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:52 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1935977713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:52.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:52 smithi131 ceph-mon[36065]: pgmap v335: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:52.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3445482865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:52.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:52 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1935977713' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 -- 172.21.15.119:0/3326895636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 msgr2=0x7fe8e8109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3326895636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e8109b10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009930 tx=0x7fe8d802bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 -- 172.21.15.119:0/3326895636 shutdown_connections 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3326895636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 0x7fe8e81105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3326895636 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e8103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3326895636 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e8109b10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 -- 172.21.15.119:0/3326895636 >> 172.21.15.119:0/3326895636 conn(0x7fe8e80ff1d0 msgr2=0x7fe8e81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 -- 172.21.15.119:0/3326895636 shutdown_connections 2024-03-28T18:54:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.064+0000 7fe8ed600640 1 -- 172.21.15.119:0/3326895636 wait complete. 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.065+0000 7fe8ed600640 1 Processor -- start 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 -- start start 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 0x7fe8e819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8113b50 con 0x7fe8e81040e0 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e81139d0 con 0x7fe8e8103720 2024-03-28T18:54:53.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ed600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8113cd0 con 0x7fe8e8109730 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8e7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8e7e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59404/0 (socket says 172.21.15.119:59404) 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8e7e00640 1 -- 172.21.15.119:0/3516228399 learned_addr learned my addr 172.21.15.119:0/3516228399 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8ecc00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.066+0000 7fe8df400640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 0x7fe8e819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 -- 172.21.15.119:0/3516228399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 msgr2=0x7fe8e819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 -- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 msgr2=0x7fe8e819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 0x7fe8e819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 -- 172.21.15.119:0/3516228399 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e819f040 con 0x7fe8e8109730 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8e7e00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ecc00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e819e910 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009fd0 tx=0x7fe8e000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e000ce20 con 0x7fe8e8109730 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8e0005020 con 0x7fe8e8109730 2024-03-28T18:54:53.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e819f330 con 0x7fe8e8109730 2024-03-28T18:54:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0010720 con 0x7fe8e8109730 2024-03-28T18:54:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.067+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e81a6bc0 con 0x7fe8e8109730 2024-03-28T18:54:53.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.069+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e8068800 con 0x7fe8e8109730 2024-03-28T18:54:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.070+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7fe8e0005540 con 0x7fe8e8109730 2024-03-28T18:54:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.072+0000 7fe8e5600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 0x7fe8cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.072+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7fe8e00cf510 con 0x7fe8e8109730 2024-03-28T18:54:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.072+0000 7fe8e7e00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 0x7fe8cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.073+0000 7fe8e7e00640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 0x7fe8cc0a4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80096c0 tx=0x7fe8d8005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:53.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.080+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8e009c180 con 0x7fe8e8109730 2024-03-28T18:54:53.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.278+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8e810b700 con 0x7fe8e8109730 2024-03-28T18:54:53.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.280+0000 7fe8e5600640 1 -- 172.21.15.119:0/3516228399 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8e009ba80 con 0x7fe8e8109730 2024-03-28T18:54:53.280 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-03-28T18:54:53.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.281+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 msgr2=0x7fe8cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.281+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 0x7fe8cc0a4eb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe8d80096c0 tx=0x7fe8d8005a20 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.281+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 msgr2=0x7fe8e819e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.281+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e819e910 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009fd0 tx=0x7fe8e000ef10 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 shutdown_connections 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7fe8cc0a29f0 0x7fe8cc0a4eb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe8e8109730 0x7fe8e819e910 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8e81040e0 0x7fe8e819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 --2- 172.21.15.119:0/3516228399 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe8e8103720 0x7fe8e819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 >> 172.21.15.119:0/3516228399 conn(0x7fe8e80ff1d0 msgr2=0x7fe8e80ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 shutdown_connections 2024-03-28T18:54:53.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.282+0000 7fe8ed600640 1 -- 172.21.15.119:0/3516228399 wait complete. 2024-03-28T18:54:53.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 -- 172.21.15.119:0/2614207986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f109806a590 msgr2=0x7f109810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 --2- 172.21.15.119:0/2614207986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f109806a590 0x7f109810e520 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1090009860 tx=0x7f109002bb70 comp rx=0 tx=0).stop 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 -- 172.21.15.119:0/2614207986 shutdown_connections 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 --2- 172.21.15.119:0/2614207986 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f109806a590 0x7f109810e520 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 --2- 172.21.15.119:0/2614207986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098069c10 0x7f109806a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 --2- 172.21.15.119:0/2614207986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1098107660 0x7f1098107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 -- 172.21.15.119:0/2614207986 >> 172.21.15.119:0/2614207986 conn(0x7f1098077580 msgr2=0x7f1098077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:53.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 -- 172.21.15.119:0/2614207986 shutdown_connections 2024-03-28T18:54:53.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.513+0000 7f109ce00640 1 -- 172.21.15.119:0/2614207986 wait complete. 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 Processor -- start 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 -- start start 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109806a590 0x7f109819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 0x7f109819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1098111a20 con 0x7f109806a590 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10981118a0 con 0x7f1098069c10 2024-03-28T18:54:53.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f109ce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1098111ba0 con 0x7f1098107660 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f1095e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f1095e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59424/0 (socket says 172.21.15.119:59424) 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f1095e00640 1 -- 172.21.15.119:0/2699648215 learned_addr learned my addr 172.21.15.119:0/2699648215 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.515+0000 7f1096800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 0x7f109819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1095400640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109806a590 0x7f109819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 -- 172.21.15.119:0/2699648215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 msgr2=0x7f109819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 -- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109806a590 msgr2=0x7f109819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109806a590 0x7f109819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 -- 172.21.15.119:0/2699648215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109819f0d0 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1095e00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f1096800640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 0x7f109819e9a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f109002c0b0 tx=0x7f1090004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.516+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090041030 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.517+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f109002c510 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.517+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109819f360 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.517+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090039040 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.517+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10981a6c50 con 0x7f1098107660 2024-03-28T18:54:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.518+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1098102d90 con 0x7f1098107660 2024-03-28T18:54:53.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.520+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1090044070 con 0x7f1098107660 2024-03-28T18:54:53.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.521+0000 7f108ea00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 0x7f10780a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:53.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.522+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f10900f0110 con 0x7f1098107660 2024-03-28T18:54:53.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.522+0000 7f1095e00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 0x7f10780a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:53.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.523+0000 7f1095e00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 0x7f10780a4e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1088011440 tx=0x7f1088005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:53.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.530+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10900bce90 con 0x7f1098107660 2024-03-28T18:54:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.745+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f10980688e0 con 0x7f1098107660 2024-03-28T18:54:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.746+0000 7f108ea00640 1 -- 172.21.15.119:0/2699648215 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v99) v1 ==== 81+0+960 (secure 0 0 0) 0x7f10900bc790 con 0x7f1098107660 2024-03-28T18:54:53.747 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/393583864 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.747 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6801/2493078203 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.747 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6800/2493078203 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/1763050303 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/2790763787 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6800/3359592084 2024-03-29T18:33:28.825026+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/4209865453 2024-03-29T18:32:54.930081+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/2033406365 2024-03-29T18:33:28.825026+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6801/3103230972 2024-03-29T18:32:54.930081+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/1303314658 2024-03-29T18:32:54.930081+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/3984787959 2024-03-29T18:32:54.930081+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/748679560 2024-03-29T18:44:03.152098+0000 2024-03-28T18:54:53.748 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/2617254040 2024-03-29T18:33:28.825026+0000 2024-03-28T18:54:53.749 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6800/3103230972 2024-03-29T18:32:54.930081+0000 2024-03-28T18:54:53.749 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6801/3359592084 2024-03-29T18:33:28.825026+0000 2024-03-28T18:54:53.749 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:0/4220166218 2024-03-29T18:33:28.825026+0000 2024-03-28T18:54:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:listed 16 entries 2024-03-28T18:54:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.748+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 msgr2=0x7f10780a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.748+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 0x7f10780a4e30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1088011440 tx=0x7f1088005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.748+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 msgr2=0x7f109819e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.748+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 0x7f109819e9a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f109002c0b0 tx=0x7f1090004530 comp rx=0 tx=0).stop 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 shutdown_connections 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10780a2970 0x7f10780a4e30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1098107660 0x7f109819e9a0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109806a590 0x7f109819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 --2- 172.21.15.119:0/2699648215 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1098069c10 0x7f109819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 >> 172.21.15.119:0/2699648215 conn(0x7f1098077580 msgr2=0x7f109810cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 shutdown_connections 2024-03-28T18:54:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:53.749+0000 7f109ce00640 1 -- 172.21.15.119:0/2699648215 wait complete. 2024-03-28T18:54:53.770 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T18:54:53.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:54:53.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T18:54:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3347585847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:53 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3516228399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.882 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:54:53.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:54:53.883 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T18:54:53.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3347585847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:53 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3516228399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3347585847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:53 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3516228399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T18:54:53.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:54:53.959 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-03-28T18:54:53.996 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T18:54:53.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:54:53.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T18:54:54.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T18:54:54.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link delete ceph-brx 2024-03-28T18:54:54.241 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T18:54:54.252 INFO:tasks.cephadm:Teardown begin 2024-03-28T18:54:54.253 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T18:54:54.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:54:54.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 -- 172.21.15.119:0/966026571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1084101270 msgr2=0x7f1084105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 --2- 172.21.15.119:0/966026571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1084101270 0x7f1084105000 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1070009900 tx=0x7f107002bc50 comp rx=0 tx=0).stop 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 -- 172.21.15.119:0/966026571 shutdown_connections 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 --2- 172.21.15.119:0/966026571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10841056e0 0x7f108410d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 --2- 172.21.15.119:0/966026571 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1084101270 0x7f1084105000 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 --2- 172.21.15.119:0/966026571 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f1084100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 -- 172.21.15.119:0/966026571 >> 172.21.15.119:0/966026571 conn(0x7f1084075fb0 msgr2=0x7f10840763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 -- 172.21.15.119:0/966026571 shutdown_connections 2024-03-28T18:54:54.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.432+0000 7f108ac00640 1 -- 172.21.15.119:0/966026571 wait complete. 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.433+0000 7f108ac00640 1 Processor -- start 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.433+0000 7f108ac00640 1 -- start start 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f108410da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f108410e4c0 0x7f10841a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084111900 con 0x7f108410e4c0 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084111780 con 0x7f10841008c0 2024-03-28T18:54:54.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108ac00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1084111a80 con 0x7f10841056e0 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1089800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f108410da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37748/0 (socket says 172.21.15.119:37748) 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 -- 172.21.15.119:0/2352221121 learned_addr learned my addr 172.21.15.119:0/2352221121 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1089800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f108410da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59432/0 (socket says 172.21.15.119:59432) 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f108a200640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f108410e4c0 0x7f10841a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 -- 172.21.15.119:0/2352221121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 msgr2=0x7f108410da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f108410da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 -- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f108410e4c0 msgr2=0x7f10841a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f108410e4c0 0x7f10841a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.434+0000 7f1088e00640 1 -- 172.21.15.119:0/2352221121 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10841a66d0 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f1088e00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f107002c190 tx=0x7f10700042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070041070 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f107002ccb0 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10841a6960 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070003d90 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.435+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10841a6f00 con 0x7f10841056e0 2024-03-28T18:54:54.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.436+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10840683b0 con 0x7f10841056e0 2024-03-28T18:54:54.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.438+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f1070044070 con 0x7f10841056e0 2024-03-28T18:54:54.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.439+0000 7f1082000640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 0x7f10680a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.439+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f10700f06f0 con 0x7f10841056e0 2024-03-28T18:54:54.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.439+0000 7f1089800640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 0x7f10680a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.440+0000 7f1089800640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 0x7f10680a4ee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1074009bd0 tx=0x7f1074005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:54.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.443+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10700bce80 con 0x7f10841056e0 2024-03-28T18:54:54.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.684+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1084108970 con 0x7f10841056e0 2024-03-28T18:54:54.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.685+0000 7f1082000640 1 -- 172.21.15.119:0/2352221121 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f10700d4cf0 con 0x7f10841056e0 2024-03-28T18:54:54.686 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:54.686 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:54:54.686 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:54:54.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 msgr2=0x7f10680a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 0x7f10680a4ee0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1074009bd0 tx=0x7f1074005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:54.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 msgr2=0x7f108410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f107002c190 tx=0x7f10700042f0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 shutdown_connections 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f10680a2a20 0x7f10680a4ee0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f108410e4c0 0x7f10841a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841056e0 0x7f108410df80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 --2- 172.21.15.119:0/2352221121 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f10841008c0 0x7f108410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 >> 172.21.15.119:0/2352221121 conn(0x7f1084075fb0 msgr2=0x7f10840fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 shutdown_connections 2024-03-28T18:54:54.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.688+0000 7f108ac00640 1 -- 172.21.15.119:0/2352221121 wait complete. 2024-03-28T18:54:54.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T18:54:54.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:54 smithi131 ceph-mon[36065]: pgmap v336: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:54.846 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:54 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2699648215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:54:54.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 -- 172.21.15.119:0/3518547371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c104210 msgr2=0x7f6b9c110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/3518547371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c104210 0x7f6b9c110720 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009c10 tx=0x7f6b9402bef0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 -- 172.21.15.119:0/3518547371 shutdown_connections 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/3518547371 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c104210 0x7f6b9c110720 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/3518547371 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/3518547371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c109860 0x7f6b9c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.925+0000 7f6ba3a00640 1 -- 172.21.15.119:0/3518547371 >> 172.21.15.119:0/3518547371 conn(0x7f6b9c0ff2e0 msgr2=0x7f6b9c101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:54.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.926+0000 7f6ba3a00640 1 -- 172.21.15.119:0/3518547371 shutdown_connections 2024-03-28T18:54:54.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.926+0000 7f6ba3a00640 1 -- 172.21.15.119:0/3518547371 wait complete. 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.926+0000 7f6ba3a00640 1 Processor -- start 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 -- start start 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 0x7f6b9c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c109860 0x7f6b9c19e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c113c80 con 0x7f6b9c104210 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c113b00 con 0x7f6b9c103850 2024-03-28T18:54:54.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba3a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c113e00 con 0x7f6b9c109860 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba1000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba1000640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59454/0 (socket says 172.21.15.119:59454) 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba1000640 1 -- 172.21.15.119:0/1933853753 learned_addr learned my addr 172.21.15.119:0/1933853753 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6ba1a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c109860 0x7f6b9c19e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.927+0000 7f6b9be00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 0x7f6b9c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 -- 172.21.15.119:0/1933853753 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c109860 msgr2=0x7f6b9c19e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c109860 0x7f6b9c19e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 -- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 msgr2=0x7f6b9c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 0x7f6b9c19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:54.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 -- 172.21.15.119:0/1933853753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c19f110 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6b9be00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 0x7f6b9c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba1000640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000e9f0 tx=0x7f6b9000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000cd80 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c19f400 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b90010910 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90018990 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.928+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c1a6c90 con 0x7f6b9c103850 2024-03-28T18:54:54.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.929+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c068490 con 0x7f6b9c103850 2024-03-28T18:54:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.931+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6b90010ac0 con 0x7f6b9c103850 2024-03-28T18:54:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.932+0000 7f6b99600640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 0x7f6b840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.933+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f6b900cf2c0 con 0x7f6b9c103850 2024-03-28T18:54:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.933+0000 7f6b9be00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 0x7f6b840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:54.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.933+0000 7f6b9be00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 0x7f6b840a4e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c101910 tx=0x7f6b8c009260 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:54 smithi196 ceph-mon[36300]: pgmap v336: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:54.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:54 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2699648215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:54:54.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:54.936+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b9009bf30 con 0x7f6b9c103850 2024-03-28T18:54:55.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:54 smithi119 ceph-mon[31813]: pgmap v336: 97 pgs: 97 active+clean; 2.2 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T18:54:55.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:54 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2699648215' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T18:54:55.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.175+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b9c19b470 con 0x7f6b9c103850 2024-03-28T18:54:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.176+0000 7f6b99600640 1 -- 172.21.15.119:0/1933853753 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11475 (secure 0 0 0) 0x7f6b9009b830 con 0x7f6b9c103850 2024-03-28T18:54:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:55.177 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2486653544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2486653544},{"type":"v1","addr":"172.21.15.119:6835","nonce":2486653544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14400,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2841831538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2841831538},{"type":"v1","addr":"172.21.15.119:6837","nonce":2841831538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T18:41:20.510917+0000","modified":"2024-03-28T18:43:44.783845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24538,"mds_2":24517,"mds_3":24515,"mds_4":24505},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14448":{"gid":14448,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6839/1834295684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":1834295684},{"type":"v1","addr":"172.21.15.119:6839","nonce":1834295684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14460":{"gid":14460,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6841/683166305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6840","nonce":683166305},{"type":"v1","addr":"172.21.15.119:6841","nonce":683166305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6833/4160748351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":4160748351},{"type":"v1","addr":"172.21.15.131:6833","nonce":4160748351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/1244574561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1244574561},{"type":"v1","addr":"172.21.15.196:6833","nonce":1244574561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24485":{"gid":24485,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.131:6835/1538863538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":1538863538},{"type":"v1","addr":"172.21.15.131:6835","nonce":1538863538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24505":{"gid":24505,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.196:6835/480911404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":480911404},{"type":"v1","addr":"172.21.15.196:6835","nonce":480911404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":8,"addr":"172.21.15.131:6837/2241845025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2241845025},{"type":"v1","addr":"172.21.15.131:6837","nonce":2241845025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24517":{"gid":24517,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.196:6837/4226502296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":4226502296},{"type":"v1","addr":"172.21.15.196:6837","nonce":4226502296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24538":{"gid":24538,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2667663000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2667663000},{"type":"v1","addr":"172.21.15.196:6839","nonce":2667663000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.131:6839/2855046426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":2855046426},{"type":"v1","addr":"172.21.15.131:6839","nonce":2855046426}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24505,24515,24563,24538,24517]},"id":1}]} 2024-03-28T18:54:55.177 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 28 2024-03-28T18:54:55.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 msgr2=0x7f6b840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 0x7f6b840a4e30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c101910 tx=0x7f6b8c009260 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 msgr2=0x7f6b9c19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000e9f0 tx=0x7f6b9000eeb0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 shutdown_connections 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6b840a2970 0x7f6b840a4e30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b9c109860 0x7f6b9c19e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b9c104210 0x7f6b9c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 --2- 172.21.15.119:0/1933853753 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6b9c103850 0x7f6b9c19a120 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 >> 172.21.15.119:0/1933853753 conn(0x7f6b9c0ff2e0 msgr2=0x7f6b9c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 shutdown_connections 2024-03-28T18:54:55.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.179+0000 7f6ba3a00640 1 -- 172.21.15.119:0/1933853753 wait complete. 2024-03-28T18:54:55.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T18:54:55.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.412+0000 7f4da8200640 1 -- 172.21.15.119:0/2142060272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 msgr2=0x7f4da0109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.412+0000 7f4da8200640 1 --2- 172.21.15.119:0/2142060272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da0109c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4d94009930 tx=0x7f4d9402bc20 comp rx=0 tx=0).stop 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.412+0000 7f4da8200640 1 -- 172.21.15.119:0/2142060272 shutdown_connections 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 --2- 172.21.15.119:0/2142060272 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da0110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 --2- 172.21.15.119:0/2142060272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da0103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 --2- 172.21.15.119:0/2142060272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da0109c40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 -- 172.21.15.119:0/2142060272 >> 172.21.15.119:0/2142060272 conn(0x7f4da00ff320 msgr2=0x7f4da0101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 -- 172.21.15.119:0/2142060272 shutdown_connections 2024-03-28T18:54:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.413+0000 7f4da8200640 1 -- 172.21.15.119:0/2142060272 wait complete. 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.414+0000 7f4da8200640 1 Processor -- start 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.414+0000 7f4da8200640 1 -- start start 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.414+0000 7f4da8200640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.414+0000 7f4da8200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da8200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da8200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0113c80 con 0x7f4da0109860 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da8200640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0113b00 con 0x7f4da0103850 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da8200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0113e00 con 0x7f4da0104210 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59486/0 (socket says 172.21.15.119:59486) 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 -- 172.21.15.119:0/3643213008 learned_addr learned my addr 172.21.15.119:0/3643213008 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da4e00640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da6200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 -- 172.21.15.119:0/3643213008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 msgr2=0x7f4da019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 -- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 msgr2=0x7f4da019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da5800640 1 -- 172.21.15.119:0/3643213008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da019f1b0 con 0x7f4da0103850 2024-03-28T18:54:55.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.415+0000 7f4da4e00640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:55.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4da5800640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4d94005ab0 tx=0x7f4d94005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:55.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94043020 con 0x7f4da0103850 2024-03-28T18:54:55.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d94005000 con 0x7f4da0103850 2024-03-28T18:54:55.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da019f440 con 0x7f4da0103850 2024-03-28T18:54:55.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94003a80 con 0x7f4da0103850 2024-03-28T18:54:55.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.416+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da01a6d30 con 0x7f4da0103850 2024-03-28T18:54:55.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.417+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da00688e0 con 0x7f4da0103850 2024-03-28T18:54:55.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.419+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f4d94046020 con 0x7f4da0103850 2024-03-28T18:54:55.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.421+0000 7f4d9e000640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 0x7f4d880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.421+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f4d940f11e0 con 0x7f4da0103850 2024-03-28T18:54:55.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.421+0000 7f4da4e00640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 0x7f4d880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.421+0000 7f4da4e00640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 0x7f4d880a4e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4da0108f50 tx=0x7f4d90005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:55.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.427+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4d9404d020 con 0x7f4da0103850 2024-03-28T18:54:55.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.629+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4da010bdc0 con 0x7f4da0103850 2024-03-28T18:54:55.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.631+0000 7f4d9e000640 1 -- 172.21.15.119:0/3643213008 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+23434 (secure 0 0 0) 0x7f4d940bdf60 con 0x7f4da0103850 2024-03-28T18:54:55.632 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-28T18:54:55.633 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"5ff7da1e-ed31-11ee-95d0-87774f69a715","created":"2024-03-28T18:32:22.573529+0000","modified":"2024-03-28T18:54:09.747892+0000","last_up_change":"2024-03-28T18:40:22.643165+0000","last_in_change":"2024-03-28T18:40:03.171833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T18:36:13.518512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T18:41:18.605969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T18:41:20.021661+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f9791ee-9ecd-4e9e-b59e-90533047d529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6803","nonce":532063015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6805","nonce":532063015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6809","nonce":532063015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":532063015},{"type":"v1","addr":"172.21.15.119:6807","nonce":532063015}]},"public_addr":"172.21.15.119:6803/532063015","cluster_addr":"172.21.15.119:6805/532063015","heartbeat_back_addr":"172.21.15.119:6809/532063015","heartbeat_front_addr":"172.21.15.119:6807/532063015","state":["exists","up"]},{"osd":1,"uuid":"e56d8077-84c3-494e-8b08-9f5ea7b7cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6801","nonce":1240079334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6803","nonce":1240079334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6807","nonce":1240079334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1240079334},{"type":"v1","addr":"172.21.15.131:6805","nonce":1240079334}]},"public_addr":"172.21.15.131:6801/1240079334","cluster_addr":"172.21.15.131:6803/1240079334","heartbeat_back_addr":"172.21.15.131:6807/1240079334","heartbeat_front_addr":"172.21.15.131:6805/1240079334","state":["exists","up"]},{"osd":2,"uuid":"4c080cdf-65ee-41c9-be0f-18988515b0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6801","nonce":1413796404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6803","nonce":1413796404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6807","nonce":1413796404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":1413796404},{"type":"v1","addr":"172.21.15.196:6805","nonce":1413796404}]},"public_addr":"172.21.15.196:6801/1413796404","cluster_addr":"172.21.15.196:6803/1413796404","heartbeat_back_addr":"172.21.15.196:6807/1413796404","heartbeat_front_addr":"172.21.15.196:6805/1413796404","state":["exists","up"]},{"osd":3,"uuid":"15e6e92d-b038-4f17-8cb8-9a0c6db99e93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6811","nonce":837858906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6813","nonce":837858906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6817","nonce":837858906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":837858906},{"type":"v1","addr":"172.21.15.119:6815","nonce":837858906}]},"public_addr":"172.21.15.119:6811/837858906","cluster_addr":"172.21.15.119:6813/837858906","heartbeat_back_addr":"172.21.15.119:6817/837858906","heartbeat_front_addr":"172.21.15.119:6815/837858906","state":["exists","up"]},{"osd":4,"uuid":"ce922e09-9a15-418f-a01b-b70921dfbe71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6809","nonce":1834627175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6811","nonce":1834627175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6815","nonce":1834627175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1834627175},{"type":"v1","addr":"172.21.15.131:6813","nonce":1834627175}]},"public_addr":"172.21.15.131:6809/1834627175","cluster_addr":"172.21.15.131:6811/1834627175","heartbeat_back_addr":"172.21.15.131:6815/1834627175","heartbeat_front_addr":"172.21.15.131:6813/1834627175","state":["exists","up"]},{"osd":5,"uuid":"6b57956f-6e7d-4bc7-8b89-5390058b7ba8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6809","nonce":3397729850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6811","nonce":3397729850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6815","nonce":3397729850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":3397729850},{"type":"v1","addr":"172.21.15.196:6813","nonce":3397729850}]},"public_addr":"172.21.15.196:6809/3397729850","cluster_addr":"172.21.15.196:6811/3397729850","heartbeat_back_addr":"172.21.15.196:6815/3397729850","heartbeat_front_addr":"172.21.15.196:6813/3397729850","state":["exists","up"]},{"osd":6,"uuid":"7b527cff-096d-4e00-bfda-8e3e77bc70e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6819","nonce":2797182795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6821","nonce":2797182795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6825","nonce":2797182795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2797182795},{"type":"v1","addr":"172.21.15.119:6823","nonce":2797182795}]},"public_addr":"172.21.15.119:6819/2797182795","cluster_addr":"172.21.15.119:6821/2797182795","heartbeat_back_addr":"172.21.15.119:6825/2797182795","heartbeat_front_addr":"172.21.15.119:6823/2797182795","state":["exists","up"]},{"osd":7,"uuid":"044ba726-17cb-4d14-9040-5efd8e892821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6817","nonce":417830389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6819","nonce":417830389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6823","nonce":417830389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":417830389},{"type":"v1","addr":"172.21.15.131:6821","nonce":417830389}]},"public_addr":"172.21.15.131:6817/417830389","cluster_addr":"172.21.15.131:6819/417830389","heartbeat_back_addr":"172.21.15.131:6823/417830389","heartbeat_front_addr":"172.21.15.131:6821/417830389","state":["exists","up"]},{"osd":8,"uuid":"4c9fcc00-0b32-4fc0-9ade-3f152779462e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6817","nonce":1701924552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6819","nonce":1701924552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6823","nonce":1701924552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":1701924552},{"type":"v1","addr":"172.21.15.196:6821","nonce":1701924552}]},"public_addr":"172.21.15.196:6817/1701924552","cluster_addr":"172.21.15.196:6819/1701924552","heartbeat_back_addr":"172.21.15.196:6823/1701924552","heartbeat_front_addr":"172.21.15.196:6821/1701924552","state":["exists","up"]},{"osd":9,"uuid":"db64ebe3-7820-4944-9d3d-9efdad1576bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6827","nonce":1454392710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6829","nonce":1454392710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6833","nonce":1454392710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1454392710},{"type":"v1","addr":"172.21.15.119:6831","nonce":1454392710}]},"public_addr":"172.21.15.119:6827/1454392710","cluster_addr":"172.21.15.119:6829/1454392710","heartbeat_back_addr":"172.21.15.119:6833/1454392710","heartbeat_front_addr":"172.21.15.119:6831/1454392710","state":["exists","up"]},{"osd":10,"uuid":"46b91f6b-8ffe-4d4e-9c36-d3faafc12d12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6825","nonce":2982886262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6827","nonce":2982886262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6831","nonce":2982886262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":2982886262},{"type":"v1","addr":"172.21.15.131:6829","nonce":2982886262}]},"public_addr":"172.21.15.131:6825/2982886262","cluster_addr":"172.21.15.131:6827/2982886262","heartbeat_back_addr":"172.21.15.131:6831/2982886262","heartbeat_front_addr":"172.21.15.131:6829/2982886262","state":["exists","up"]},{"osd":11,"uuid":"5eb4209c-3199-4138-b104-8f07d626633d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6825","nonce":2114329937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6827","nonce":2114329937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6831","nonce":2114329937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2114329937},{"type":"v1","addr":"172.21.15.196:6829","nonce":2114329937}]},"public_addr":"172.21.15.196:6825/2114329937","cluster_addr":"172.21.15.196:6827/2114329937","heartbeat_back_addr":"172.21.15.196:6831/2114329937","heartbeat_front_addr":"172.21.15.196:6829/2114329937","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:21.294961+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:35:46.019759+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:09.553393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:36:41.538284+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:07.217606+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:37:31.299602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:04.218401+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:30.704122+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:38:55.828920+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:28.621865+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:39:55.589386+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T18:40:21.055597+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1c","mappings":[{"from":5,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/393583864":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6801/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/2493078203":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/1763050303":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2790763787":"2024-03-29T18:44:03.152098+0000","172.21.15.119:6800/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4209865453":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/2033406365":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6801/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/1303314658":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/3984787959":"2024-03-29T18:32:54.930081+0000","172.21.15.119:0/748679560":"2024-03-29T18:44:03.152098+0000","172.21.15.119:0/2617254040":"2024-03-29T18:33:28.825026+0000","172.21.15.119:6800/3103230972":"2024-03-29T18:32:54.930081+0000","172.21.15.119:6801/3359592084":"2024-03-29T18:33:28.825026+0000","172.21.15.119:0/4220166218":"2024-03-29T18:33:28.825026+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T18:54:55.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 msgr2=0x7f4d880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 0x7f4d880a4e30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4da0108f50 tx=0x7f4d90005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 msgr2=0x7f4da019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4d94005ab0 tx=0x7f4d94005e80 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 shutdown_connections 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f4d880a2970 0x7f4d880a4e30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4da0109860 0x7f4da019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4da0104210 0x7f4da019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 --2- 172.21.15.119:0/3643213008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4da0103850 0x7f4da019a1c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 >> 172.21.15.119:0/3643213008 conn(0x7f4da00ff320 msgr2=0x7f4da00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.634+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 shutdown_connections 2024-03-28T18:54:55.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.635+0000 7f4da8200640 1 -- 172.21.15.119:0/3643213008 wait complete. 2024-03-28T18:54:55.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 -- 172.21.15.119:0/2811892641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f000ffc10 msgr2=0x7f7f00100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 --2- 172.21.15.119:0/2811892641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f000ffc10 0x7f7f00100070 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009930 tx=0x7f7eec02bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 -- 172.21.15.119:0/2811892641 shutdown_connections 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 --2- 172.21.15.119:0/2811892641 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001005b0 0x7f7f0010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 --2- 172.21.15.119:0/2811892641 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f000ffc10 0x7f7f00100070 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 --2- 172.21.15.119:0/2811892641 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f001076d0 0x7f7f00107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.870+0000 7f7f06e00640 1 -- 172.21.15.119:0/2811892641 >> 172.21.15.119:0/2811892641 conn(0x7f7f00076bd0 msgr2=0x7f7f000fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.871+0000 7f7f06e00640 1 -- 172.21.15.119:0/2811892641 shutdown_connections 2024-03-28T18:54:55.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.871+0000 7f7f06e00640 1 -- 172.21.15.119:0/2811892641 wait complete. 2024-03-28T18:54:55.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 Processor -- start 2024-03-28T18:54:55.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 -- start start 2024-03-28T18:54:55.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f000ffc10 0x7f7f00100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 0x7f7f00101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f00111b60 con 0x7f7f001076d0 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f001119e0 con 0x7f7f000ffc10 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.872+0000 7f7f06e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f00111ce0 con 0x7f7f001005b0 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.119:37804/0 (socket says 172.21.15.119:37804) 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 -- 172.21.15.119:0/3011144432 learned_addr learned my addr 172.21.15.119:0/3011144432 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7effe00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f000ffc10 0x7f7f00100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7f04e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 0x7f7f00101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 -- 172.21.15.119:0/3011144432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f000ffc10 msgr2=0x7f7f00100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f000ffc10 0x7f7f00100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 -- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 msgr2=0x7f7f00101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 0x7f7f00101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 -- 172.21.15.119:0/3011144432 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f001a6540 con 0x7f7f001005b0 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7f04e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 0x7f7f00101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.873+0000 7f7eff400640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009900 tx=0x7f7eec004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.874+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec043020 con 0x7f7f001005b0 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.874+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f001a67d0 con 0x7f7f001005b0 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.874+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7eec0054e0 con 0x7f7f001005b0 2024-03-28T18:54:55.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.874+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eec003d00 con 0x7f7f001005b0 2024-03-28T18:54:55.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.874+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f001a6d20 con 0x7f7f001005b0 2024-03-28T18:54:55.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.875+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f00102320 con 0x7f7f001005b0 2024-03-28T18:54:55.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.877+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f7eec046020 con 0x7f7f001005b0 2024-03-28T18:54:55.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.878+0000 7f7efcc00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 0x7f7ee40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:55.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.879+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8666+0+0 (secure 0 0 0) 0x7f7eec0f1540 con 0x7f7f001005b0 2024-03-28T18:54:55.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.879+0000 7f7effe00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 0x7f7ee40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:55.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.879+0000 7f7effe00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 0x7f7ee40a4e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000bbe0 tx=0x7f7ef000b390 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:55.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:55.883+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7eec04d020 con 0x7f7f001005b0 2024-03-28T18:54:55.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:55 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/2352221121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:55.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:55 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1933853753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:55 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/2352221121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:55.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:55 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1933853753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:55 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/2352221121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:55 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1933853753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.128+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7f00109c50 con 0x7f7f001005b0 2024-03-28T18:54:56.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.137+0000 7f7efcc00640 1 -- 172.21.15.119:0/3011144432 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7eec0be2c0 con 0x7f7f001005b0 2024-03-28T18:54:56.137 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T18:54:56.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 msgr2=0x7f7ee40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 0x7f7ee40a4e30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000bbe0 tx=0x7f7ef000b390 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 msgr2=0x7f7f00101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009900 tx=0x7f7eec004430 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 shutdown_connections 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f7ee40a2970 0x7f7ee40a4e30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f001076d0 0x7f7f00101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f001005b0 0x7f7f00101330 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 --2- 172.21.15.119:0/3011144432 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7f000ffc10 0x7f7f00100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.140+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 >> 172.21.15.119:0/3011144432 conn(0x7f7f00076bd0 msgr2=0x7f7f000fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:56.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.141+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 shutdown_connections 2024-03-28T18:54:56.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.141+0000 7f7f06e00640 1 -- 172.21.15.119:0/3011144432 wait complete. 2024-03-28T18:54:56.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T18:54:56.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 -- 172.21.15.119:0/3020146588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 msgr2=0x7f9940073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:56.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 --2- 172.21.15.119:0/3020146588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f9940073e20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9930009ca0 tx=0x7f993002c190 comp rx=0 tx=0).stop 2024-03-28T18:54:56.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 -- 172.21.15.119:0/3020146588 shutdown_connections 2024-03-28T18:54:56.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 --2- 172.21.15.119:0/3020146588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 0x7f9940114900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 --2- 172.21.15.119:0/3020146588 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f9940111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 --2- 172.21.15.119:0/3020146588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f9940073e20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 -- 172.21.15.119:0/3020146588 >> 172.21.15.119:0/3020146588 conn(0x7f99400fd010 msgr2=0x7f99400ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 -- 172.21.15.119:0/3020146588 shutdown_connections 2024-03-28T18:54:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.347+0000 7f993fe00640 1 -- 172.21.15.119:0/3020146588 wait complete. 2024-03-28T18:54:56.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.348+0000 7f993fe00640 1 Processor -- start 2024-03-28T18:54:56.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.348+0000 7f993fe00640 1 -- start start 2024-03-28T18:54:56.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.348+0000 7f993fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:56.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.348+0000 7f993fe00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f994019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:56.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 0x7f99401a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9940117f10 con 0x7f9940073a40 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993fe00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9940117d90 con 0x7f9940074360 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9940118090 con 0x7f9940112510 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39730/0 (socket says 172.21.15.119:39730) 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993ea00640 1 -- 172.21.15.119:0/1439344709 learned_addr learned my addr 172.21.15.119:0/1439344709 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f994019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993f400640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 0x7f99401a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 -- 172.21.15.119:0/1439344709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 msgr2=0x7f99401a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:56.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 0x7f99401a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 -- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 msgr2=0x7f994019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f9936000640 1 -- 172.21.15.119:0/1439344709 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99401a33f0 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.349+0000 7f993ea00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f9936000640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f994019e990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f992800e9f0 tx=0x7f992800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992800ccc0 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99401a36e0 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9928010910 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928018960 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.350+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99401aaf70 con 0x7f9940074360 2024-03-28T18:54:56.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.351+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99400683b0 con 0x7f9940074360 2024-03-28T18:54:56.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.353+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f9928010ac0 con 0x7f9940074360 2024-03-28T18:54:56.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.354+0000 7f9937e00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 0x7f99240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:56.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.354+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9052+0+0 (secure 0 0 0) 0x7f99280cf490 con 0x7f9940074360 2024-03-28T18:54:56.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.354+0000 7f993ea00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 0x7f99240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:56.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.355+0000 7f993ea00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 0x7f99240a4e30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9930009a60 tx=0x7f9930005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:56.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.358+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f992809b930 con 0x7f9940074360 2024-03-28T18:54:56.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.435 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mds[66645]: -- 172.21.15.196:0/1156710237 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559aad11ba40 con 0x559aad0eec00 2024-03-28T18:54:56.436 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: starting mds.c at 2024-03-28T18:54:56.436 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.436 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.436 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mds[66334]: -- 172.21.15.196:0/1433485250 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e9c309fa40 con 0x55e9c3073000 2024-03-28T18:54:56.436 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: starting mds.f at 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mds[66958]: -- 172.21.15.196:0/4211111087 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564074fb3a40 con 0x564074f86800 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: starting mds.i at 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l[66019]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.437 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l[66019]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.438 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mds[66023]: -- 172.21.15.196:0/1285911576 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5613529e7a40 con 0x5613529bb000 2024-03-28T18:54:56.438 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l[66019]: starting mds.l at 2024-03-28T18:54:56.444 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mds[65405]: -- 172.21.15.131:0/2676031754 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e764e59a40 con 0x55e764e2cc00 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: starting mds.e at 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mds[65719]: -- 172.21.15.131:0/1566035082 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55935a3b1a40 con 0x55935a385000 2024-03-28T18:54:56.445 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: starting mds.h at 2024-03-28T18:54:56.446 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.446 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.446 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mds[65083]: -- 172.21.15.131:0/941991225 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5599ab8bfa40 con 0x5599ab892c00 2024-03-28T18:54:56.446 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: starting mds.k at 2024-03-28T18:54:56.447 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.447 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.447 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mds[66023]: -- 172.21.15.131:0/2207859828 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a9d4ddda40 con 0x55a9d4db0800 2024-03-28T18:54:56.447 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: starting mds.b at 2024-03-28T18:54:56.523 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: ignoring --setuser ceph since I am not root 2024-03-28T18:54:56.523 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: ignoring --setgroup ceph since I am not root 2024-03-28T18:54:56.523 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mds[74620]: -- 172.21.15.119:0/1529869734 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b4650cda40 con 0x55b4650a1000 2024-03-28T18:54:56.523 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: starting mds.a at 2024-03-28T18:54:56.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a[31809]: 2024-03-28T18:54:56.129+0000 7f2880600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:54:56.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:56.603+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f99401049e0 con 0x7f9940074360 2024-03-28T18:54:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: pgmap v337: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3643213008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/3011144432' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:54:56.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:56 smithi119 ceph-mon[31813]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:54:56.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: pgmap v337: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3643213008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/3011144432' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:54:56.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:56 smithi196 ceph-mon[36300]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:54:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: pgmap v337: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T18:54:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3643213008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T18:54:56.944 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/3011144432' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T18:54:56.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:56 smithi131 ceph-mon[36065]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T18:54:57.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.138+0000 7f9937e00640 1 -- 172.21.15.119:0/1439344709 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7f992809f9d0 con 0x7f9940074360 2024-03-28T18:54:57.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.140+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 msgr2=0x7f99240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.140+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 0x7f99240a4e30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9930009a60 tx=0x7f9930005d80 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.140+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 msgr2=0x7f994019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.140+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f994019e990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f992800e9f0 tx=0x7f992800eeb0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 shutdown_connections 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f99240a2970 0x7f99240a4e30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9940112510 0x7f99401a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f9940074360 0x7f994019e990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 --2- 172.21.15.119:0/1439344709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9940073a40 0x7f994019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 >> 172.21.15.119:0/1439344709 conn(0x7f99400fd010 msgr2=0x7f99400fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 shutdown_connections 2024-03-28T18:54:57.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.141+0000 7f993fe00640 1 -- 172.21.15.119:0/1439344709 wait complete. 2024-03-28T18:54:57.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T18:54:57.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 -- 172.21.15.119:0/2250677321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 msgr2=0x7f380810d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:57.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 --2- 172.21.15.119:0/2250677321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380810d940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f37f4009900 tx=0x7f37f402bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:57.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 -- 172.21.15.119:0/2250677321 shutdown_connections 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 --2- 172.21.15.119:0/2250677321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f380810de80 0x7f3808110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 --2- 172.21.15.119:0/2250677321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380810d940 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 --2- 172.21.15.119:0/2250677321 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38081002d0 0x7f38081006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.350+0000 7f380cc00640 1 -- 172.21.15.119:0/2250677321 >> 172.21.15.119:0/2250677321 conn(0x7f38080fbef0 msgr2=0x7f38080fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.351+0000 7f380cc00640 1 -- 172.21.15.119:0/2250677321 shutdown_connections 2024-03-28T18:54:57.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.351+0000 7f380cc00640 1 -- 172.21.15.119:0/2250677321 wait complete. 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.351+0000 7f380cc00640 1 Processor -- start 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 -- start start 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38081002d0 0x7f380819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 0x7f380819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:57.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808113b70 con 0x7f3808100c80 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38081139f0 con 0x7f380810de80 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f380cc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808113cf0 con 0x7f38081002d0 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f3806a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f3806a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39748/0 (socket says 172.21.15.119:39748) 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f3806a00640 1 -- 172.21.15.119:0/1578842870 learned_addr learned my addr 172.21.15.119:0/1578842870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f3807e00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 0x7f380819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.352+0000 7f3807400640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38081002d0 0x7f380819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 -- 172.21.15.119:0/1578842870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38081002d0 msgr2=0x7f380819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38081002d0 0x7f380819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 -- 172.21.15.119:0/1578842870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 msgr2=0x7f380819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 0x7f380819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 -- 172.21.15.119:0/1578842870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380819f0a0 con 0x7f3808100c80 2024-03-28T18:54:57.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3807e00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 0x7f380819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f3806a00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37f402c240 tx=0x7f37f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4043020 con 0x7f3808100c80 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37f40054e0 con 0x7f3808100c80 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380819f330 con 0x7f3808100c80 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.353+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4003b20 con 0x7f3808100c80 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.354+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38081a6c20 con 0x7f3808100c80 2024-03-28T18:54:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.354+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38080683b0 con 0x7f3808100c80 2024-03-28T18:54:57.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.357+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f37f4046020 con 0x7f3808100c80 2024-03-28T18:54:57.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.359+0000 7f37ffe00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 0x7f37ec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:57.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.360+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9052+0+0 (secure 0 0 0) 0x7f37f40f1220 con 0x7f3808100c80 2024-03-28T18:54:57.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.360+0000 7f3807400640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 0x7f37ec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:57.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.360+0000 7f3807400640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 0x7f37ec0a4eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37f80095f0 tx=0x7f37f8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:57.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.361+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f37f404d020 con 0x7f3808100c80 2024-03-28T18:54:57.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:57.549+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f380819b400 con 0x7f3808100c80 2024-03-28T18:54:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1439344709' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:54:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:54:57.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: Cluster is now healthy 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6835/3059940073,v1:172.21.15.196:6838/3059940073] up:boot 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6836/3661986312,v1:172.21.15.196:6839/3661986312] up:boot 2024-03-28T18:54:57.936 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6833/3036752397,v1:172.21.15.131:6835/3036752397] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6836/1057731329,v1:172.21.15.131:6837/1057731329] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6832/4118628661,v1:172.21.15.196:6834/4118628661] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.196:6833/3299934270,v1:172.21.15.196:6837/3299934270] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6832/2998043170,v1:172.21.15.131:6834/2998043170] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.131:6838/828396338,v1:172.21.15.131:6839/828396338] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6838/2528633418,v1:172.21.15.119:6840/2528633418] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: mds.? [v2:172.21.15.119:6839/3689668328,v1:172.21.15.119:6841/3689668328] up:boot 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: fsmap 12 up:standby 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:54:57.937 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:54:57.938 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:54:57.938 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:54:57.938 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: pgmap v339: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-28T18:54:57.938 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:57 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1439344709' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: Cluster is now healthy 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:54:57.945 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6835/3059940073,v1:172.21.15.196:6838/3059940073] up:boot 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6836/3661986312,v1:172.21.15.196:6839/3661986312] up:boot 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6833/3036752397,v1:172.21.15.131:6835/3036752397] up:boot 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6836/1057731329,v1:172.21.15.131:6837/1057731329] up:boot 2024-03-28T18:54:57.946 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6832/4118628661,v1:172.21.15.196:6834/4118628661] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.196:6833/3299934270,v1:172.21.15.196:6837/3299934270] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6832/2998043170,v1:172.21.15.131:6834/2998043170] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.131:6838/828396338,v1:172.21.15.131:6839/828396338] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6838/2528633418,v1:172.21.15.119:6840/2528633418] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: mds.? [v2:172.21.15.119:6839/3689668328,v1:172.21.15.119:6841/3689668328] up:boot 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: fsmap 12 up:standby 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:54:57.947 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: pgmap v339: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-28T18:54:57.948 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:57 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1439344709' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T18:54:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T18:54:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T18:54:58.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: Cluster is now healthy 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T18:54:58.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6835/3059940073,v1:172.21.15.196:6838/3059940073] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6836/3661986312,v1:172.21.15.196:6839/3661986312] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6833/3036752397,v1:172.21.15.131:6835/3036752397] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6836/1057731329,v1:172.21.15.131:6837/1057731329] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6832/4118628661,v1:172.21.15.196:6834/4118628661] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.196:6833/3299934270,v1:172.21.15.196:6837/3299934270] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6832/2998043170,v1:172.21.15.131:6834/2998043170] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.131:6838/828396338,v1:172.21.15.131:6839/828396338] up:boot 2024-03-28T18:54:58.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6838/2528633418,v1:172.21.15.119:6840/2528633418] up:boot 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: mds.? [v2:172.21.15.119:6839/3689668328,v1:172.21.15.119:6841/3689668328] up:boot 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: fsmap 12 up:standby 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='mgr.25418 172.21.15.119:0/252411410' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: pgmap v339: 97 pgs: 97 active+clean; 2.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 11 op/s 2024-03-28T18:54:58.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:57 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:58.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.137+0000 7f37ffe00640 1 -- 172.21.15.119:0/1578842870 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f380819b400 con 0x7f3808100c80 2024-03-28T18:54:58.137 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 msgr2=0x7f37ec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 0x7f37ec0a4eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f37f80095f0 tx=0x7f37f8005f90 comp rx=0 tx=0).stop 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 msgr2=0x7f380819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f37f402c240 tx=0x7f37f40042f0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 shutdown_connections 2024-03-28T18:54:58.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f37ec0a29f0 0x7f37ec0a4eb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f380810de80 0x7f380819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3808100c80 0x7f380819a5f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 --2- 172.21.15.119:0/1578842870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38081002d0 0x7f380819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.141+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 >> 172.21.15.119:0/1578842870 conn(0x7f38080fbef0 msgr2=0x7f380810f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:58.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.142+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 shutdown_connections 2024-03-28T18:54:58.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.142+0000 7f380cc00640 1 -- 172.21.15.119:0/1578842870 wait complete. 2024-03-28T18:54:58.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.364+0000 7f6de3e00640 1 -- 172.21.15.119:0/667808033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4108800 msgr2=0x7f6de410c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.364+0000 7f6de3e00640 1 --2- 172.21.15.119:0/667808033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4108800 0x7f6de410c7c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009900 tx=0x7f6dd002bc30 comp rx=0 tx=0).stop 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 -- 172.21.15.119:0/667808033 shutdown_connections 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 --2- 172.21.15.119:0/667808033 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de410cea0 0x7f6de4114a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 --2- 172.21.15.119:0/667808033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4108800 0x7f6de410c7c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 --2- 172.21.15.119:0/667808033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de4107e50 0x7f6de4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 -- 172.21.15.119:0/667808033 >> 172.21.15.119:0/667808033 conn(0x7f6de40774c0 msgr2=0x7f6de41064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:58.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 -- 172.21.15.119:0/667808033 shutdown_connections 2024-03-28T18:54:58.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.365+0000 7f6de3e00640 1 -- 172.21.15.119:0/667808033 wait complete. 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.366+0000 7f6de3e00640 1 Processor -- start 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.366+0000 7f6de3e00640 1 -- start start 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.366+0000 7f6de3e00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de410cea0 0x7f6de419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de41a2d50 0x7f6de41a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4117f50 con 0x7f6de410cea0 2024-03-28T18:54:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3e00640 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4117dd0 con 0x7f6de4107e50 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de41180d0 con 0x7f6de41a2d50 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de410cea0 0x7f6de419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.119:59586/0 (socket says 172.21.15.119:59586) 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 -- 172.21.15.119:0/775073821 learned_addr learned my addr 172.21.15.119:0/775073821 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de3400640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de41a2d50 0x7f6de41a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 -- 172.21.15.119:0/775073821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de41a2d50 msgr2=0x7f6de41a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:58.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de41a2d50 0x7f6de41a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 -- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de410cea0 msgr2=0x7f6de419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de410cea0 0x7f6de419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.367+0000 7f6de2a00640 1 -- 172.21.15.119:0/775073821 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de41a3840 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de2a00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400e9f0 tx=0x7f6dd400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd400cd80 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd4005500 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4010690 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de41a3b30 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.368+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de41ab2a0 con 0x7f6de4107e50 2024-03-28T18:54:58.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.369+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4068800 con 0x7f6de4107e50 2024-03-28T18:54:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.371+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 27) v1 ==== 137422+0+0 (secure 0 0 0) 0x7f6dd40046c0 con 0x7f6de4107e50 2024-03-28T18:54:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.373+0000 7f6de8a00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 0x7f6dc80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T18:54:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.373+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8432+0+0 (secure 0 0 0) 0x7f6dd40cf1d0 con 0x7f6de4107e50 2024-03-28T18:54:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.373+0000 7f6de2000640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 0x7f6dc80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T18:54:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.374+0000 7f6de2000640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 0x7f6dc80a4eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0000c00 tx=0x7f6dd0005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T18:54:58.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.377+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6dd409bf50 con 0x7f6de4107e50 2024-03-28T18:54:58.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:58.562+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6de419f7e0 con 0x7f6de4107e50 2024-03-28T18:54:59.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.159+0000 7f6de8a00640 1 -- 172.21.15.119:0/775073821 <== mon.2 v2:172.21.15.131:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6dd409b850 con 0x7f6de4107e50 2024-03-28T18:54:59.160 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-03-28T18:54:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 msgr2=0x7f6dc80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:59.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 0x7f6dc80a4eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0000c00 tx=0x7f6dd0005fd0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 msgr2=0x7f6de419e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400e9f0 tx=0x7f6dd400edf0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 shutdown_connections 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:6800/212877697,v1:172.21.15.119:6801/212877697] conn(0x7f6dc80a29f0 0x7f6dc80a4eb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6de41a2d50 0x7f6de41a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de410cea0 0x7f6de419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 --2- 172.21.15.119:0/775073821 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6de4107e50 0x7f6de419e490 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T18:54:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.164+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 >> 172.21.15.119:0/775073821 conn(0x7f6de40774c0 msgr2=0x7f6de410e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T18:54:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.165+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 shutdown_connections 2024-03-28T18:54:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-28T18:54:59.165+0000 7f6de3e00640 1 -- 172.21.15.119:0/775073821 wait complete. 2024-03-28T18:54:59.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:54:59.278 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:54:59.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:54:59.349 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T18:54:59.350 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:54:59.407 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:54:59.424 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T18:54:59.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:54:59.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:59 smithi196 ceph-mon[36300]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:54:59.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:59 smithi196 ceph-mon[36300]: from='client.? 172.21.15.119:0/775073821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:54:59 smithi196 ceph-mon[36300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.443 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T18:54:59.443 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T18:54:59.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a 2024-03-28T18:54:59.444 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:54:59.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:59 smithi131 ceph-mon[36065]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:54:59.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:59 smithi131 ceph-mon[36065]: from='client.? 172.21.15.119:0/775073821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.445 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:54:59 smithi131 ceph-mon[36065]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/1578842870' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T18:54:59.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:59 smithi119 ceph-mon[31813]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T18:54:59.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:59 smithi119 ceph-mon[31813]: from='client.? 172.21.15.119:0/775073821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:59 smithi119 ceph-mon[31813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T18:54:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:54:59 smithi119 systemd[1]: Stopping Ceph mon.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:00.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a[31809]: 2024-03-28T18:55:00.068+0000 7f2887400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:00.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a[31809]: 2024-03-28T18:55:00.068+0000 7f2887400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T18:55:00.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 podman[103108]: 2024-03-28 18:55:00.27490796 +0000 UTC m=+0.378838013 container died 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:55:00.727 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:55:00.409+0000 7f09ce000700 -1 client.25468: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-28T18:55:00.727 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:55:00.412+0000 7f09ce000700 -1 client.25468: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-28T18:55:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 podman[103108]: 2024-03-28 18:55:00.727519663 +0000 UTC m=+0.831449709 container cleanup 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T18:55:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 bash[103108]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a 2024-03-28T18:55:01.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Mar 28 18:55:00 smithi119 podman[103122]: 2024-03-28 18:55:00.920462546 +0000 UTC m=+0.629789004 container remove 38d4767e084a19595bbc73823e230dafb2e2c1e3acd92c05fc0b65881d81496c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T18:55:01.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T18:55:01.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:01.539 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T18:55:01.539 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T18:55:01.539 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.b 2024-03-28T18:55:01.863 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:01 smithi131 systemd[1]: Stopping Ceph mon.b for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:02.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-b[36061]: 2024-03-28T18:55:01.862+0000 7f69ccd61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:02.194 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-b[36061]: 2024-03-28T18:55:01.862+0000 7f69ccd61700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T18:55:02.195 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:02 smithi131 podman[71989]: 2024-03-28 18:55:02.047485827 +0000 UTC m=+0.312163665 container died 1527ac86c6d646463c92b408bcecbe6f651eaa5d4fe98faca35405fe6789b27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T18:55:02.694 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:02 smithi131 podman[71989]: 2024-03-28 18:55:02.43077923 +0000 UTC m=+0.695457050 container cleanup 1527ac86c6d646463c92b408bcecbe6f651eaa5d4fe98faca35405fe6789b27a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:55:02.695 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 28 18:55:02 smithi131 bash[71989]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-b 2024-03-28T18:55:02.852 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T18:55:02.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:02.912 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T18:55:02.912 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T18:55:02.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.c 2024-03-28T18:55:03.280 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:02 smithi196 systemd[1]: Stopping Ceph mon.c for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:03.531 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-c[36294]: 2024-03-28T18:55:03.280+0000 7fe4666ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:03.531 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-c[36294]: 2024-03-28T18:55:03.280+0000 7fe4666ea700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T18:55:03.531 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:03 smithi196 podman[72834]: 2024-03-28 18:55:03.311871484 +0000 UTC m=+0.197157999 container died e852c2239cf9d4567d409dc32fa7a21b99f7d9c73972872db4f15cf8b7b4391a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:55:03.831 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:03 smithi196 podman[72834]: 2024-03-28 18:55:03.6144533 +0000 UTC m=+0.499739806 container cleanup e852c2239cf9d4567d409dc32fa7a21b99f7d9c73972872db4f15cf8b7b4391a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T18:55:03.831 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 28 18:55:03 smithi196 bash[72834]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mon-c 2024-03-28T18:55:03.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T18:55:04.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:04.003 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T18:55:04.003 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T18:55:04.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x 2024-03-28T18:55:04.568 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:04 smithi119 systemd[1]: Stopping Ceph mgr.x for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:05.055 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:55:04.656+0000 7f0a77200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:05.055 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x[32045]: 2024-03-28T18:55:04.656+0000 7f0a77200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T18:55:05.056 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:04 smithi119 podman[103227]: 2024-03-28 18:55:04.760038203 +0000 UTC m=+0.328338759 container died fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T18:55:05.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:05 smithi119 podman[103227]: 2024-03-28 18:55:05.233154585 +0000 UTC m=+0.801455107 container cleanup fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T18:55:05.318 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:05 smithi119 bash[103227]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x 2024-03-28T18:55:05.818 INFO:journalctl@ceph.mgr.x.smithi119.stdout:Mar 28 18:55:05 smithi119 podman[103241]: 2024-03-28 18:55:05.468130376 +0000 UTC m=+0.694133041 container remove fc993220303c66a4c2c9a7d82b1dbda3bd9312f1841847c83a271267f2db890d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-x, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-28T18:55:05.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T18:55:06.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:06.085 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T18:55:06.085 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T18:55:06.085 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.y 2024-03-28T18:55:06.423 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:55:06 smithi131 systemd[1]: Stopping Ceph mgr.y for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:06.694 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:55:06 smithi131 podman[72109]: 2024-03-28 18:55:06.423172103 +0000 UTC m=+0.147550998 container died 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T18:55:07.062 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:55:06 smithi131 podman[72109]: 2024-03-28 18:55:06.836501876 +0000 UTC m=+0.560880820 container cleanup 776175699c69271821e299b426d4db94edfc0882d8680933ba6b59dfaf5e381e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:55:07.062 INFO:journalctl@ceph.mgr.y.smithi131.stdout:Mar 28 18:55:06 smithi131 bash[72109]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-y 2024-03-28T18:55:07.170 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T18:55:07.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:07.206 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T18:55:07.206 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T18:55:07.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.z 2024-03-28T18:55:07.556 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:55:07 smithi196 systemd[1]: Stopping Ceph mgr.z for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:07.844 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:55:07 smithi196 podman[72957]: 2024-03-28 18:55:07.556866128 +0000 UTC m=+0.162157129 container died 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:55:08.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:55:07 smithi196 podman[72957]: 2024-03-28 18:55:07.936316981 +0000 UTC m=+0.541607971 container cleanup 47eb4f47362111218cf7dae76a12cb4e5207cb6cf54969fd107606c6dcb73979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T18:55:08.184 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 28 18:55:07 smithi196 bash[72957]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mgr-z 2024-03-28T18:55:08.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T18:55:08.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:08.440 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T18:55:08.440 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T18:55:08.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.0 2024-03-28T18:55:08.817 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:08 smithi119 systemd[1]: Stopping Ceph osd.0 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:09.147 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0[46839]: 2024-03-28T18:55:08.953+0000 7efdb2400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:09.147 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0[46839]: 2024-03-28T18:55:08.953+0000 7efdb2400700 -1 osd.0 102 *** Got signal Terminated *** 2024-03-28T18:55:09.147 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0[46839]: 2024-03-28T18:55:08.953+0000 7efdb2400700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:14.718 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:14 smithi119 podman[103348]: 2024-03-28 18:55:14.383729816 +0000 UTC m=+5.557719016 container died 77b07b7c6e33c123fd130225fe3b32a8e1e767e2f698608de2b37b8b4f6b8ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T18:55:15.068 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:14 smithi119 podman[103348]: 2024-03-28 18:55:14.854958543 +0000 UTC m=+6.028947740 container cleanup 77b07b7c6e33c123fd130225fe3b32a8e1e767e2f698608de2b37b8b4f6b8ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:55:15.069 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:14 smithi119 bash[103348]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0 2024-03-28T18:55:15.568 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:15 smithi119 podman[103378]: 2024-03-28 18:55:15.098482608 +0000 UTC m=+0.702932595 container remove 77b07b7c6e33c123fd130225fe3b32a8e1e767e2f698608de2b37b8b4f6b8ee1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:16.196 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:15 smithi119 podman[103469]: 2024-03-28 18:55:15.788041793 +0000 UTC m=+0.036337059 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:16.196 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:15 smithi119 podman[103469]: 2024-03-28 18:55:15.907546177 +0000 UTC m=+0.155841478 container create 403486b5fcb27984c67da99df4636c4e4785bb666a8f0c52573500e9f49df178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:55:16.196 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:16 smithi119 podman[103469]: 2024-03-28 18:55:16.195079978 +0000 UTC m=+0.443375286 container init 403486b5fcb27984c67da99df4636c4e4785bb666a8f0c52573500e9f49df178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T18:55:16.568 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:16 smithi119 podman[103469]: 2024-03-28 18:55:16.207841546 +0000 UTC m=+0.456136820 container start 403486b5fcb27984c67da99df4636c4e4785bb666a8f0c52573500e9f49df178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:55:16.568 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:16 smithi119 podman[103469]: 2024-03-28 18:55:16.285482554 +0000 UTC m=+0.533777862 container attach 403486b5fcb27984c67da99df4636c4e4785bb666a8f0c52573500e9f49df178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:55:16.992 INFO:journalctl@ceph.osd.0.smithi119.stdout:Mar 28 18:55:16 smithi119 podman[103469]: 2024-03-28 18:55:16.638690411 +0000 UTC m=+0.886985745 container died 403486b5fcb27984c67da99df4636c4e4785bb666a8f0c52573500e9f49df178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T18:55:17.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T18:55:17.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:17.567 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T18:55:17.567 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T18:55:17.568 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.1 2024-03-28T18:55:17.882 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:17 smithi131 systemd[1]: Stopping Ceph osd.1 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:18.194 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1[40686]: 2024-03-28T18:55:17.881+0000 7faec8845700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:18.194 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1[40686]: 2024-03-28T18:55:17.881+0000 7faec8845700 -1 osd.1 102 *** Got signal Terminated *** 2024-03-28T18:55:18.194 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1[40686]: 2024-03-28T18:55:17.881+0000 7faec8845700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:23.616 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:23 smithi131 podman[72233]: 2024-03-28 18:55:23.303600249 +0000 UTC m=+5.539149090 container died d1627e40f90b8701ec17e7dca1a33698b260bfb17caeb297d63cd3384145ab07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T18:55:23.945 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:23 smithi131 podman[72233]: 2024-03-28 18:55:23.715293722 +0000 UTC m=+5.950842548 container cleanup d1627e40f90b8701ec17e7dca1a33698b260bfb17caeb297d63cd3384145ab07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:55:23.945 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:23 smithi131 bash[72233]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1 2024-03-28T18:55:24.376 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:23 smithi131 podman[72249]: 2024-03-28 18:55:23.967068466 +0000 UTC m=+0.665056171 container remove d1627e40f90b8701ec17e7dca1a33698b260bfb17caeb297d63cd3384145ab07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:55:24.653 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.279553628 +0000 UTC m=+0.017784750 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:24.654 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.401133699 +0000 UTC m=+0.139364848 container create 9ee9df549edb3fa4161f2d39bc727b95898ce71e38d0ef5980663b948c62c077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T18:55:24.654 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.654158153 +0000 UTC m=+0.392389261 container init 9ee9df549edb3fa4161f2d39bc727b95898ce71e38d0ef5980663b948c62c077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:55:24.944 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.658387692 +0000 UTC m=+0.396618806 container start 9ee9df549edb3fa4161f2d39bc727b95898ce71e38d0ef5980663b948c62c077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:55:24.945 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.779181059 +0000 UTC m=+0.517412180 container attach 9ee9df549edb3fa4161f2d39bc727b95898ce71e38d0ef5980663b948c62c077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:55:25.256 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 28 18:55:24 smithi131 podman[72344]: 2024-03-28 18:55:24.951961351 +0000 UTC m=+0.690192459 container died 9ee9df549edb3fa4161f2d39bc727b95898ce71e38d0ef5980663b948c62c077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T18:55:25.597 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T18:55:25.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:25.641 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T18:55:25.641 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T18:55:25.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.2 2024-03-28T18:55:25.935 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:25 smithi196 systemd[1]: Stopping Ceph osd.2 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:26.437 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2[41453]: 2024-03-28T18:55:25.989+0000 7f770bcf8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:26.437 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2[41453]: 2024-03-28T18:55:25.989+0000 7f770bcf8700 -1 osd.2 102 *** Got signal Terminated *** 2024-03-28T18:55:26.437 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2[41453]: 2024-03-28T18:55:25.989+0000 7f770bcf8700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:31.616 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:31 smithi196 podman[73092]: 2024-03-28 18:55:31.309342544 +0000 UTC m=+5.472679112 container died a8ef1c8b5993472228aff783baffafe53ef669a6409935cda1d4d72bbb7681a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:31.900 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:31 smithi196 podman[73092]: 2024-03-28 18:55:31.708102265 +0000 UTC m=+5.871438815 container cleanup a8ef1c8b5993472228aff783baffafe53ef669a6409935cda1d4d72bbb7681a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T18:55:31.900 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:31 smithi196 bash[73092]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2 2024-03-28T18:55:32.184 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:31 smithi196 podman[73110]: 2024-03-28 18:55:31.901143182 +0000 UTC m=+0.585304465 container remove a8ef1c8b5993472228aff783baffafe53ef669a6409935cda1d4d72bbb7681a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T18:55:32.685 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.225087314 +0000 UTC m=+0.018161294 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:32.685 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.327246375 +0000 UTC m=+0.120320331 container create b90aa00e7249e424715dda0de25aaf28ac86c3463d5e537d3815dadf698be766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:55:32.975 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.688286182 +0000 UTC m=+0.481360155 container init b90aa00e7249e424715dda0de25aaf28ac86c3463d5e537d3815dadf698be766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T18:55:32.976 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.691558726 +0000 UTC m=+0.484632695 container start b90aa00e7249e424715dda0de25aaf28ac86c3463d5e537d3815dadf698be766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T18:55:32.976 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.789668817 +0000 UTC m=+0.582742792 container attach b90aa00e7249e424715dda0de25aaf28ac86c3463d5e537d3815dadf698be766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:55:33.242 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 28 18:55:32 smithi196 podman[73199]: 2024-03-28 18:55:32.976370131 +0000 UTC m=+0.769444097 container died b90aa00e7249e424715dda0de25aaf28ac86c3463d5e537d3815dadf698be766 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:33.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T18:55:33.600 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:33.600 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T18:55:33.600 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T18:55:33.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.3 2024-03-28T18:55:34.068 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:33 smithi119 systemd[1]: Stopping Ceph osd.3 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:34.568 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3[53750]: 2024-03-28T18:55:34.173+0000 7f05ac600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:34.568 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3[53750]: 2024-03-28T18:55:34.173+0000 7f05ac600700 -1 osd.3 102 *** Got signal Terminated *** 2024-03-28T18:55:34.568 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:34 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3[53750]: 2024-03-28T18:55:34.173+0000 7f05ac600700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:35.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:35.615+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:36.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:36.567+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:37.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:36 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:36.722+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:37.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:36.917+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:37.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:37.613+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:38.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:37 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:37.733+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:38.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:37.907+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:38.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:38.608+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:38.970 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:38 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:38.711+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:39.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:38 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:38.931+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:39.627 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:39.199+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:39.845 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:39 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:39.665+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:39.845 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:39 smithi119 podman[103574]: 2024-03-28 18:55:39.5921987 +0000 UTC m=+5.546622168 container died 10b7de6ed13aa29fcb7b8db4b8630c87ade97917eeb081ee8f5b11b80f76c5d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T18:55:39.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:39.627+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:40.199 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:40 smithi119 podman[103574]: 2024-03-28 18:55:40.034092604 +0000 UTC m=+5.988516069 container cleanup 10b7de6ed13aa29fcb7b8db4b8630c87ade97917eeb081ee8f5b11b80f76c5d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T18:55:40.199 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:40 smithi119 bash[103574]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3 2024-03-28T18:55:40.434 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:40.166+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:40.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:39 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:39.976+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:40.536 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:40 smithi119 podman[103592]: 2024-03-28 18:55:40.344004294 +0000 UTC m=+0.736926838 container remove 10b7de6ed13aa29fcb7b8db4b8630c87ade97917eeb081ee8f5b11b80f76c5d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T18:55:40.817 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:40.693+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:40.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:40 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:40.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:40.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:40.644+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:41.434 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:41.158+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:41.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:40 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:40.949+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:41.448 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.000280668 +0000 UTC m=+0.035126715 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:41.448 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.120097778 +0000 UTC m=+0.154943846 container create 6c088833ca64009ff5cbe489fc589e468116e18e9baf9b194c42ca3c72b466e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:55:41.449 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.448931538 +0000 UTC m=+0.483777609 container init 6c088833ca64009ff5cbe489fc589e468116e18e9baf9b194c42ca3c72b466e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T18:55:41.729 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:41 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:41.590+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:41.730 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.459566902 +0000 UTC m=+0.494412997 container start 6c088833ca64009ff5cbe489fc589e468116e18e9baf9b194c42ca3c72b466e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:55:41.730 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.56453126 +0000 UTC m=+0.599377352 container attach 6c088833ca64009ff5cbe489fc589e468116e18e9baf9b194c42ca3c72b466e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:55:41.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:41.611+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:42.068 INFO:journalctl@ceph.osd.3.smithi119.stdout:Mar 28 18:55:41 smithi119 podman[103691]: 2024-03-28 18:55:41.87653981 +0000 UTC m=+0.911385865 container died 6c088833ca64009ff5cbe489fc589e468116e18e9baf9b194c42ca3c72b466e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-28T18:55:42.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:41 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:41.729+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:42.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:41 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:41.915+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:42.626 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:42.191+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:42.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T18:55:42.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:42.796 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T18:55:42.796 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T18:55:42.796 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.4 2024-03-28T18:55:42.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:42.625+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:43.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:42 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:42.775+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:43.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:42 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:42.608+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:43.118 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:42 smithi131 systemd[1]: Stopping Ceph osd.4 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:43.118 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:42 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:42.959+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:43.445 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:43 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:43.118+0000 7f5789eba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:43.445 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:43 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:43.118+0000 7f5789eba700 -1 osd.4 102 *** Got signal Terminated *** 2024-03-28T18:55:43.445 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:43 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:43.118+0000 7f5789eba700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:43.652 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:43.189+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:43.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:43.652+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:44.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:43 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:43.762+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:44.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:43 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:43.583+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:44.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:43 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:43.922+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:44.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:43 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:43.887+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:44.435 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:44.184+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:44.435 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:44.184+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:45.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:44 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:44.606+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:45.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:44 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:44.768+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:45.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:44 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:44.768+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:44.685+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:45.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:44.915+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:45.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:44 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:44.875+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:45.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:44 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:44.930+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:45.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:44 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:44.930+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:45.646 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:45.218+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:45.646 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:45.218+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:45.934 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:45.876+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:45.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:45.646+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:46.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:45 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:45.597+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:46.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:45 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:45.797+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:46.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:45 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:45.797+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:46.194 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:45 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:45.858+0000 7f57864ae700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.008740+0000 front 2024-03-28T18:55:22.008790+0000 (oldest deadline 2024-03-28T18:55:44.908999+0000) 2024-03-28T18:55:46.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:45 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:45.924+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:46.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:45 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:45.914+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:46.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:45 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:45.914+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:46.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:46 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:46.222+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:46.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:46 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:46.222+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:47.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:46.589+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:47.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:46.821+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:47.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:46 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:46.821+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:47.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:46 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:46.861+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:47.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:46 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:46.692+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:47.194 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:46 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:46.823+0000 7f57864ae700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.008740+0000 front 2024-03-28T18:55:22.008790+0000 (oldest deadline 2024-03-28T18:55:44.908999+0000) 2024-03-28T18:55:47.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:46 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:46.897+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:47.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:46 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:46.926+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:47.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:46 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:46.926+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:47.662 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:47 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:47.235+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:47.662 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:47 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:47.235+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:47.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:47.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:47.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:47.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:47.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:47 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:47.662+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:47.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:47 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:47.842+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:48.194 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4[47399]: 2024-03-28T18:55:47.833+0000 7f57864ae700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.008740+0000 front 2024-03-28T18:55:22.008790+0000 (oldest deadline 2024-03-28T18:55:44.908999+0000) 2024-03-28T18:55:48.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:47.912+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:48.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:47.972+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:48.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:47 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:47.972+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:48.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:47.860+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:48.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:47 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:47.860+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:48.679 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:48.254+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:48.679 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:48.254+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:48.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:48.548+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:48.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:48.548+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:48.877 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:48 smithi131 podman[72462]: 2024-03-28 18:55:48.561258036 +0000 UTC m=+5.563773119 container died a1ad531d9c535bb87cb07926e7d2713959d91437db22419d940179327d0936c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:48.934 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:48.838+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:48.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:48.679+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:48.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:48 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:48.679+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:49.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:48 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:48.894+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:49.194 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:48 smithi131 podman[72462]: 2024-03-28 18:55:48.985457266 +0000 UTC m=+5.987972349 container cleanup a1ad531d9c535bb87cb07926e7d2713959d91437db22419d940179327d0936c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:55:49.195 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:48 smithi131 bash[72462]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4 2024-03-28T18:55:49.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:48 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:48.981+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:49.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:48 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:48.981+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:49.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:48.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:49.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:48.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:49.648 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72479]: 2024-03-28 18:55:49.228614732 +0000 UTC m=+0.662185859 container remove a1ad531d9c535bb87cb07926e7d2713959d91437db22419d940179327d0936c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T18:55:49.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:49.301+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:49.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:49.301+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:49.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:49.526+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:49.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:49.526+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:49.918 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72568]: 2024-03-28 18:55:49.552255619 +0000 UTC m=+0.017864678 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:49.918 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72568]: 2024-03-28 18:55:49.66291863 +0000 UTC m=+0.128527694 container create 3a0d6e3d53238278685b5212912ea8ce2e3d3ba38a6f19fd6bec2fb85245cf7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:55:49.919 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72568]: 2024-03-28 18:55:49.865493505 +0000 UTC m=+0.331102560 container init 3a0d6e3d53238278685b5212912ea8ce2e3d3ba38a6f19fd6bec2fb85245cf7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:55:49.919 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72568]: 2024-03-28 18:55:49.868538542 +0000 UTC m=+0.334147589 container start 3a0d6e3d53238278685b5212912ea8ce2e3d3ba38a6f19fd6bec2fb85245cf7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T18:55:49.934 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:49.796+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:49.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:49.684+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:49.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:49 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:49.684+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:50.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:49 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:49.918+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:50.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:49 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:49.965+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:50.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:49 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:49.965+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:50.196 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:49 smithi131 podman[72568]: 2024-03-28 18:55:49.96561542 +0000 UTC m=+0.431224473 container attach 3a0d6e3d53238278685b5212912ea8ce2e3d3ba38a6f19fd6bec2fb85245cf7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T18:55:50.196 INFO:journalctl@ceph.osd.4.smithi131.stdout:Mar 28 18:55:50 smithi131 podman[72568]: 2024-03-28 18:55:50.166164004 +0000 UTC m=+0.631773048 container died 3a0d6e3d53238278685b5212912ea8ce2e3d3ba38a6f19fd6bec2fb85245cf7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:55:50.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:49.918+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:50.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:49 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:49.918+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:50.649 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:50.276+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:50.650 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:50.276+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:50.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:50.560+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:50.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:50.560+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:50.850 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T18:55:50.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:50.894 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T18:55:50.894 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T18:55:50.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.5 2024-03-28T18:55:50.924 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:50.794+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:50.925 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:50.650+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:50.925 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:50 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:50.650+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:51.184 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:50 smithi196 systemd[1]: Stopping Ceph osd.5 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:51.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:50 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:50.966+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:51.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:50 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:50.963+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:51.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:50 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:50.963+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:51.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:50.911+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:51.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:50 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:50.911+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:51.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:51.220+0000 7f781be4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:55:51.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:51.220+0000 7f781be4a700 -1 osd.5 102 *** Got signal Terminated *** 2024-03-28T18:55:51.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:51.220+0000 7f781be4a700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:55:51.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:51.253+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:51.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:51.253+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:51.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:51.583+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:51.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:51.583+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:52.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:51.749+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:52.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:51.694+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:52.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:51 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:51.694+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:52.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:51.929+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:52.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:51.929+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:52.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:51 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:51.929+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:52.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:51 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:51.967+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:52.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:52.000+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:52.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:52.000+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:52.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:52.300+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:52.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:52.300+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:52.967 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:52.628+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:52.967 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:52.628+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:53.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:52.798+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:53.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:52.719+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:53.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:52.719+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:53.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:52 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:52.719+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:53.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:52.966+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:53.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:52.966+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:53.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:52 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:52.966+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:53.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:52 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:52.989+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:53.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:53.043+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:53.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:53.043+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:53.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:53.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:53.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:53.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:53.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:53.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:26.495324+0000 front 2024-03-28T18:55:26.495276+0000 (oldest deadline 2024-03-28T18:55:52.395389+0000) 2024-03-28T18:55:53.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:53.621+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:53.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:53.621+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:54.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:53.752+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:54.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:53.752+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:54.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:53.769+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:54.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:53.769+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:54.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:53 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:53.769+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:54.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:53.996+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:54.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:53.996+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:54.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:53 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:53.996+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:54.345 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:54.034+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:54.346 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:54.034+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:54.346 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:53.953+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:54.346 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:53 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:53.953+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:54.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:54.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:54.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:54.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:54.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:54.317+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:26.495324+0000 front 2024-03-28T18:55:26.495276+0000 (oldest deadline 2024-03-28T18:55:52.395389+0000) 2024-03-28T18:55:55.029 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:54.605+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:55.029 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:54.605+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:55.029 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:54.605+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:55:55.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:54.783+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:55.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:54.783+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:55.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:54.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:55.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:54.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:55.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:54 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:54.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:55.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:55.028+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:55.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:55.028+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:55.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:55.028+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:55.360 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:54.965+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:55.360 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:54 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:54.965+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:55.360 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:55.076+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:55.360 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:55.076+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:55.360 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:55.076+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:55:55.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:55.337+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.493434+0000 front 2024-03-28T18:55:12.493212+0000 (oldest deadline 2024-03-28T18:55:38.893761+0000) 2024-03-28T18:55:55.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:55.337+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:18.894502+0000 front 2024-03-28T18:55:18.894617+0000 (oldest deadline 2024-03-28T18:55:43.594463+0000) 2024-03-28T18:55:55.685 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5[48432]: 2024-03-28T18:55:55.337+0000 7f7814436700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:26.495324+0000 front 2024-03-28T18:55:26.495276+0000 (oldest deadline 2024-03-28T18:55:52.395389+0000) 2024-03-28T18:55:56.001 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:55.615+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:56.001 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:55.615+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:56.001 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:55 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:55.615+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:55:56.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:55.818+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:56.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:55.818+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:56.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:55.757+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:56.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:55.757+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:56.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:55 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:55.757+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:56.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:55.921+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:56.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:55 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:55.921+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:56.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:56.065+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:56.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:56.065+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:56.194 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:56.065+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:55:56.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.000+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:56.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.000+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:56.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.000+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:56.954 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:56 smithi196 podman[73318]: 2024-03-28 18:55:56.692841165 +0000 UTC m=+5.605083669 container died ac4fc015386363308954fc023b9f267468e948c4994a2824627310826c56c5ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T18:55:56.955 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:56.781+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:56.955 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:56.781+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:56.955 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:56.713+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:56.955 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:56.713+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:56.955 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:56 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:56.713+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:56.965 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:56.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:56.965 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:56.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:56.965 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:56.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:55:57.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:56.904+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:57.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:56 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:56.904+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:57.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:57.087+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:57.196 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:57.087+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:57.196 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:57.087+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:55:57.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.965+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:57.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.965+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:57.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:56 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:56.965+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:57.329 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73318]: 2024-03-28 18:55:57.053749948 +0000 UTC m=+5.965992460 container cleanup ac4fc015386363308954fc023b9f267468e948c4994a2824627310826c56c5ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:57.329 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 bash[73318]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5 2024-03-28T18:55:57.672 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73336]: 2024-03-28 18:55:57.329801913 +0000 UTC m=+0.637297834 container remove ac4fc015386363308954fc023b9f267468e948c4994a2824627310826c56c5ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-28T18:55:57.935 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73429]: 2024-03-28 18:55:57.575952482 +0000 UTC m=+0.018082234 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:55:57.935 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73429]: 2024-03-28 18:55:57.697521451 +0000 UTC m=+0.139651186 container create 2fcc14c1c416bfe9a335827b280e8448826a6a0656bcfef29ebffcd794eb26a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T18:55:57.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:57.770+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:57.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:57.770+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:57.936 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:57.672+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:57.936 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:57.672+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:57.936 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:57 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:57.672+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:57.955 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:57.680+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:57.955 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:57.680+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:57.955 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:57.680+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:55:58.240 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73429]: 2024-03-28 18:55:57.941632222 +0000 UTC m=+0.383761957 container init 2fcc14c1c416bfe9a335827b280e8448826a6a0656bcfef29ebffcd794eb26a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:55:58.240 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:57 smithi196 podman[73429]: 2024-03-28 18:55:57.944709371 +0000 UTC m=+0.386839108 container start 2fcc14c1c416bfe9a335827b280e8448826a6a0656bcfef29ebffcd794eb26a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T18:55:58.241 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:58 smithi196 podman[73429]: 2024-03-28 18:55:58.041658868 +0000 UTC m=+0.483788617 container attach 2fcc14c1c416bfe9a335827b280e8448826a6a0656bcfef29ebffcd794eb26a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-28T18:55:58.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:57.954+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:58.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:57.954+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:58.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:57 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:57.954+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:58.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:57.951+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:58.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:57 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:57.951+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:58.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:58.116+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:58.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:58.116+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:58.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:58.116+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:55:58.684 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 28 18:55:58 smithi196 podman[73429]: 2024-03-28 18:55:58.240814234 +0000 UTC m=+0.682943979 container died 2fcc14c1c416bfe9a335827b280e8448826a6a0656bcfef29ebffcd794eb26a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:55:58.909 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:58.655+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:55:58.909 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:58.655+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:55:58.909 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:58.655+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:55:59.070 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:58.797+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:59.070 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:58.797+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:59.070 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:58.700+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:59.070 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:58.700+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:59.070 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:58 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:58.700+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:55:59.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T18:55:59.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:55:59.177 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T18:55:59.177 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T18:55:59.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.6 2024-03-28T18:55:59.265 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:58.908+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:55:59.265 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:58.908+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:55:59.266 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:58.908+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:55:59.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:58.998+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:55:59.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:55:58 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:55:58.998+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:55:59.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:59.136+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:55:59.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:59.136+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:55:59.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:55:59 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:55:59.136+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:55:59.568 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 systemd[1]: Stopping Ceph osd.6 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:55:59.934 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:59.830+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:55:59.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:55:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:55:59.830+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:55:59.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:59.674+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:55:59.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:59.674+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:55:59.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:55:59 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:55:59.674+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:00.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:59.662+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:59.662+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:55:59.662+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.743+0000 7f77f9c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.743+0000 7f77f9c00700 -1 osd.6 102 *** Got signal Terminated *** 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.743+0000 7f77f9c00700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.888+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.888+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:56:00.069 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:55:59 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:55:59.888+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:56:00.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:00.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:00.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:00.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:00.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:00.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:00.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:00.099+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:00.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:00.099+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:00.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:00 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:00.099+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:01.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:00.902+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:56:01.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:00.902+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:56:01.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:00.902+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:56:01.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:00.660+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:01.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:00.660+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:01.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:00 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:00.660+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:01.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:00.838+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:01.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:00.838+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:01.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:00.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:01.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:00.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:01.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:00 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:00.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:01.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:01.048+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:01.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:01.048+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:01.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:01.048+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:01.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:01.142+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:01.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:01.142+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:01.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:01.142+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:01.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:01 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:01.142+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:01.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:01.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:01.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:01.893+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.224504+0000 front 2024-03-28T18:55:35.224347+0000 (oldest deadline 2024-03-28T18:56:01.625533+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:01.648+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:01.648+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:02.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:01 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:01.648+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:02.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:01.857+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:02.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:01.857+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:02.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:01.857+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:02.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:01.734+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:02.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:01.734+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:02.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:01 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:01.734+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.001+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:02.097+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:02.097+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:02.097+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:02.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:02.097+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:02.942 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:02.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:02.942 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:02.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:02.943 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:02.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:02.943 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:02.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:03.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:02.903+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:03.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:02.904+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:03.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:02.904+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:03.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:02.725+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:03.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:02.725+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:03.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:02 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:02.725+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:03.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:02.942+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:56:03.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:02.942+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:56:03.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:02.942+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:56:03.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:02.942+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.224504+0000 front 2024-03-28T18:55:35.224347+0000 (oldest deadline 2024-03-28T18:56:01.625533+0000) 2024-03-28T18:56:03.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.996+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:03.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.996+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:03.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:02 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:02.996+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:03.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:03 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:03.074+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:03.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:03 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:03.074+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:03.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:03 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:03.074+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:03.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:03 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:03.074+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:03.915 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:03.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:03.915 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:03.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:03.915 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:03.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:03.916 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:03.641+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:04.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:03.906+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:04.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:03.906+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:04.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:03.906+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:04.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:03.758+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:04.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:03.758+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:04.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:03 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:03.758+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:04.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:03.915+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.116563+0000 front 2024-03-28T18:55:13.117344+0000 (oldest deadline 2024-03-28T18:55:36.616875+0000) 2024-03-28T18:56:04.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:03.915+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.519006+0000 front 2024-03-28T18:55:22.518937+0000 (oldest deadline 2024-03-28T18:55:44.720292+0000) 2024-03-28T18:56:04.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:03.915+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.222291+0000 front 2024-03-28T18:55:28.222328+0000 (oldest deadline 2024-03-28T18:55:51.722540+0000) 2024-03-28T18:56:04.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:03 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6[60731]: 2024-03-28T18:56:03.915+0000 7f77f0400700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.224504+0000 front 2024-03-28T18:55:35.224347+0000 (oldest deadline 2024-03-28T18:56:01.625533+0000) 2024-03-28T18:56:04.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.016+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:04.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.016+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:04.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.016+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:04.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:04.052+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:04.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:04.052+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:04.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:04.052+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:04.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:04.052+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:05.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:04.676+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:05.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:04.676+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:05.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:04.676+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:05.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:04 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:04.676+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:05.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:04.952+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:05.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:04.952+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:05.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:04.952+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:05.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:04.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:05.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:04.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:05.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:04 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:04.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:05.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.983+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:05.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.983+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:05.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:04 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:04.983+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:05.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:05.084+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:05.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:05.084+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:05.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:05.084+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:05.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:05.084+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:05.551 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:05 smithi119 podman[103796]: 2024-03-28 18:56:05.258833144 +0000 UTC m=+5.653657353 container died a686821ef9368fe1fbc26f032176450fe9fb921673baba39f3e2f53bb464a8aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:56:05.818 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:05 smithi119 podman[103796]: 2024-03-28 18:56:05.703742878 +0000 UTC m=+6.098567067 container cleanup a686821ef9368fe1fbc26f032176450fe9fb921673baba39f3e2f53bb464a8aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T18:56:05.818 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:05 smithi119 bash[103796]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6 2024-03-28T18:56:05.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:05.656+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:05.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:05.656+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:05.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:05.656+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:05.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:05 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:05.656+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:05.995 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:05.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:05.995 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:05.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:05.995 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:05.722+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:06.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:05.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:06.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:05.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:06.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:05.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:06.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:06.042+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:06.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:06.042+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:06.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:06.042+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:06.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:06.042+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:06.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:05 smithi119 podman[103816]: 2024-03-28 18:56:05.955360851 +0000 UTC m=+0.681058388 container remove a686821ef9368fe1fbc26f032176450fe9fb921673baba39f3e2f53bb464a8aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-28T18:56:06.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:05.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:06.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:05.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:06.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:05 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:05.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:06.949 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:06 smithi119 podman[103901]: 2024-03-28 18:56:06.638656098 +0000 UTC m=+0.038689741 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:06.996 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:06 smithi119 podman[103901]: 2024-03-28 18:56:06.7811921 +0000 UTC m=+0.181225716 container create 19c7f2834a76da0de46b49d8618852e54f03e0e50cc5333a53bc91d32bb3009a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-28T18:56:06.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:06 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:06.634+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:06.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:06 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:06.634+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:06.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:06 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:06.634+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:06.997 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:06 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:06.634+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:07.038 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:06 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:06.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:07.038 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:06 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:06.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:07.038 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:06 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:06.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:07.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:06.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:06.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:06 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:06.939+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:07.064+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:07.064+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:07.064+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:07.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:07.064+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:07.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:07 smithi119 podman[103901]: 2024-03-28 18:56:07.05969789 +0000 UTC m=+0.459731542 container init 19c7f2834a76da0de46b49d8618852e54f03e0e50cc5333a53bc91d32bb3009a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True) 2024-03-28T18:56:07.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:07 smithi119 podman[103901]: 2024-03-28 18:56:07.070921632 +0000 UTC m=+0.470955272 container start 19c7f2834a76da0de46b49d8618852e54f03e0e50cc5333a53bc91d32bb3009a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T18:56:07.318 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:07 smithi119 podman[103901]: 2024-03-28 18:56:07.150360265 +0000 UTC m=+0.550393897 container attach 19c7f2834a76da0de46b49d8618852e54f03e0e50cc5333a53bc91d32bb3009a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T18:56:07.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:07.039+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:07.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:07.039+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:07.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:07.039+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:07.819 INFO:journalctl@ceph.osd.6.smithi119.stdout:Mar 28 18:56:07 smithi119 podman[103901]: 2024-03-28 18:56:07.499881098 +0000 UTC m=+0.899914734 container died 19c7f2834a76da0de46b49d8618852e54f03e0e50cc5333a53bc91d32bb3009a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T18:56:07.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:07 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:07.666+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:07.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:07 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:07.666+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:07.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:07 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:07.666+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:07.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:07 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:07.666+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:08.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.002+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:08.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.002+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:08.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.002+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:08.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:07.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:08.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:07.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:08.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:07 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:07.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:08.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T18:56:08.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:08.336 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T18:56:08.336 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T18:56:08.336 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.7 2024-03-28T18:56:08.367 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:08.027+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:08.367 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:08.027+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:08.367 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:08.027+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:08.367 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:08.027+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:08.368 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:07.974+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:08.368 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:07.974+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:08.368 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:07 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:07.974+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:08.694 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 systemd[1]: Stopping Ceph osd.7 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:08.989 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.723+0000 7fef53db0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:08.989 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.723+0000 7fef53db0700 -1 osd.7 102 *** Got signal Terminated *** 2024-03-28T18:56:08.989 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.723+0000 7fef53db0700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:09.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:08.690+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:09.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:08.690+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:09.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:08.690+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:09.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:08 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:08.690+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:09.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.977+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:09.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.977+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:09.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:08.977+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:09.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:08.732+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:09.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:08.732+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:09.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:08 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:08.732+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:09.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.988+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.988+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:08 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:08.989+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:09.032+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:09.032+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:09.032+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:09.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:09.032+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:10.027 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:09 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:09.713+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:10.027 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:09 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:09.713+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:10.027 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:09 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:09.713+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:10.027 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:09 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:09.713+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:10.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:09.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:10.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:09.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:10.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:09.995+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:10.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:09.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:10.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:09.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:10.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:09 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:09.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:10.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:09.989+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:10.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:09.989+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:10.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:09.989+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:10.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:10.003+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:10.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:10.003+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:10.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:10.003+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:10.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:10.003+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:10.933 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:10 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:10.679+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:10.933 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:10 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:10.679+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:10.934 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:10 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:10.679+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:10.934 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:10 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:10.679+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:11.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:10.961+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:11.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:10.961+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:11.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:10.961+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:11.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:10.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:11.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:10.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:11.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:10.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:11.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:10 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:10.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:11.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:10.977+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:11.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:10.977+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:10.977+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:10 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:10.977+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:44.343489+0000 front 2024-03-28T18:55:44.343537+0000 (oldest deadline 2024-03-28T18:56:10.243585+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:11.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:11.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:11.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:11.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:11.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:12.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:11 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:11.642+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:12.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:11 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:11.642+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:12.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:11 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:11.642+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:12.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:11 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:11.642+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:12.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:11 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:11.642+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:12.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:11.994+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:12.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:11.994+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:12.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:11.994+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:12.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:11.994+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:12.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:11.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:12.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:11.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:12.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:11.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:12.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:11 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:11.766+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:11.950+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:11.950+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:11.950+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:11 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:11.950+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:44.343489+0000 front 2024-03-28T18:55:44.343537+0000 (oldest deadline 2024-03-28T18:56:10.243585+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:12.009+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:12.009+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:12.009+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:12.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:12.009+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:13.001 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:12 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:12.731+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:13.001 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:12 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:12.731+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:13.001 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:12 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:12.731+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:13.001 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:12 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:12.731+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:13.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:12 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:12.620+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:13.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:12 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:12.620+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:13.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:12 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:12.620+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:13.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:12 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:12.620+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:13.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:12 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:12.620+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:13.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.001+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:13.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.001+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:13.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.001+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:13.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.001+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:13.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:12.964+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.637104+0000 front 2024-03-28T18:55:22.637149+0000 (oldest deadline 2024-03-28T18:55:43.638011+0000) 2024-03-28T18:56:13.444 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:12.964+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.939409+0000 front 2024-03-28T18:55:28.939359+0000 (oldest deadline 2024-03-28T18:55:53.039373+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:12.964+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:36.941516+0000 front 2024-03-28T18:55:36.941634+0000 (oldest deadline 2024-03-28T18:55:59.741995+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:12 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7[54129]: 2024-03-28T18:56:12.964+0000 7fef4c39c700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:44.343489+0000 front 2024-03-28T18:55:44.343537+0000 (oldest deadline 2024-03-28T18:56:10.243585+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:13.058+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:13.058+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:13.058+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:13.058+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:13.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:13.058+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:13.981 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:13.706+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:13.981 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:13.706+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:13.981 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:13.706+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:13.981 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:13.706+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:14.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:13 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:13.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:14.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:13 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:13.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:14.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:13 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:13.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:14.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:13 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:13.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:14.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:13 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:13.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:14.380 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:14 smithi131 podman[72685]: 2024-03-28 18:56:14.062770835 +0000 UTC m=+5.526753716 container died 95c27047623ff3d1183c0429b9dfc3cf3bdd5033907729a09f0c2e596ab94d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:56:14.381 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:14 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:14.085+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:14.381 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:14 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:14.085+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:14.381 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:14 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:14.085+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:14.381 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:14 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:14.085+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:14.381 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:14 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:14.085+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:14.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.982+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:14.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.982+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:14.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.982+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:14.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:13 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:13.982+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:14.694 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:14 smithi131 podman[72685]: 2024-03-28 18:56:14.471268369 +0000 UTC m=+5.935251227 container cleanup 95c27047623ff3d1183c0429b9dfc3cf3bdd5033907729a09f0c2e596ab94d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T18:56:14.694 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:14 smithi131 bash[72685]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7 2024-03-28T18:56:15.061 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:14 smithi131 podman[72702]: 2024-03-28 18:56:14.697440799 +0000 UTC m=+0.632374395 container remove 95c27047623ff3d1183c0429b9dfc3cf3bdd5033907729a09f0c2e596ab94d5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:56:15.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:14 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:14.587+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:15.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:14 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:14.587+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:15.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:14 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:14.587+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:15.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:14 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:14.587+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:15.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:14 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:14.587+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:15.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:14.965+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:15.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:14.965+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:15.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:14.965+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:15.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:14.965+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:15.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:14.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:15.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:14.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:15.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:14.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:15.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:14 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:14.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.041251382 +0000 UTC m=+0.017950119 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.156519413 +0000 UTC m=+0.133218128 container create d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.358989951 +0000 UTC m=+0.335688676 container init d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:15 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:15.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:15 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:15.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:15 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:15.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:15 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:15.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:15.358 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:15 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:15.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:15.652 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.362156721 +0000 UTC m=+0.338855444 container start d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:56:15.653 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.434265542 +0000 UTC m=+0.410964284 container attach d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T18:56:15.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:15 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:15.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:15.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:15 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:15.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:15.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:15 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:15.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:15.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:15 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:15.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:15.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:15 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:15.545+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:15.919 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:15 smithi131 podman[72792]: 2024-03-28 18:56:15.654032481 +0000 UTC m=+0.630731205 container died d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T18:56:16.009 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:15 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:15.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:16.009 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:15 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:15.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:16.009 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:15 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:15.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:16.009 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:15 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:15.750+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:16.163 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T18:56:16.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:16 smithi131 podman[72818]: 2024-03-28 18:56:16.145899484 +0000 UTC m=+0.486680796 container remove d9d2a39ef02fca9240e86fa7e107aaba337665b013df9c2d22b7f08e131c37e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T18:56:16.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:16 smithi131 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-28T18:56:16.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:16 smithi131 systemd[1]: Stopped Ceph osd.7 for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:56:16.194 INFO:journalctl@ceph.osd.7.smithi131.stdout:Mar 28 18:56:16 smithi131 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.7.service: Consumed 32.360s CPU time. 2024-03-28T18:56:16.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:16 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:16.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:16.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:16 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:16.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:16.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:16 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:16.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:16.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:16 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:16.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:16.195 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:16 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:16.037+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:16.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:16.234 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T18:56:16.235 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T18:56:16.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.8 2024-03-28T18:56:16.265 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.009+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:16.265 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.009+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:16.265 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.009+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:16.265 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.009+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:16.538 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 systemd[1]: Stopping Ceph osd.8 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:16.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:16.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:16.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:16.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:16.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:16.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:16.935 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:16.783+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:16.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.538+0000 7fa010dba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:16.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.538+0000 7fa010dba700 -1 osd.8 102 *** Got signal Terminated *** 2024-03-28T18:56:16.935 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:16 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:16.538+0000 7fa010dba700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:17.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:16 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:16.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:17.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:16 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:16.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:17.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:16 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:16.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:17.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:16 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:16.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:17.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:16 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:16.573+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:17.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.032+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:17.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.032+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:17.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.032+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:17.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.032+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:17.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:17.039+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:17.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:17.039+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:17.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:17.039+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:17.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:17.039+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:17.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:17.039+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:18.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:17 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:17.618+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:18.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:17.794+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:18.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:17.794+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:18.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:17.794+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:18.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:17.794+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:18.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.996+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:18.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.996+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:18.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.996+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:18.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.996+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:18.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:17 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:17.996+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:54.753472+0000 front 2024-03-28T18:55:54.753450+0000 (oldest deadline 2024-03-28T18:56:17.653492+0000) 2024-03-28T18:56:18.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:18 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:18.063+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:18.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:18 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:18.063+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:18.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:18 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:18.063+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:18.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:18 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:18.063+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:18.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:18 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:18.063+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:19.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:19.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:19.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:19.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:19.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:19.069 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:18 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:18.659+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:19.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:19.040+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:19.184 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:19.040+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:19.040+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:19.040+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:19.040+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:54.753472+0000 front 2024-03-28T18:55:54.753450+0000 (oldest deadline 2024-03-28T18:56:17.653492+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:18 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:18.810+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:18 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:18.810+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:18 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:18.810+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:18 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:18.810+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:19.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:18 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:18.810+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:19.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:19 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:19.031+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:20.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:19 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:19.611+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:20.028+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:20.029+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:20.029+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:20.029+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:20.029+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:54.753472+0000 front 2024-03-28T18:55:54.753450+0000 (oldest deadline 2024-03-28T18:56:17.653492+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:19.785+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:19.785+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:19.785+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:19.785+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:20.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:19 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:19.785+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:20.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:20.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:20.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:20.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:20.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:20.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:20 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:20.067+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:20.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:20.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:20.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:20.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:20.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:20.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:20 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:20.576+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:21.036 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:20.736+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:21.037 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:20.736+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:21.037 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:20.736+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:21.037 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:20.736+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:21.037 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:20 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:20.736+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:21.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:21.037+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:20.047200+0000 front 2024-03-28T18:55:20.047290+0000 (oldest deadline 2024-03-28T18:55:44.747098+0000) 2024-03-28T18:56:21.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:21.037+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:30.949553+0000 front 2024-03-28T18:55:30.949491+0000 (oldest deadline 2024-03-28T18:55:53.249638+0000) 2024-03-28T18:56:21.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:21.037+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.551096+0000 front 2024-03-28T18:55:35.550638+0000 (oldest deadline 2024-03-28T18:56:01.450476+0000) 2024-03-28T18:56:21.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:21.037+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:45.452159+0000 front 2024-03-28T18:55:45.452063+0000 (oldest deadline 2024-03-28T18:56:11.852564+0000) 2024-03-28T18:56:21.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8[55016]: 2024-03-28T18:56:21.037+0000 7fa00d3ae700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:54.753472+0000 front 2024-03-28T18:55:54.753450+0000 (oldest deadline 2024-03-28T18:56:17.653492+0000) 2024-03-28T18:56:21.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:21.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:21.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:21.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:21.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:21.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:21 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:21.062+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:21.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:21.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:21.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:21.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:21.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:21.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:21 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:21.552+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:21 smithi196 podman[73547]: 2024-03-28 18:56:21.89152638 +0000 UTC m=+5.462253462 container died b7ef2b3fc51324a1f96e5316987857dbeba38d77ae91bd62db93d15ac1e18941 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:21.774+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:21.774+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:21.774+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:21.774+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:22.155 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:21.774+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:22.434 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:22 smithi196 podman[73547]: 2024-03-28 18:56:22.246984686 +0000 UTC m=+5.817711731 container cleanup b7ef2b3fc51324a1f96e5316987857dbeba38d77ae91bd62db93d15ac1e18941 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T18:56:22.435 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:22 smithi196 bash[73547]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8 2024-03-28T18:56:22.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:22.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:22.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:22.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:22.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:22.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:22 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:22.094+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:22.791 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:22 smithi196 podman[73565]: 2024-03-28 18:56:22.531347682 +0000 UTC m=+0.638727709 container remove b7ef2b3fc51324a1f96e5316987857dbeba38d77ae91bd62db93d15ac1e18941 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T18:56:23.059 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:22 smithi196 podman[73658]: 2024-03-28 18:56:22.757118178 +0000 UTC m=+0.017972408 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:23.059 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:22 smithi196 podman[73658]: 2024-03-28 18:56:22.865504313 +0000 UTC m=+0.126358520 container create d49bf9d103192544bb5db0d175e85066308b03c2c8502995a08fbda834d64cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:23.059 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:22 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:22.791+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:23.060 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:22 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:22.791+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:23.060 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:22 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:22.791+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:23.060 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:22 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:22.791+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:23.060 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:22 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:22.791+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:23.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:22 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:22.569+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:23.348 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:23 smithi196 podman[73658]: 2024-03-28 18:56:23.059577723 +0000 UTC m=+0.320431938 container init d49bf9d103192544bb5db0d175e85066308b03c2c8502995a08fbda834d64cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:56:23.348 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:23 smithi196 podman[73658]: 2024-03-28 18:56:23.063501656 +0000 UTC m=+0.324355871 container start d49bf9d103192544bb5db0d175e85066308b03c2c8502995a08fbda834d64cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True) 2024-03-28T18:56:23.348 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:23 smithi196 podman[73658]: 2024-03-28 18:56:23.15951701 +0000 UTC m=+0.420371228 container attach d49bf9d103192544bb5db0d175e85066308b03c2c8502995a08fbda834d64cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T18:56:23.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:23.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:23.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:23.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:23.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:23.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:23 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:23.125+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:23.653 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 28 18:56:23 smithi196 podman[73658]: 2024-03-28 18:56:23.349443842 +0000 UTC m=+0.610298067 container died d49bf9d103192544bb5db0d175e85066308b03c2c8502995a08fbda834d64cc4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-8-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T18:56:23.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:23.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:23.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:23.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:23.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:23.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:23 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:23.544+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:23.904 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:23 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:23.800+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:23.904 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:23 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:23.800+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:23.904 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:23 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:23.800+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:23.904 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:23 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:23.800+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:23.904 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:23 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:23.800+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:23.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T18:56:24.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:24.009 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T18:56:24.009 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T18:56:24.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.9 2024-03-28T18:56:24.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:24.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:24.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:24.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:24.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:24.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:24 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:24.145+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:24.499 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 systemd[1]: Stopping Ceph osd.9 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.499+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.629+0000 7f5d17600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.629+0000 7f5d17600700 -1 osd.9 102 *** Got signal Terminated *** 2024-03-28T18:56:24.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:24 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:24.629+0000 7f5d17600700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:25.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:24 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:24.836+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:25.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:24 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:24.836+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:25.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:24 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:24.836+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:25.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:24 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:24.836+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:25.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:24 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:24.836+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:25.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:25.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:25.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:25.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:25.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:25.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:25 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:25.165+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:25.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:25.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:25.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:25.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:25.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:25.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:25 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:25.502+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:26.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:25.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:26.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:25.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:26.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:25.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:26.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:25.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:26.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:25.793+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:26.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:26 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:26.140+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:26.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:26.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:26.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:26.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:26.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:26.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:26 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:26.512+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:27.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:26 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:26.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:27.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:26 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:26.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:27.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:26 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:26.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:27.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:26 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:26.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:27.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:26 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:26.747+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:27.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:27.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:27.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:27.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:27.444 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:27.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:27 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:27.176+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:27.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:27 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:27.536+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:28.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:28.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:28.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:28.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:28.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:28.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:27 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:27.704+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:28.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:28.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:28.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:28.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:28.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:28.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:28 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:28.206+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:28.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:28.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:28.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:28.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:28.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:28.819 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:28 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:28.559+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:29.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:29.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:29.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:29.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:29.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:29.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:28 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:28.726+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:29.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:29.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:29.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:29.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:29.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:29.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:29.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:29 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:29.226+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.776588+0000 front 2024-03-28T18:55:13.776470+0000 (oldest deadline 2024-03-28T18:55:39.676680+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.677564+0000 front 2024-03-28T18:55:19.677621+0000 (oldest deadline 2024-03-28T18:55:46.678974+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.380930+0000 front 2024-03-28T18:55:28.380737+0000 (oldest deadline 2024-03-28T18:55:53.680454+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:37.182456+0000 front 2024-03-28T18:55:37.182243+0000 (oldest deadline 2024-03-28T18:56:02.481687+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.486502+0000 front 2024-03-28T18:55:47.486549+0000 (oldest deadline 2024-03-28T18:56:10.986700+0000) 2024-03-28T18:56:29.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9[67799]: 2024-03-28T18:56:29.541+0000 7f5d12e00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:52.688770+0000 front 2024-03-28T18:55:52.688850+0000 (oldest deadline 2024-03-28T18:56:16.789015+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:30.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:29.767+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:30.260 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:29 smithi119 podman[104018]: 2024-03-28 18:56:29.983735247 +0000 UTC m=+5.507402000 container died d2b45de65bf9f391c53bc27eec4102a9569f10671879549882e7604ec2f6872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:56:30.568 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:30 smithi119 podman[104018]: 2024-03-28 18:56:30.364133864 +0000 UTC m=+5.887800610 container cleanup d2b45de65bf9f391c53bc27eec4102a9569f10671879549882e7604ec2f6872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:30.568 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:30 smithi119 bash[104018]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9 2024-03-28T18:56:30.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:30.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:30.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:30.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:30.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:30.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:30.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:30 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:30.201+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:31.068 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:30 smithi119 podman[104033]: 2024-03-28 18:56:30.623749646 +0000 UTC m=+0.630461049 container remove d2b45de65bf9f391c53bc27eec4102a9569f10671879549882e7604ec2f6872f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T18:56:31.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:31.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:31.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:31.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:31.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:31.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:30 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:30.780+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:31.511 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.126498834 +0000 UTC m=+0.034210534 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:31.511 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.23272336 +0000 UTC m=+0.140435123 container create 57a07e900114aa521c150ab111ece9b925aca4b9cecee33975e48e9ef023a139 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-28T18:56:31.512 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.51141969 +0000 UTC m=+0.419131433 container init 57a07e900114aa521c150ab111ece9b925aca4b9cecee33975e48e9ef023a139 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T18:56:31.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:31.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:31.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:31.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:31.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:31.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:31.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:31 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:31.194+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:31.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.523779204 +0000 UTC m=+0.431490973 container start 57a07e900114aa521c150ab111ece9b925aca4b9cecee33975e48e9ef023a139 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T18:56:31.818 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.643776016 +0000 UTC m=+0.551487725 container attach 57a07e900114aa521c150ab111ece9b925aca4b9cecee33975e48e9ef023a139 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:56:32.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:32.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:32.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:32.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:32.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:32.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:31 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:31.739+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:32.308 INFO:journalctl@ceph.osd.9.smithi119.stdout:Mar 28 18:56:31 smithi119 podman[104123]: 2024-03-28 18:56:31.95651409 +0000 UTC m=+0.864225833 container died 57a07e900114aa521c150ab111ece9b925aca4b9cecee33975e48e9ef023a139 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T18:56:32.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:32.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:32.238+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:32.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T18:56:32.838 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:32.838 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T18:56:32.839 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T18:56:32.839 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.10 2024-03-28T18:56:33.139 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:32 smithi131 systemd[1]: Stopping Ceph osd.10 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:33.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:33.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:33.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:33.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:33.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:33.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:32 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:32.779+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.139+0000 7febf713d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.139+0000 7febf713d700 -1 osd.10 102 *** Got signal Terminated *** 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.139+0000 7febf713d700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:33.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:33.446 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:33.446 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:33 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:33.231+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:34.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:33.821+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:34.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:34.694 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:34.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:34.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:34.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:34.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:34.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:34 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:34.237+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:35.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:34 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:34.782+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:35.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:35.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:35.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:35.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:35.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:35.696 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:35.696 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:35 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:35.242+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:36.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:35 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:35.753+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:36.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:36 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:36.232+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:37.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:37.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:37.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:37.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:37.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:37.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:36 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:36.772+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:37.695 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:13.913908+0000 front 2024-03-28T18:55:13.914093+0000 (oldest deadline 2024-03-28T18:55:36.213685+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:19.114608+0000 front 2024-03-28T18:55:19.114651+0000 (oldest deadline 2024-03-28T18:55:44.414717+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:29.115774+0000 front 2024-03-28T18:55:29.115954+0000 (oldest deadline 2024-03-28T18:55:55.015898+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6814 osd.3 since back 2024-03-28T18:55:35.016808+0000 front 2024-03-28T18:55:35.016668+0000 (oldest deadline 2024-03-28T18:56:00.916467+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.818060+0000 front 2024-03-28T18:55:47.818078+0000 (oldest deadline 2024-03-28T18:56:12.518193+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:53.019608+0000 front 2024-03-28T18:55:53.019548+0000 (oldest deadline 2024-03-28T18:56:18.319371+0000) 2024-03-28T18:56:37.717 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:37 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10[60775]: 2024-03-28T18:56:37.275+0000 7febf3731700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:04.721159+0000 front 2024-03-28T18:56:04.721309+0000 (oldest deadline 2024-03-28T18:56:28.221039+0000) 2024-03-28T18:56:38.086 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:38.086 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:38.087 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:38.087 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:38.087 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:38.087 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:38.087 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:37 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:37.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:38.944 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:38 smithi131 podman[72908]: 2024-03-28 18:56:38.66078053 +0000 UTC m=+5.625530889 container died 98e9cc69b8fb751779c8709fed3d49ac61f4a72abc6825cd48582eabec02e740 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T18:56:39.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:39.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:39.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:39.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:39.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:39.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:39.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:38 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:38.824+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:39.289 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[72908]: 2024-03-28 18:56:39.030946491 +0000 UTC m=+5.995696850 container cleanup 98e9cc69b8fb751779c8709fed3d49ac61f4a72abc6825cd48582eabec02e740 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T18:56:39.289 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 bash[72908]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10 2024-03-28T18:56:39.639 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[72926]: 2024-03-28 18:56:39.290180909 +0000 UTC m=+0.621460176 container remove 98e9cc69b8fb751779c8709fed3d49ac61f4a72abc6825cd48582eabec02e740 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T18:56:39.944 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[73015]: 2024-03-28 18:56:39.543073067 +0000 UTC m=+0.017106606 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:39.944 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[73015]: 2024-03-28 18:56:39.649073835 +0000 UTC m=+0.123107360 container create aa3a684916c74398903717691bba2a1d21a50854023996a550e51224b0ab24f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:56:39.944 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[73015]: 2024-03-28 18:56:39.884840537 +0000 UTC m=+0.358874069 container init aa3a684916c74398903717691bba2a1d21a50854023996a550e51224b0ab24f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-28T18:56:39.944 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[73015]: 2024-03-28 18:56:39.887964489 +0000 UTC m=+0.361998025 container start aa3a684916c74398903717691bba2a1d21a50854023996a550e51224b0ab24f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T18:56:40.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:40.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:39 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:39.838+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:40.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:39 smithi131 podman[73015]: 2024-03-28 18:56:39.97211237 +0000 UTC m=+0.446145919 container attach aa3a684916c74398903717691bba2a1d21a50854023996a550e51224b0ab24f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:56:40.445 INFO:journalctl@ceph.osd.10.smithi131.stdout:Mar 28 18:56:40 smithi131 podman[73015]: 2024-03-28 18:56:40.177282085 +0000 UTC m=+0.651315626 container died aa3a684916c74398903717691bba2a1d21a50854023996a550e51224b0ab24f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:56:40.740 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T18:56:40.780 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:40.780 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T18:56:40.780 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T18:56:40.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.11 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:40.792+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:40 smithi196 systemd[1]: Stopping Ceph osd.11 for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:41.081 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.080+0000 7f175761d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:41.434 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.080+0000 7f175761d700 -1 osd.11 102 *** Got signal Terminated *** 2024-03-28T18:56:41.434 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.080+0000 7f175761d700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:42.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:41 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:41.807+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:43.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:42 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:42.770+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:44.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:44.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:44.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:44.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:44.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:44.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:44.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:43 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:43.798+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:45.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:45.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:44 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:44.806+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:46.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-03-28T18:55:12.012058+0000 front 2024-03-28T18:55:12.012083+0000 (oldest deadline 2024-03-28T18:55:34.911998+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6804 osd.1 since back 2024-03-28T18:55:22.513699+0000 front 2024-03-28T18:55:22.513627+0000 (oldest deadline 2024-03-28T18:55:48.413681+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-28T18:55:28.414147+0000 front 2024-03-28T18:55:28.414012+0000 (oldest deadline 2024-03-28T18:55:52.514200+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6812 osd.4 since back 2024-03-28T18:55:47.617558+0000 front 2024-03-28T18:55:47.617572+0000 (oldest deadline 2024-03-28T18:56:09.917582+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6812 osd.5 since back 2024-03-28T18:55:55.818533+0000 front 2024-03-28T18:55:55.818595+0000 (oldest deadline 2024-03-28T18:56:18.718711+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.119:6822 osd.6 since back 2024-03-28T18:56:03.320130+0000 front 2024-03-28T18:56:03.320238+0000 (oldest deadline 2024-03-28T18:56:26.820106+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.131:6820 osd.7 since back 2024-03-28T18:56:13.221635+0000 front 2024-03-28T18:56:13.221657+0000 (oldest deadline 2024-03-28T18:56:37.321403+0000) 2024-03-28T18:56:46.185 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:45 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11[61623]: 2024-03-28T18:56:45.764+0000 7f1753c11700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.196:6820 osd.8 since back 2024-03-28T18:56:20.222260+0000 front 2024-03-28T18:56:20.222295+0000 (oldest deadline 2024-03-28T18:56:45.522362+0000) 2024-03-28T18:56:46.721 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:46 smithi196 podman[73786]: 2024-03-28 18:56:46.449930143 +0000 UTC m=+5.475838558 container died 30e5c6193ef04ccf5b60bf0ff5215697ac021e35cbd1c4f21cba403e08b471db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T18:56:46.721 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:46 smithi196 podman[73786]: 2024-03-28 18:56:46.722179544 +0000 UTC m=+5.748087940 container cleanup 30e5c6193ef04ccf5b60bf0ff5215697ac021e35cbd1c4f21cba403e08b471db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:56:47.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:46 smithi196 bash[73786]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11 2024-03-28T18:56:47.184 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:46 smithi196 podman[73804]: 2024-03-28 18:56:46.956784131 +0000 UTC m=+0.507389123 container remove 30e5c6193ef04ccf5b60bf0ff5215697ac021e35cbd1c4f21cba403e08b471db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-28T18:56:47.559 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.177438298 +0000 UTC m=+0.017632142 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T18:56:47.559 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.28239669 +0000 UTC m=+0.122590572 container create e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-28T18:56:47.560 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.501685937 +0000 UTC m=+0.341879782 container init e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-28T18:56:47.560 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.504718629 +0000 UTC m=+0.344912469 container start e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:47.560 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.559722006 +0000 UTC m=+0.399915858 container attach e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T18:56:47.934 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:47 smithi196 podman[73899]: 2024-03-28 18:56:47.785413955 +0000 UTC m=+0.625607798 container died e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:48.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T18:56:48.338 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:48 smithi196 podman[73925]: 2024-03-28 18:56:48.28835855 +0000 UTC m=+0.497066962 container remove e9c61f654e6cc3ef7f41aae42a69565ddc98ca0b48c24bb59244ae71289248d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:48.338 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:48 smithi196 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-28T18:56:48.339 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:48 smithi196 systemd[1]: Stopped Ceph osd.11 for 5ff7da1e-ed31-11ee-95d0-87774f69a715. 2024-03-28T18:56:48.339 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 28 18:56:48 smithi196 systemd[1]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@osd.11.service: Consumed 30.616s CPU time. 2024-03-28T18:56:48.354 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:48.354 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T18:56:48.354 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T18:56:48.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.a 2024-03-28T18:56:48.818 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:48 smithi119 systemd[1]: Stopping Ceph mds.a for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:49.318 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: 2024-03-28T18:56:48.921+0000 7f1d13e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:49.318 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:48 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a[74616]: 2024-03-28T18:56:48.921+0000 7f1d13e00700 -1 mds.a *** got signal Terminated *** 2024-03-28T18:56:52.568 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:52 smithi119 podman[104251]: 2024-03-28 18:56:52.284560618 +0000 UTC m=+3.507565923 container died 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T18:56:52.959 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:52 smithi119 podman[104251]: 2024-03-28 18:56:52.698166634 +0000 UTC m=+3.921171932 container cleanup 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T18:56:52.959 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:52 smithi119 bash[104251]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a 2024-03-28T18:56:53.317 INFO:journalctl@ceph.mds.a.smithi119.stdout:Mar 28 18:56:52 smithi119 podman[104265]: 2024-03-28 18:56:52.959452307 +0000 UTC m=+0.659595242 container remove 943dcc038444adbb8925cf348211fecd9ca084881073cd8e8b18393ba1afc7f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-28T18:56:53.469 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T18:56:53.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:53.579 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T18:56:53.579 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T18:56:53.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.d 2024-03-28T18:56:54.067 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:53 smithi119 systemd[1]: Stopping Ceph mds.d for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:54.568 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d[73650]: 2024-03-28T18:56:54.158+0000 7f0861a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:54.568 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:54 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d[73650]: 2024-03-28T18:56:54.158+0000 7f0861a00700 -1 mds.d *** got signal Terminated *** 2024-03-28T18:56:56.765 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:56 smithi119 podman[104370]: 2024-03-28 18:56:56.466290469 +0000 UTC m=+2.432697186 container died 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-28T18:56:57.067 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:56 smithi119 podman[104370]: 2024-03-28 18:56:56.878484404 +0000 UTC m=+2.844891133 container cleanup 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T18:56:57.068 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:56 smithi119 bash[104370]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d 2024-03-28T18:56:57.419 INFO:journalctl@ceph.mds.d.smithi119.stdout:Mar 28 18:56:57 smithi119 podman[104385]: 2024-03-28 18:56:57.16289186 +0000 UTC m=+0.677257746 container remove 9e4486e6ac164984f1ca4707f1b2774eec17be410d08c00c83f745e7d66f8fda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T18:56:57.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T18:56:57.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:56:57.570 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T18:56:57.570 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T18:56:57.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.g 2024-03-28T18:56:58.068 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:56:57 smithi119 systemd[1]: Stopping Ceph mds.g for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:56:58.568 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:56:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g[74262]: 2024-03-28T18:56:58.178+0000 7fbde3c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:56:58.568 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:56:58 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g[74262]: 2024-03-28T18:56:58.178+0000 7fbde3c00700 -1 mds.g *** got signal Terminated *** 2024-03-28T18:57:00.568 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:57:00 smithi119 podman[104491]: 2024-03-28 18:57:00.28050426 +0000 UTC m=+2.250973687 container died 9831f119a1654262791054828386033ec2da9fef5343f1421f55a3b989b6fd16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:57:00.949 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:57:00 smithi119 podman[104491]: 2024-03-28 18:57:00.706754707 +0000 UTC m=+2.677224155 container cleanup 9831f119a1654262791054828386033ec2da9fef5343f1421f55a3b989b6fd16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:57:00.950 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:57:00 smithi119 bash[104491]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g 2024-03-28T18:57:01.318 INFO:journalctl@ceph.mds.g.smithi119.stdout:Mar 28 18:57:00 smithi119 podman[104505]: 2024-03-28 18:57:00.949610948 +0000 UTC m=+0.651574591 container remove 9831f119a1654262791054828386033ec2da9fef5343f1421f55a3b989b6fd16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:57:01.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T18:57:01.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:01.597 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T18:57:01.597 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T18:57:01.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.j 2024-03-28T18:57:02.068 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:01 smithi119 systemd[1]: Stopping Ceph mds.j for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:02.569 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j[73285]: 2024-03-28T18:57:02.215+0000 7f63b7400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:02.569 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:02 smithi119 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j[73285]: 2024-03-28T18:57:02.215+0000 7f63b7400700 -1 mds.j *** got signal Terminated *** 2024-03-28T18:57:04.577 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:04 smithi119 podman[104611]: 2024-03-28 18:57:04.330723876 +0000 UTC m=+2.242623937 container died 94cbb2f839dec4af88b6ae9c12449df5cef9ec7b65f573e7d8bfd3e2b72e3454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:57:04.869 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:04 smithi119 podman[104611]: 2024-03-28 18:57:04.660204302 +0000 UTC m=+2.572104427 container cleanup 94cbb2f839dec4af88b6ae9c12449df5cef9ec7b65f573e7d8bfd3e2b72e3454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:57:04.869 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:04 smithi119 bash[104611]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j 2024-03-28T18:57:05.318 INFO:journalctl@ceph.mds.j.smithi119.stdout:Mar 28 18:57:04 smithi119 podman[104637]: 2024-03-28 18:57:04.869322963 +0000 UTC m=+0.525853126 container remove 94cbb2f839dec4af88b6ae9c12449df5cef9ec7b65f573e7d8bfd3e2b72e3454 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T18:57:05.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T18:57:05.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:05.525 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T18:57:05.526 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T18:57:05.526 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.b 2024-03-28T18:57:05.896 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:05 smithi131 systemd[1]: Stopping Ceph mds.b for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:06.156 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: 2024-03-28T18:57:05.894+0000 7f9923e8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:06.156 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:05 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b[66019]: 2024-03-28T18:57:05.894+0000 7f9923e8f700 -1 mds.b *** got signal Terminated *** 2024-03-28T18:57:08.470 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:08 smithi131 podman[73120]: 2024-03-28 18:57:08.214678514 +0000 UTC m=+2.496516000 container died db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T18:57:08.754 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:08 smithi131 podman[73120]: 2024-03-28 18:57:08.554189624 +0000 UTC m=+2.836027104 container cleanup db7dc93fa3eb45d5718638e75c847dddb5b56fd641882df66ea3715ed470bb6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T18:57:08.754 INFO:journalctl@ceph.mds.b.smithi131.stdout:Mar 28 18:57:08 smithi131 bash[73120]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-b 2024-03-28T18:57:08.974 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T18:57:09.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:09.012 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T18:57:09.012 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T18:57:09.012 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.e 2024-03-28T18:57:09.332 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:09 smithi131 systemd[1]: Stopping Ceph mds.e for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:09.694 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: 2024-03-28T18:57:09.332+0000 7f51dfcb3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:09.694 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:09 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e[65401]: 2024-03-28T18:57:09.332+0000 7f51dfcb3700 -1 mds.e *** got signal Terminated *** 2024-03-28T18:57:12.574 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:12 smithi131 podman[73253]: 2024-03-28 18:57:12.21266214 +0000 UTC m=+2.978786973 container died 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T18:57:12.858 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:12 smithi131 podman[73253]: 2024-03-28 18:57:12.649386103 +0000 UTC m=+3.415510926 container cleanup 85797476721be833c5af7792638ce0c3831fcf7984b8eec7f6494eaa1d7c50b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-28T18:57:12.858 INFO:journalctl@ceph.mds.e.smithi131.stdout:Mar 28 18:57:12 smithi131 bash[73253]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-e 2024-03-28T18:57:13.068 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T18:57:13.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:13.103 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T18:57:13.103 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T18:57:13.103 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.h 2024-03-28T18:57:13.444 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:13 smithi131 systemd[1]: Stopping Ceph mds.h for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:13.444 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: 2024-03-28T18:57:13.411+0000 7f620088c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:13.444 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:13 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h[65715]: 2024-03-28T18:57:13.411+0000 7f620088c700 -1 mds.h *** got signal Terminated *** 2024-03-28T18:57:16.536 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:16 smithi131 podman[73376]: 2024-03-28 18:57:16.211751483 +0000 UTC m=+2.898439355 container died c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-28T18:57:16.845 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:16 smithi131 podman[73376]: 2024-03-28 18:57:16.636092023 +0000 UTC m=+3.322779885 container cleanup c7c6084b52b7b439cb1f0d18c02d0c2601993a828efd09afc0099c31a4d672a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T18:57:16.845 INFO:journalctl@ceph.mds.h.smithi131.stdout:Mar 28 18:57:16 smithi131 bash[73376]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-h 2024-03-28T18:57:17.066 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T18:57:17.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:17.101 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T18:57:17.101 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T18:57:17.101 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.k 2024-03-28T18:57:17.406 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:17 smithi131 systemd[1]: Stopping Ceph mds.k for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:17.694 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: 2024-03-28T18:57:17.406+0000 7fd32934e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:17.694 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:17 smithi131 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k[65079]: 2024-03-28T18:57:17.406+0000 7fd32934e700 -1 mds.k *** got signal Terminated *** 2024-03-28T18:57:20.532 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:20 smithi131 podman[73499]: 2024-03-28 18:57:20.205583358 +0000 UTC m=+2.897114526 container died bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T18:57:20.532 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:20 smithi131 podman[73499]: 2024-03-28 18:57:20.53337544 +0000 UTC m=+3.224906570 container cleanup bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-28T18:57:20.941 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:20 smithi131 bash[73499]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k 2024-03-28T18:57:20.941 INFO:journalctl@ceph.mds.k.smithi131.stdout:Mar 28 18:57:20 smithi131 podman[73513]: 2024-03-28 18:57:20.709875501 +0000 UTC m=+0.498408000 container remove bee28058a7d463a0cd0b5d179599af9a96e50edd634f0c804991abb6d76a7660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T18:57:20.951 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T18:57:20.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:20.985 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T18:57:20.985 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T18:57:20.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.c 2024-03-28T18:57:21.328 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:21 smithi196 systemd[1]: Stopping Ceph mds.c for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:21.684 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: 2024-03-28T18:57:21.328+0000 7f19fafe5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:21.684 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:21 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c[66641]: 2024-03-28T18:57:21.328+0000 7f19fafe5700 -1 mds.c *** got signal Terminated *** 2024-03-28T18:57:24.543 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:24 smithi196 podman[74005]: 2024-03-28 18:57:24.208555869 +0000 UTC m=+3.018501698 container died 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T18:57:24.544 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:24 smithi196 podman[74005]: 2024-03-28 18:57:24.544535636 +0000 UTC m=+3.354481447 container cleanup 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T18:57:24.934 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:24 smithi196 bash[74005]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c 2024-03-28T18:57:24.935 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 28 18:57:24 smithi196 podman[74032]: 2024-03-28 18:57:24.72908374 +0000 UTC m=+0.519329319 container remove 4890878252c12b86e6904cbb2089b19e2362e34bda04f2bda36fc1e31ea9dca9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T18:57:24.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T18:57:25.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:25.002 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T18:57:25.002 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T18:57:25.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.f 2024-03-28T18:57:25.330 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:25 smithi196 systemd[1]: Stopping Ceph mds.f for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:25.684 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: 2024-03-28T18:57:25.330+0000 7f3291f48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:25.684 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:25 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f[66330]: 2024-03-28T18:57:25.330+0000 7f3291f48700 -1 mds.f *** got signal Terminated *** 2024-03-28T18:57:28.522 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:28 smithi196 podman[74141]: 2024-03-28 18:57:28.208593969 +0000 UTC m=+2.982128876 container died dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T18:57:28.934 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:28 smithi196 podman[74141]: 2024-03-28 18:57:28.522450856 +0000 UTC m=+3.295985764 container cleanup dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T18:57:28.934 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:28 smithi196 bash[74141]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f 2024-03-28T18:57:28.934 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 28 18:57:28 smithi196 podman[74154]: 2024-03-28 18:57:28.715225972 +0000 UTC m=+0.507006909 container remove dfc3be8469c942b87ad5ca3804b41cfc04488117ef7cff7c1529cb7fcb1532ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T18:57:28.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T18:57:28.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:28.982 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T18:57:28.982 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T18:57:28.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.i 2024-03-28T18:57:29.317 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:29 smithi196 systemd[1]: Stopping Ceph mds.i for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:29.684 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: 2024-03-28T18:57:29.317+0000 7f893df8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:29.684 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:29 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i[66954]: 2024-03-28T18:57:29.317+0000 7f893df8e700 -1 mds.i *** got signal Terminated *** 2024-03-28T18:57:32.475 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:32 smithi196 podman[74262]: 2024-03-28 18:57:32.199475633 +0000 UTC m=+2.997694363 container died 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:57:32.828 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:32 smithi196 podman[74262]: 2024-03-28 18:57:32.561170861 +0000 UTC m=+3.359389572 container cleanup 0d9bbf43cdc0e09309bf260cf9b405184b24957439ab1503b2e7e885dc4c124e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T18:57:32.828 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 28 18:57:32 smithi196 bash[74262]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-i 2024-03-28T18:57:32.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T18:57:33.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:33.028 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T18:57:33.028 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T18:57:33.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.l 2024-03-28T18:57:33.434 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:33 smithi196 systemd[1]: Stopping Ceph mds.l for 5ff7da1e-ed31-11ee-95d0-87774f69a715... 2024-03-28T18:57:33.434 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l[66019]: 2024-03-28T18:57:33.339+0000 7fd5f7513700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T18:57:33.434 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:33 smithi196 ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l[66019]: 2024-03-28T18:57:33.339+0000 7fd5f7513700 -1 mds.l *** got signal Terminated *** 2024-03-28T18:57:36.481 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:36 smithi196 podman[74384]: 2024-03-28 18:57:36.208466974 +0000 UTC m=+2.963109475 container died d9354d6b3c93eeae163c5d86948ec7f758103c430656ca91bd4cf32613769e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:57:36.814 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:36 smithi196 podman[74384]: 2024-03-28 18:57:36.572871752 +0000 UTC m=+3.327514253 container cleanup d9354d6b3c93eeae163c5d86948ec7f758103c430656ca91bd4cf32613769e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T18:57:36.815 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 28 18:57:36 smithi196 bash[74384]: ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715-mds-l 2024-03-28T18:57:36.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5ff7da1e-ed31-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T18:57:36.980 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T18:57:36.980 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T18:57:36.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:57:37.302 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:57:42.281 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:57:42.471 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:57:44.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T18:57:45.129 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:57:47.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:57:47.632 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:57:47.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T18:57:47.695 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T18:57:47.697 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi119/crash 2024-03-28T18:57:47.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:57:47.807 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:57:47.807 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:57:47.812 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi131/crash 2024-03-28T18:57:47.813 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:57:47.859 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:57:47.859 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:57:47.861 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi196/crash 2024-03-28T18:57:47.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash -- . 2024-03-28T18:57:47.941 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T18:57:47.942 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-03-28T18:57:47.943 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T18:57:47.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T18:57:48.049 INFO:teuthology.orchestra.run.smithi119.stdout:2024-03-28T18:34:36.540217+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:57:48.050 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T18:57:48.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[SEC\]' /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T18:57:48.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]' /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T18:57:48.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[WRN\]' /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-28T18:57:48.320 INFO:teuthology.orchestra.run.smithi119.stdout:2024-03-28T18:34:36.540217+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T18:57:48.321 INFO:tasks.cephadm:Compressing logs... 2024-03-28T18:57:48.321 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:57:48.325 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:57:48.327 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T18:57:48.355 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:57:48.355 INFO:teuthology.orchestra.run.smithi131.stderr:: No such file or directory 2024-03-28T18:57:48.356 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T18:57:48.356 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:57:48.356 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.b.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.4.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:gzip/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.k.log: -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T18:57:48.358 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:: No such file or directory 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T18:57:48.360 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T18:57:48.363 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:57:48.363 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:57:48.366 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T18:57:48.368 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log: 87.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:57:48.368 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T18:57:48.371 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T18:57:48.371 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:57:48.373 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:57:48.381 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:57:48.381 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T18:57:48.384 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T18:57:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T18:57:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T18:57:48.386 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T18:57:48.387 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T18:57:48.388 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:57:48.389 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T18:57:48.389 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log: 83.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:57:48.390 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T18:57:48.391 INFO:teuthology.orchestra.run.smithi131.stderr: 87.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:57:48.392 INFO:teuthology.orchestra.run.smithi131.stderr: 91.8% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T18:57:48.392 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T18:57:48.394 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T18:57:48.394 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T18:57:48.394 INFO:teuthology.orchestra.run.smithi119.stderr: 90.3% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T18:57:48.395 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T18:57:48.396 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T18:57:48.398 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.x.log: 94.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T18:57:48.400 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T18:57:48.403 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.0.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.9%gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log 2024-03-28T18:57:48.404 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T18:57:48.405 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T18:57:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T18:57:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose --/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.9.log: 87.4% /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T18:57:48.412 INFO:teuthology.orchestra.run.smithi119.stderr: -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T18:57:48.416 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T18:57:48.419 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:57:48.428 INFO:teuthology.orchestra.run.smithi196.stderr: 94.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T18:57:48.429 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T18:57:48.430 INFO:teuthology.orchestra.run.smithi131.stderr: 94.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T18:57:48.431 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.d.log: /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.a.log: 94.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T18:57:48.431 INFO:teuthology.orchestra.run.smithi131.stderr: 90.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:57:48.434 INFO:teuthology.orchestra.run.smithi196.stderr: 92.7% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T18:57:48.436 INFO:teuthology.orchestra.run.smithi196.stderr: 92.8% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T18:57:48.440 INFO:teuthology.orchestra.run.smithi196.stderr: 92.7% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T18:57:48.440 INFO:teuthology.orchestra.run.smithi119.stderr: 90.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T18:57:48.452 INFO:teuthology.orchestra.run.smithi131.stderr: 93.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T18:57:48.466 INFO:teuthology.orchestra.run.smithi119.stderr: 94.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T18:57:48.505 INFO:teuthology.orchestra.run.smithi119.stderr: 94.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T18:57:48.755 INFO:teuthology.orchestra.run.smithi119.stderr: 89.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T18:57:48.868 INFO:teuthology.orchestra.run.smithi131.stderr: 93.0% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T18:57:48.950 INFO:teuthology.orchestra.run.smithi196.stderr: 91.9% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T18:57:49.030 INFO:teuthology.orchestra.run.smithi131.stderr: 91.8% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T18:57:49.688 INFO:teuthology.orchestra.run.smithi119.stderr: 90.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T18:57:53.006 INFO:teuthology.orchestra.run.smithi119.stderr: 93.4% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T18:57:53.177 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T18:57:53.409 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T18:57:53.782 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T18:57:54.022 INFO:teuthology.orchestra.run.smithi131.stderr: 93.7% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T18:57:54.056 INFO:teuthology.orchestra.run.smithi131.stderr: 93.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T18:57:54.183 INFO:teuthology.orchestra.run.smithi131.stderr: 93.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T18:57:54.410 INFO:teuthology.orchestra.run.smithi196.stderr: 93.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T18:57:54.412 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-28T18:57:54.412 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m6.067s 2024-03-28T18:57:54.412 INFO:teuthology.orchestra.run.smithi196.stderr:user 0m21.343s 2024-03-28T18:57:54.412 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m0.794s 2024-03-28T18:57:54.434 INFO:teuthology.orchestra.run.smithi131.stderr: 93.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T18:57:54.436 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-03-28T18:57:54.436 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m6.095s 2024-03-28T18:57:54.436 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m23.766s 2024-03-28T18:57:54.436 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m0.838s 2024-03-28T18:57:54.973 INFO:teuthology.orchestra.run.smithi119.stderr: 93.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T18:57:55.033 INFO:teuthology.orchestra.run.smithi119.stderr: 93.6% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T18:57:55.111 INFO:teuthology.orchestra.run.smithi119.stderr: 93.5% -- replaced with /var/log/ceph/5ff7da1e-ed31-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T18:57:55.116 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-28T18:57:55.116 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m6.760s 2024-03-28T18:57:55.116 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m24.022s 2024-03-28T18:57:55.116 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m2.048s 2024-03-28T18:57:55.116 INFO:tasks.cephadm:Archiving logs... 2024-03-28T18:57:55.116 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi119/log 2024-03-28T18:57:55.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T18:57:58.542 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi131/log 2024-03-28T18:57:58.543 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T18:58:02.089 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi196/log 2024-03-28T18:58:02.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T18:58:05.333 INFO:tasks.cephadm:Removing cluster... 2024-03-28T18:58:05.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force 2024-03-28T18:58:05.656 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:58:06.338 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force 2024-03-28T18:58:06.528 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:58:06.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5ff7da1e-ed31-11ee-95d0-87774f69a715 --force 2024-03-28T18:58:07.019 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 5ff7da1e-ed31-11ee-95d0-87774f69a715 2024-03-28T18:58:07.329 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T18:58:07.329 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T18:58:07.387 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T18:58:07.403 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T18:58:07.418 INFO:tasks.cephadm:Teardown complete 2024-03-28T18:58:07.418 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T18:58:07.430 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T18:58:07.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T18:58:07.437 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T18:58:07.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T18:58:07.595 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T18:58:07.607 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T18:58:07.607 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:58:07.615 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:58:07.617 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T18:58:07.630 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:58:07.631 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +532us[ +547us] +/- 73ms 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -1367us[-1352us] +/- 62ms 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +1192us[+1208us] +/- 76ms 2024-03-28T18:58:07.635 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:58:07.636 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:58:07.636 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-28T18:58:07.637 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +590us[ +590us] +/- 73ms 2024-03-28T18:58:07.637 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -1332us[-1320us] +/- 62ms 2024-03-28T18:58:07.637 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +1239us[+1239us] +/- 76ms 2024-03-28T18:58:07.637 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:58:07.661 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +519us[ +519us] +/- 73ms 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -1379us[-1326us] +/- 62ms 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +1191us[+1243us] +/- 76ms 2024-03-28T18:58:07.671 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T18:58:07.672 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T18:58:07.683 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T18:58:07.683 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T18:58:07.692 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T18:58:07.930 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T18:58:08.007 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T18:58:08.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:58:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:08.184 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:58:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:08.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T18:58:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:08.247 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T18:58:08.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T18:58:08.268 INFO:teuthology.task.internal:Duration was 2096.039707 seconds 2024-03-28T18:58:08.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T18:58:08.279 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T18:58:08.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T18:58:08.287 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T18:58:08.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T18:58:08.332 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:58:08.333 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:58:08.449 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T18:58:08.901 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T18:58:08.901 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-03-28T18:58:08.902 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T18:58:08.960 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-03-28T18:58:08.961 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T18:58:08.990 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-28T18:58:08.990 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T18:58:09.010 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T18:58:09.010 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T18:58:09.017 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T18:58:09.033 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T18:58:09.108 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T18:58:09.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T18:58:09.113 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T18:58:09.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T18:58:09.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T18:58:09.642 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T18:58:09.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T18:58:09.735 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T18:58:09.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T18:58:09.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T18:58:09.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:09.809 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:09.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:09.834 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-03-28T18:58:09.855 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-28T18:58:09.908 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-03-28T18:58:09.962 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:10.006 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:10.020 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T18:58:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T18:58:10.034 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T18:58:10.045 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T18:58:10.046 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi119 2024-03-28T18:58:10.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T18:58:10.159 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi131 2024-03-28T18:58:10.159 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T18:58:10.197 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387/remote/smithi196 2024-03-28T18:58:10.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T18:58:10.235 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T18:58:10.235 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T18:58:10.243 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T18:58:10.245 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T18:58:10.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T18:58:10.304 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T18:58:10.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T18:58:10.359 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T18:58:10.359 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T18:58:10.367 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T18:58:10.369 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T18:58:10.381 INFO:teuthology.orchestra.run.smithi131.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 18:58 /home/ubuntu/cephtest 2024-03-28T18:58:10.383 INFO:teuthology.orchestra.run.smithi196.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 18:58 /home/ubuntu/cephtest 2024-03-28T18:58:10.418 INFO:teuthology.orchestra.run.smithi119.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 18:58 /home/ubuntu/cephtest 2024-03-28T18:58:10.421 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T18:58:10.432 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T18:58:10.455 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T18:58:10.480 DEBUG:teuthology.nuke:shortname: smithi119 2024-03-28T18:58:10.481 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:58:10.505 DEBUG:teuthology.nuke:shortname: smithi131 2024-03-28T18:58:10.505 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:58:10.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.843969', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3D2qClkYq8nAEu0rZhpmCEfxBkuWlfDN9eMuzEB+giRqXCcxNfU/Q/95KjT8tI9mZyHFB8XoWlbjVGhezySRY='} 2024-03-28T18:58:10.527 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-28T18:58:10.527 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T18:58:10.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.847614', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKsiWx3n6ppHzrTRPwWzQgokE0w78ZQ/LD2WGKTuWbkTncUKSqWiAYECxeD/DE/dGPOiPY+Z0ltCPxHhvgNMUJM='} 2024-03-28T18:58:10.550 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625387', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 18:13:37.845921', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOd8P2VxQIiY24rYqHX0TVDB22s7ROE0EWQlxXTK5wXxAgVlTIgOEuRcmwPH3udymPbUxcO38D8N2MXGLM/4GAY='} 2024-03-28T18:58:10.560 INFO:teuthology.orchestra.console.smithi119:Power off 2024-03-28T18:58:10.560 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T18:58:10.583 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-03-28T18:58:10.584 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:10.601 INFO:teuthology.orchestra.console.smithi131:Power off 2024-03-28T18:58:10.601 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T18:58:10.613 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-28T18:58:10.624 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-03-28T18:58:10.624 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:10.640 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-28T18:58:10.641 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T18:58:10.649 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-28T18:58:10.662 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-28T18:58:10.662 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:10.685 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-28T18:58:14.614 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:14.650 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:14.686 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:14.724 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-28T18:58:14.761 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-28T18:58:14.795 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-28T18:58:18.729 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:18.762 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:18.796 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:18.839 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-28T18:58:18.870 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-03-28T18:58:18.904 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-28T18:58:18.972 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-03-28T18:58:22.842 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:22.905 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T18:58:22.952 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-03-28T18:58:23.015 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-28T18:58:23.053 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-03-28T18:58:23.115 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-28T18:58:23.220 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} duration: 2096.0397069454193 failure_reason: '"2024-03-28T18:34:36.540217+0000 mon.a (mon.0) 188 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T18:58:23.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T18:58:23.330 INFO:teuthology.run:FAIL